The Final Result
Let me start this article by showing you the final result of this article. Below you see a screengrab of me
- checking the new mails I received today, from across 3 separate mail accounts (including a gmail account)
- deleting two mails on the fly, while skimming the remaining two
- searching through all my mails for the keywords "thalia AND FIFA" and finding the result immediately
Everything is done within Emacs, no context switching or anything else needed. In this post, I'll describe my set-up as minimalistically as possible.
Setting up the framework
Mu
First, you need to install Mu from the sources, which works as a indexing and search engine on your offline email cache. On Ubuntu 20.04, this consists in extracting the sources, and running
./configure
within the extracted folder. For my setup, I additionally needed to install three dependencies for the configuration to cleanly run through:
sudo apt install libgtk2.0-dev libgmime-3.0-dev libxapian-dev
After this, a simple
make
make install
installed mu.
Encrypting passwords etc.
In what follows, this file will require numerous sensitive data entries, such as passwords, tokens etc. In order to avoid storing these entries in plain text, I follow this stackexchange answer but add a few tweaks. First, if not already existing, create a gpg key via
gpg --full-generate-key
Next, download and install pass and set it up by
pass init "Name of your GPG key"
With this setup, encrypting any type of information is as simple as
pass insert foo/bar
and the prompt will ask you for it, and store it in the given file destination.
Next, save the following python script as something like ~/.offlineimap.py :
import os
import subprocess
def mailpasswd(acct):
acct = os.path.basename(acct)
path = "/home/<username>/.passwd/%s.gpg" % acct
args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
try:
return subprocess.check_output(args).strip()
except subprocess.CalledProcessError:
return ""
def prime_gpg_agent():
ret = False
i = 1
while not ret:
ret = (mailpasswd("prime") == "prime")
if i > 2:
from offlineimap.ui import getglobalui
sys.stderr.write("Error reading in passwords. Terminating.\n")
getglobalui().terminate()
i += 1
return ret
prime_gpg_agent()
change <username> accordingly.
Setting up offlineimap
Offlineimap is already pre-installed in Ubuntu, so the only "setting up" to do is the configuration of the aptly named file ~/.offlineimaprc.
GMail
For GMail, the necessary encryption standards have improved in recent times, and Gmail no longer likes the encryption provided by sending the user/password combination directly (via remoteuser and remotepasseval). Rather, it expects communication via XOAUTH2. Hence, following the official documentation, I created and received my oauth2_client_id and oauth2_client_secret from Google. After that, further following the official documentation, I cloned the gmail-oauth2-tools repo and created the oauth2_token as described, using
python python/oauth2.py --generate_oauth2_token \
--client_id=YOUR_CLIENT_ID --client_secret=YOUR_CLIENT_SECRET
This generated both an access token and a refresh token, both of which I put into encrypted files, as described above. I finalized the offlineimaprc setup, it now looks like this, including one other account that uses basic password security:
[general]
accounts = gmail, sastibe
maxsyncaccounts = 2
pythonfile = ~/.offlineimap.py
[Account gmail]
localrepository = LocalGmail
remoterepository = RemoteGmail
quick = 10
postsynchook = mu index --maildir ~/Mails
[Repository LocalGmail]
type = Maildir
localfolders = ~/Mails/gmail
[Repository RemoteGmail]
type = Gmail
maxconnections = 2
remoteuser = sebastian.a.schweer@gmail.com
oauth2_request_url = https://accounts.google.com/o/oauth2/token
oauth2_client_id_eval = mailpasswd("gmail_oauth2_client_id")
oauth2_client_secret_eval = mailpasswd("gmail_oauth2_client_secret")
oauth2_refresh_token_eval = mailpasswd("gmail_oauth2_refresh_token")
folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail', '[Gmail]/Important']
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
[Account sastibe]
localrepository = Localsastibe
remoterepository = Remotesastibe
quick = 10
postsynchook = mu index --maildir ~/Mails
[Repository Localsastibe]
type = Maildir
localfolders = ~/Mails/sastibe
[Repository Remotesastibe]
type = IMAP
remotehost = imap.1und1.de
ssl = yes
remoteuser = sastibear@sastibe.de
remotepasseval = mailpasswd("sastibe_password")
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
Config for .emacs
Including mu4e
This is fortunately quite simple after the previous installation of mu, just add
(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e/")
(require 'mu4e)
to your .emacs and you are good to go.
Basic setup
These are the necessary settings for using mu4e with offlineimap and XOAUTH2 encryption:
(require 'smtpmail)
(require 'auth-source-xoauth2)
(setq mail-user-agent 'mu4e-user-agent)
(setq mu4e-sent-messages-behavior 'delete)
(setq mu4e-maildir "/home/sastibe/Mails")
(setq mu4e-update-interval 300)
(setq message-send-mail-function 'smtpmail-send-it)
(setq mu4e-get-mail-command "offlineimap")
(setq mu4e-headers-date-format "%d-%m-%Y %H:%M")
(setq mu4e-headers-fields '((:human-date . 20)
(:flags . 6)
(:from . 22)
(:maildir . 8)
(:subject)))
Encryption for XOAUTH2
For XOAUTH2 communication, GMail expects a specific set of parameters to be transmitted.
(defun my-xoauth2-get-secrets (host user port)
(when (and (string= host "smtp.gmail.com")
(string= user "sebastian.a.schweer@gmail.com"))
(list
:token-url "https://accounts.google.com/o/oauth2/token"
:client-id (auth-source-pass-get 'secret "Email/gmail_oauth2_client_id")
:client-secret (auth-source-pass-get 'secret "Email/gmail_oauth2_client_secret")
:refresh-token (auth-source-pass-get 'secret "Email/gmail_oauth2_refresh_token"))))
(setq auth-source-xoauth2-creds 'my-xoauth2-get-secrets)
(add-to-list 'smtpmail-auth-supported 'xoauth2)
(auth-source-xoauth2-enable)
(setq auth-sources (quote (xoauth2 password-store)))
Mu4e Contexts
In order to separate the accounts and keep the commands adjustable, I use mu4e-contexts like so:
(setq mu4e-contexts
`( ,(make-mu4e-context
:name "gmail"
:match-func (lambda (msg) (when msg
(string-prefix-p "/gmail" (mu4e-message-field msg :maildir))))
:vars '(
(mu4e-refile-folder . "/Gmail/[Gmail].Archive")
(mu4e-drafts-folder . "/gmail/[Gmail].Drafts")
(mu4e-sent-folder . "/gmail/[Gmail].Sent Mail")
(mu4e-trash-folder . "/gmail/[Gmail].Trash")
(smtpmail-stream-type . ssl)
(smtpmail-default-smtp-server . "smtp.gmail.com")
(smtpmail-smtp-server . "smtp.gmail.com")
(smtpmail-smtp-service . 465)
(smtpmail-smtp-user . "sebastian.a.schweer@gmail.com")
(auth-source-xoauth2-creds . 'my-xoauth2-get-secrets)
))
,(make-mu4e-context
:name "sastibe"
:match-func (lambda (msg) (when msg
(string-prefix-p "/sastibe" (mu4e-message-field msg :maildir))))
:vars '(
(mu4e-sent-folder . "/sastibe/Gesendete Objekte")
(mu4e-trash-folder . "/sastibe/Gesendete Objekte")
(mu4e-refile-folder . "/sastibe/Spam")
(mu4e-drafts-folder . "/sastibe/Entw&APw-rfe")
(user-mail-address . "sastibear@sastibe.de")
(smtpmail-default-smtp-server . "smtp.1und1.de")
(smtpmail-smtp-user . "sastibear@sastibe.de")
(smtpmail-smtp-server . "smtp.1und1.de")
(smtpmail-stream-type . starttls)
(smtpmail-smtp-service . 587)
))
))
Debugging
If things don't run as expected, the following settings allow for deeper troubleshooting:
(setq auth-source-debug t)
(setq smtpmail-debug-info t)
Sources & Acknowledgements
*Update March 7, 2021*: I modified the article to reflect in the code a much better way of keeping mu4e up to date. The previous version ran into conflicts with different processes trying to keep mu and mu4e up to date at the same time.
Essentially I follow this post from 2017, and adjust it according to my setup (Ubuntu 20.04) as well as new security policies as ordained by Google Mail. I am much indebted to two of the three discussions in the repo for auth-source-pass to figure out the proper way to apply this package.
