The Final Result
Let me start this article by showing you the final result of this article. Below you see a screengrab of me
- checking the new mails I received today, from across 3 separate mail accounts (including a gmail account)
- deleting two mails on the fly, while skimming the remaining two
- searching through all my mails for the keywords "thalia AND FIFA" and finding the result immediately
Everything is done within Emacs, no context switching or anything else needed. In this post, I'll describe my set-up as minimalistically as possible.
Setting up the framework
First, you need to install
Mu from the sources, which works as a indexing and search engine on your offline email cache. On Ubuntu 20.04, this consists in extracting the sources, and running
within the extracted folder. For my setup, I additionally needed to install three dependencies for the configuration to cleanly run through:
sudo apt install libgtk2.0-dev libgmime-3.0-dev libxapian-dev
After this, a simple
make make install
Encrypting passwords etc.
In what follows, this file will require numerous sensitive data entries, such as passwords, tokens etc. In order to avoid storing these entries in plain text, I follow this stackexchange answer but add a few tweaks. First, if not already existing, create a gpg key via
Next, download and install pass and set it up by
pass init "Name of your GPG key"
With this setup, encrypting any type of information is as simple as
pass insert foo/bar
and the prompt will ask you for it, and store it in the given file destination.
Next, save the following python script as something like
import os import subprocess def mailpasswd(acct): acct = os.path.basename(acct) path = "/home/<username>/.passwd/%s.gpg" % acct args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path] try: return subprocess.check_output(args).strip() except subprocess.CalledProcessError: return "" def prime_gpg_agent(): ret = False i = 1 while not ret: ret = (mailpasswd("prime") == "prime") if i > 2: from offlineimap.ui import getglobalui sys.stderr.write("Error reading in passwords. Terminating.\n") getglobalui().terminate() i += 1 return ret prime_gpg_agent()
Setting up offlineimap
Offlineimap is already pre-installed in Ubuntu, so the only "setting up" to do is the configuration of the aptly named file
Automatic start with the system
In order for offlineimap to simply run in the background instead of starting it manually, the service needs to be registered in
systemctl. First, the appropriate unit files, which can be found in the source code in the folder
/contrib/systemd/ need to be copied to
/usr/lib/systemd/user, the daemon can then be started via
systemctl --user enable offlineimap.service
For GMail, the necessary encryption standards have improved in recent times, and Gmail no longer likes the encryption provided by sending the user/password combination directly (via
remotepasseval). Rather, it expects communication via XOAUTH2. Hence, following the official documentation, I created and received my
oauth2_client_secret from Google. After that, further following the official documentation, I cloned the
gmail-oauth2-tools repo and created the oauth2_token as described, using
python python/oauth2.py --generate_oauth2_token \ --client_id=YOUR_CLIENT_ID --client_secret=YOUR_CLIENT_SECRET
This generated both an access token and a refresh token, both of which I put into encrypted files, as described above. I finalized the
offlineimaprc setup, it now looks like this, including one other account that uses basic password security:
[general] accounts = gmail, sastibe maxsyncaccounts = 2 pythonfile = ~/.offlineimap.py [Account gmail] localrepository = LocalGmail remoterepository = RemoteGmail autorefresh = 5 quick = 10 postsynchook = mu index --maildir ~/Mails [Repository LocalGmail] type = Maildir localfolders = ~/Mails/gmail [Repository RemoteGmail] type = Gmail maxconnections = 2 remoteuser = firstname.lastname@example.org oauth2_request_url = https://accounts.google.com/o/oauth2/token oauth2_client_id_eval = mailpasswd("gmail_oauth2_client_id") oauth2_client_secret_eval = mailpasswd("gmail_oauth2_client_secret") oauth2_refresh_token_eval = mailpasswd("gmail_oauth2_refresh_token") folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail', '[Gmail]/Important'] sslcacertfile = /etc/ssl/certs/ca-certificates.crt [Account sastibe] localrepository = Localsastibe remoterepository = Remotesastibe autorefresh = 5 quick = 10 postsynchook = mu index --maildir ~/Mails [Repository Localsastibe] type = Maildir localfolders = ~/Mails/sastibe [Repository Remotesastibe] type = IMAP remotehost = imap.1und1.de ssl = yes remoteuser = email@example.com remotepasseval = mailpasswd("sastibe_password") sslcacertfile = /etc/ssl/certs/ca-certificates.crt
Config for .emacs
This is fortunately quite simple after the previous installation of
mu, just add
(add-to-list 'load-path "/usr/local/share/emacs/site-lisp/mu4e/") (require 'mu4e)
.emacs and you are good to go.
These are the necessary settings for using
offlineimap and XOAUTH2 encryption:
(require 'smtpmail) (require 'auth-source-xoauth2) (setq mail-user-agent 'mu4e-user-agent) (setq mu4e-sent-messages-behavior 'delete) (setq message-send-mail-function 'smtpmail-send-it) (setq mu4e-get-mail-command "offlineimap") (setq mu4e-headers-date-format "%d-%m-%Y %H:%M") (setq mu4e-headers-fields '((:human-date . 20) (:flags . 6) (:from . 22) (:maildir . 8) (:subject)))
Encryption for XOAUTH2
For XOAUTH2 communication, GMail expects a specific set of parameters to be transmitted.
(defun my-xoauth2-get-secrets (host user port) (when (and (string= host "smtp.gmail.com") (string= user "firstname.lastname@example.org")) (list :token-url "https://accounts.google.com/o/oauth2/token" :client-id (auth-source-pass-get 'secret "Email/gmail_oauth2_client_id") :client-secret (auth-source-pass-get 'secret "Email/gmail_oauth2_client_secret") :refresh-token (auth-source-pass-get 'secret "Email/gmail_oauth2_refresh_token")))) (setq auth-source-xoauth2-creds 'my-xoauth2-get-secrets) (add-to-list 'smtpmail-auth-supported 'xoauth2) (auth-source-xoauth2-enable) (setq auth-sources (quote (xoauth2 password-store)))
In order to separate the accounts and keep the commands adjustable, I use
mu4e-contexts like so:
(setq mu4e-contexts `( ,(make-mu4e-context :name "gmail" :match-func (lambda (msg) (when msg (string-prefix-p "/gmail" (mu4e-message-field msg :maildir)))) :vars '( (mu4e-refile-folder . "/Gmail/[Gmail].Archive") (mu4e-drafts-folder . "/gmail/[Gmail].Drafts") (mu4e-sent-folder . "/gmail/[Gmail].Sent Mail") (mu4e-trash-folder . "/gmail/[Gmail].Trash") (smtpmail-stream-type . ssl) (smtpmail-default-smtp-server . "smtp.gmail.com") (smtpmail-smtp-server . "smtp.gmail.com") (smtpmail-smtp-service . 465) (smtpmail-smtp-user . "email@example.com") (auth-source-xoauth2-creds . 'my-xoauth2-get-secrets) )) ,(make-mu4e-context :name "sastibe" :match-func (lambda (msg) (when msg (string-prefix-p "/sastibe" (mu4e-message-field msg :maildir)))) :vars '( (mu4e-sent-folder . "/sastibe/Gesendete Objekte") (mu4e-trash-folder . "/sastibe/Gesendete Objekte") (mu4e-refile-folder . "/sastibe/Spam") (mu4e-drafts-folder . "/sastibe/Entw&APw-rfe") (user-mail-address . "firstname.lastname@example.org") (smtpmail-default-smtp-server . "smtp.1und1.de") (smtpmail-smtp-user . "email@example.com") (smtpmail-smtp-server . "smtp.1und1.de") (smtpmail-stream-type . starttls) (smtpmail-smtp-service . 587) )) ))
If things don't run as expected, the following settings allow for deeper troubleshooting:
(setq auth-source-debug t) (setq smtpmail-debug-info t)
Sources & Acknowledgements
Essentially I follow this post from 2017, and adjust it according to my setup (Ubuntu 20.04) as well as new security policies as ordained by Google Mail. I am much indebted to two of the three discussions in the repo for
auth-source-pass to figure out the proper way to apply this package.