How to use certbot for setting up Letsencrypt certificates behind a reverse proxy

Getting the official "certbot" client for Letsencrypt to run on a host that is not directly reachable via HTTP and/or HTTPS is a bit tricky. One specific example where this makes sense is a virtual machine that compartmentalizes some services such as mail (SMTP, IMAP) or real-time communication (XMPP, MQTT, SIP, etc.) and does intentionally not host the associated HTTP server in the same VM. There are multiple options to get Letsencrypt certificates set up on such a host, including to run the certbot client on the actual web server and then copying over the created private keys and certificates. However, this option assumes a strong trust relationships between the VMs, which we try to avoid for abvious security reasons (if the web server is successfully attacked and is configured to be able to ssh into other services for automatic certificate renewal, then the compartmentalization suddenly doesn't make much sense any more). We therefore want to execute the certbot client on the host that will actually use the certificate.

In the following, we assume a setup similar to this:

  • There is only a single IPv4 address assigned to the pool of VMs (which is often the case). In this tutorial, we use the domain test.net and assume this is assigned and under control of the admin.
  • The VM "web" is reachable via HTTP and HTTPS from the public Internet via port forwarding and is using apache2 (nginx or other HTTP servers would work just as well, but the configuration examples below assume apache 2.4).
  • The VM "mail" is reachable via SMTP and IMAPS ports from the public Internet via port forwarding, but can not be directly reached on its HTTP and HTTPS ports. The mail services should use Letsencrypt certificates and be automatically renewed from inside this VM without any manual interaction. Its internal IP address is assumed to be 192.168.1.70.
  • All hosts run Debian Jessie (8.0).

The quickest way to allow certbot to execute on VM "mail" (including renewal) is to:

  1. Setup forwarding of the Letsencrypt authentication challenges from web to mail with a new virtual host in the apache2 instance:
    <VirtualHost *:80>
            ServerName mail.test.net
            ServerAlias imap.test.net
            ServerAlias smtp.test.net

            ProxyPass /.well-known/acme-challenge/ http://192.168.1.70/.well-known/acme-challenge/
            ProxyPassReverse /.well-known/acme-challenge/ http://192.168.1.70/.well-known/acme-challenge/
    </VirtualHost>
    Make sure to include all host aliases that the SMTP/IMAP certificate should be valid for.

  2. On host web, install certbot and a minimal HTTP server, even if that is not going to be used outside Letsencrypt certificate creation and renewal. We use the minimal version of nginx, because it is small and well tested. The default configuration shipped with the Debian package of nginx works without any modification for this usage.
    # echo <EOF >/etc/apt/sources.list.d/backports.list
    deb http://ftp.debian.org/debian jessie-backports main
    EOF
    apt-get update && apt-get install certbot nginx-light
  3. Execute certbot to create the first certificate. Renewal will be automatic from this point on.
    Note: The important part here is not to use standalone mode. Standalone mode uses a slightly different protocol with the Letsencrypt servers that assumes full control of the HTTP and HTTPS ports, which is not true in this scenario. Instead, use the webroot mode, which only relies on HTTP request, which we can forward based on hostnames as configured above with the apache2 virtual host block. With the default configuration of nginx, an example call is
    certbot certonly --webroot -w /var/www/html --email adminattest [dot] net --rsa-key-size 4096 -d mail.test.net -d imap.test.net -d smtp.test.net

Note 1: The list of hostnames forwarded by the HTTP virtual host needs to include at least those passed to the certbot client.

Note 2: If multiple VMs use the same method, they can all use different certbot accounts. It is not necessary to copy the contents of /etc/letsencrypt/accounts between them.

Note 3: If non-root daemons should be able to access the TLS certificates and private keys created by certbot, it is necessary to change their permissions:
chmod +x -R /etc/letsencrypt/live /etc/letsencrypt/archive

Note 4: To automatically restart services that use these certificates after a renewal, you can use a small helper script in /etc/cron.daily/local-restart-services-on-certificate-renewal:#!/bin/sh
export LANG=C
SERVICES="postfix dovecot"
for s in $SERVICES; do
        starttime=`systemctl -a status $s | sed -nr 's/.*Active: active \(running\) since (.*); .*/\1/p'`
        reloadfiles=`find /etc/letsencrypt/live/ -newermt "$starttime"`
        [ -n "$reloadfiles" ] && systemctl restart $s || true
done
Just add all services that reference these certificates (assuming they are managed by systemd).

That should be it. Enjoy auto-renewed certificates for hosts not directly reachable via HTTP or HTTPS.