tl;dr - letsencrypt is awesome, ployst/docker-letsencrypt makes it easy to use with Kubernetes (feel free to check out the blog post that describes it). There are even easier ways to do it these days that I haven’t tried: kube-lego which looks pretty amazing.

After going through figuring out how to run HTTP applications on Kubernetes, as well as how to run databases on Kubernetes, the next natural step is to figure out how to gear up to running HTTPS applications on Kubernetes. The first step in that, of course, is getting letsencrypt working.

Normally the non-kubernetes process of using letsencrypt to enable HTTPS apps looks like this for me:

0. Ensure that all domain names I need are pointing to the server I’m on
1. Stop NGINX (if it’s running)
2. Run certbot standalone with the appropriate options to obtain a cert for all domain names I need
3. Update NGINX server configuration(s) to point to appropriate certs in /etc/letsencrypt/live
4. Restart NGINX

Kuberentes obviously changes this flow a little bit, but it’s not too bad with the help of ployst/docker-letsencrypt. After reading the documentation in the README, along with a very helpful blog post by the author, the steps look to be:

0. Set up Kubernetes
1. kubectl apply/kubectl create the given resource configuration
2. Use either kubectl run or kubectl execto run the commands noted in the ployst/docker-letsencrypt README (save_certs.sh, fetch_certs.sh, etc)

One thing that will be a little different from what I can see on the README is that I have Kubernetes ingress set up and working, I think I’ll also have to create an ingress resource to help out with the ACME DNS challenge for letsencrypt’s validations step.

Welp, with this general idea of what I need to do, let’s jump into my notes on what actually happened.

Creating the right resource configuration

After a bit of experimenting with the example from the blog post, I figured out what the appropriate resource configuration should look like, with lots of help from the README (I had to remove the role annotations, they’re not a thing anymore evidently):

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: letsencrypt-helper-ing
  annotationsn:
    ingress.kubernetes.io/class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "false"
    ingress.kubernetes.io/limit-rps: "20"
spec:
  rules:
- http:
    paths:
  - path: /.well-known/acme-challenge
    backend:
      serviceName: letsencrypt-helper-svc
      servicePort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: letsencrypt-helper-svc
spec:
  type: LoadBalancer
  selector:
    app: letsencrypt-helper
  ports:
- name: http
  protocol: TCP
  port: 80
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: letsencrypt-helper-deployment
  labels:
    name: letsencrypt-helper
spec:
  replicas: 1
  selector:
    matchLabels:
      app: letsencrypt-helper
  template:
    metadata:
      name: letsencrypt
      labels:
        name: letsencrypt
        app: letsencrypt-helper
    spec:
      containers:
  - name: letsencrypt
    image: ployst/letsencrypt:0.3.0
    env:
    - name: EMAIL
      value: <your email>
    - name: DOMAINS
      value: <adddress 1> <address 2> <...>
    - name: SECRET_NAME
      value: letsencrypt-certs-all
    - name: DEPLOYMENTS
      value: <deployment 1> <deployment 2> <...>
    ports:
    - name: ssl-proxy-http
      containerPort: 80

So there’s a bit to unpack there, let’s go top to bottom – I’ll try and explain what exactly is happening so it doesn’t seem so big and scary.

First we have the letsencrypt-helper Ingress Resource – it’s job is to make the letsencrypt helper available to the outside world, in particular exposing the /.well-known/acme-challenge path that letsencrypt will be using to do domain validation.

Next, is the letsencrypt-helper Service and the letsencrypt-helper Deployment for – they’re your normal garden variety Service and Deployment, ensuring that the helper is available and accessible.

You can use the usual commands to ensure that everything is working – kubectl get ing (list the ingresses), kubectl get deployments (list the deployments), etc. Checking the logs with a command like kubectl logs letsencrypt-helper-deployment-<random string> also was very helpful, it will show requests hitting NGINX if you happen to visit the page in your browser (@ http://<your ip>/.well-known/acme-challenge).

I was able to successfully kubectl apply/kubectl create this resource configuration so the next step was to run the commands from the README and obtain the certificates.

Getting the certifcates

Just as the docs suggest, the first command I run is kubectl exec -it letsencrypt-helper-deployment-<random string> -- bash -c './fetch_certs.sh'. Since I’ve already specified the domains through the ENV, the command is nice and simple. After running this command you can go through and run the save_certs.sh command as well to save them to the secret.

And just like that, the certificates are fetched and available! You can check out the Kubernetes Dashboard if you have it installed or do a kubectl get secrets to confirm that a secret with the name SECRET_NAME that you specified earlier was made.

Using the certificates

If you run multiple apps with different domains through your ingress, you need to direct the ACME DNS challenge URL to letsencrypt-helper as well. Here’s an example (sneak peak of the next post on running HTTPS applications):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ing
  annotations:
    ingress.kubernetes.io/class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/limit-rps: "20"
spec:
  tls:
  - hosts:
    - "example.com"
    secretName: letsencrypt-certs-all
  rules:
  - host: "example.com"
    http:
      paths:
      - path: "/.well-known/acme-challenge"
        backend:
          serviceName: letsencrypt-helper-svc
          servicePort: 80
      - path: "/"
        backend:
          serviceName: example-svc
          servicePort: 80
      - path: "/api"
        backend:
          serviceName: example-svc
          servicePort: 4000
  - host: "getexample.com"
    http:
      paths:
      - path: "/.well-known/acme-challenge"
        backend:
          serviceName: letsencrypt-helper-svc
          servicePort: 80
      - path: "/"
        backend:
          serviceName: example-svc
          servicePort: 80
      - path: "/api"
        backend:
          serviceName: example-svc
          servicePort: 4000

This is what the configuration looks like for an Ingress with HTTPS enabled. Note the tls group of options – the Ingress documentation is pretty straigh forward so it was easy to write this.

Here’s what the secret looks like in Kubernetes Dashboard once everythings set up and done:

Better ways to do this

So this is a fairly manual way to retrieve the certificates – note that I’m actually running exec comamnds here.

After getting it set up this way, I came across this fully automated way of obtaining the certificates, called kube-lego. I haven’t tried it yet, but it might absolutely be worth looking to if you’re approaching this for the first time and want a less manual way of getting all this done.

Wrapping up

After going through this, I was pretty happy to have a pretty simple process for getting the certs though it’s a bit manual for my liking. Now it’s time to port some of my applications to that require HTTPS (basically all of them) – stay tuned for the next post, in which I’ll go through what it took to set up a HTTPS-enabled app (we’re already 80% there, being that regular HTTP apps were covered, and I just showed you what the ingress resource looks like).