tl/dr; I added continuous delivery to my Haskell project (after working through adding CI). The setup is somewhat convoluted, but that’s more due to personal organizational preference. This posts rambles A LOT so feel forward to skip to the end, and check out the config files that made it happen.

Here lie my notes from taking my infrastructure for a Haskell-powered application I’ve been working on from Continous Integration (CI) all the way to Continous Delivery (CD). I’ve written a bit about how I went from zero to CI, but here we’re going to try and take it even further, going all the way to implementing automatic (triggered) deployments.

Gitlab plug

Gitlab is amazing. No other code hosting solution at this point in time is even close in the amount of things it offers devevlopers for little to no cost.

If you’re already a happy Gitlab user (like I am), consider updgrading your account (like I did). Gitlab is a F/OSS project that really does a lot to empower developers, and the fact that they offer free private repos on their site can’t be cheap – give a little back if you feel so inclined. Or don’t, that’s cool too.

DISCLAIMER: I’m hugely biased, and clearly a rabid fan of Gitlab. They don’t pay me, I’m not some sort of “influencer” on their behalf, if that changes I’ll disclose it. If anything I might try and work there in the future (or at least contribute to Gitlab the F/OSS project).

Step 0: Set up automatic building & publishing of containers

I’ve written a bit in previous posts about the hype surrounding containers, and ultimately how it’s been paying off for me. Docker as a container runtime has it’s own issues, like security (rkt less so) and the increased complexity that a newcomer has to face, but in general the container hype is somewhat deserved. Just think of containers like zip files on steroids (if you were around when that’s how people deployed), or skinny AMIs, or fat binaries for the less fortunate group of developers using languages that don’t produce binaries so easily. What I’m trying to say is that if you’ve deployed other ways before, except for fat binaries containers are actually pretty useful.

One of the most important things to achieving continous deployment is nailing down your deployment artifact (Zip file, JAR, Fat binary, container, whatever), and automating the building and publishing of that artifact.

Gitlab provides an excellent guide on how to set up building & pushing with Docker. If you’re crazy and just want to see my configuration instead of reading the proper guide, here’s what my publish step in .gitlab-ci.yml looks like for me:

publish:
  stage: publish
  image: docker
  services:
    - docker:dind
  # Only publish when a release tag (vX.X.X) is pushed
  only:
    - /v[0-9|\.]+/
  except:
    - branches
  script:
    # build the project
    - apk add --update alpine-sdk git linux-headers ca-certificates gmp-dev zlib-dev ghc make curl
    - curl -sSL https://get.haskellstack.org/ | sh
    - make setup build
    # publish
    - docker login -u gitlab-ci-token --password $CI_BUILD_TOKEN registry.gitlab.com
    - make image publish

This configuration is from “the future”, as in it is the configuration after everything was up and working). I was somewhat forced into pushing images to the container registry for the given project. I’ll go into it more later, but I favor having a <project>-infra repository where I normally will push containers to but CI_BUILD_TOKEN that’s provided by Gitlab’s CI runners only allow pushing to the the repo they’re attached to (which is for the better, probably). If you want to push to another registry that isn’t the immediate project being tested (the one where the pipeline is running), you’ll have to set up a Gitlab personal access token.

You could use Gitlab Secret Variables and envsubst (provided you added some replacement step that makes temp files) to replace some variables/make this more robust, but for me, I was fine hardcoding registry.gitlab.com since it’s the only registry I use for my projects.

Of course, there’s a bit of work tucked away in my Makefile, but basically it builds the docker image (docker build -f ...) based on the version that’s present in the cabal/Makefile, and pushes it (docker push ...).

Background: Rationale behind setting up an -infra repo

I’ve recently taken to creating top level <project>-infra repositories for projects of mine. This is kind of a dirty hack to the 12 Factor apps methodology, but I’m sidestepping having configuration with my app code by just… putting in a separate but close repo alongside the project code. For me the book-keeping, transparency, and utility of having an -infra repo is worth the additional attack surface (if someone compromises my Gitlab repo or Gitlab.com as a whole obviously I’m vulnerable).

A few cases I find this pattern useful:

• Managing app-adjacent infrastructure that doesn’t quite fit with your normal app code (redis, postgres, etc). Configuration can go in the infra repo that doesn’t really belong with the app per-say.
• Containers that need to be customized/tweaked can have their Dockerfiles/other stuff in the infra repo (saves you from having to make some <app>-redis repo).
• Can store multiple Kubernetes resource configurations quite easily (and arguably they’ll belong better than if they were in the project repo)

For smaller projects, I just use a small infra folder in some projects and just use that, because there isn’t much “infrastructure” to manage. If on a small project I can get by with just the default postgres container or not many additional infrastructure-related customizations then I’ll usually use that as it’s simpler.

Step 0: RTFM on Kubernetes (+ authentication/authorization)

If you haven’t yet, read up on the differences between Authentication and Authorization, they’re different. The Kubernetes guide to authentication is excellent and a good place to start. In particular Service Accounts are the exact construct added to help non-human users/robots to interact with Kubernetes safely and securely.

The Kubernetes v1.8 release mentions Kubernetes RBAC (Role Based Access Control, very much a big authorization concept), which I believe got added in v1.6ish.

Step 0.1: Read up on Gitlab/Kubernetes integration

Gitlab offers per-project Kubernetes integration, and it’s pretty damn convenient, check it out. My original plan was to use the pre-prepared kubernetes-deploy but I decided against it, there was a little too much happening there and I wanted to take a more hands-on approach. That makes this blog post essentially me re-doing a bunch of work people have already thought through.

After going through all this documentation, my general plan is pretty simple:

1. Ensure the project I want to auto-deploy publishes when I tag a release
2. Add a service account for Gitlab CI deployments to my Kubernetes cluster
3. Update the infra repo with a stage that attempts to do the deploy (by calling kubectl)
4. ???
5. Profit

Step 1: Ensure the project is publishing images

As is par for the course, paranoia that nothing was actually working struck and I spent some time ensuring that the publish Gitlab CI/CD stage was running. The easiest way to test stuff with CI is often to just make a branch and test out changes (disable/enable other stages for speed).

I ran into one problem – insanely slow build times. A from-scratch stack build would take something like 30 minutes on the runner and that’s not ideal. It was time to take my multi-container build hack to the Gitlab environment as well, which means pushing builder-base and builder images to Gitlab. Luckily, this doesn’t change the .gitlab-ci.yml steps much (and I can rely on the steps in my Makefile more), only that they’ll fail if the builder-base and builder images aren’t found.

Step 2: Create the Kubernetes service account in the cluster

Obviously, you need to have a kubernetes cluster for this step to work, I run a single-node Kubernetes cluster on top of CoreOS on a dedicated server, I’ve also written about it if you’re interested in how that came to be.

Here’s the configuration that I created for the service account (basically just like in the service account user guide):

---
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: false
metadata:
  name: build-robot
  namespace: your-project
imagePullSecrets:
  - name: infra-gitlab-registry # I assume  you're already using some key to pull from private repos

---
apiVersion: v1
kind: Secret
metadata:
  name: build-robot-secret
  type: kubernetes.io/service-account-token
  namespace: your-project
  annotations:
    kubernetes.io/service-account.name: build-robot

There’s a bit more that’s not written above, but that you’ll likely have to do (if you haven’t already):

• Create a Kubernetes namespace for the project (which I’ve called your-project here).
• Copy the Gitlab repository container pulling secret you may or may not be using into that new namespace as well
• Update the k8s resource files you were using before to only create resources in the new namespace
• Re-deploy the old application with the updated resource files, and kill the previous app

These resources should be created relatively effortlessly.

Step 3: Turn on Gitlab Kubernetes integration for your project

Check out the Gitlab documentation, it’s pretty easy.

The things I found hardest to retrieve/fill out were the CA cert and the token. For the token, as you might imagine, kubectl + some unix magic did the trick:

kubectl describe secret build-robot-secret -n your-project | grep "token" | xclip -selection clipboard

For the CA cert you just have to have saved the CA you’ve been using to access your kubernetes cluster this whole time. Of course, use the namespace for your project in the form (you don’t have to use the auto-generated one).

After clicking OK the Kubernetes integration was green/“on” fairly quickly – I’m not sure if there’s any checking on Gitlab.com’s part.

Step 4: Add the deploy stage

Let’s start adding the stage that will actually do the deployment to our <project>-infra repo.

The first important question is which base container the CI runner should start from. Originally I wanted to use wernight/kubectl, but it turns out that due to the Dockerfile’s ENTRYPOINT, it’s not quite possible (more on this later). I also needed to refresh my memory on the process of setting credentials for kubectl.

Unfortunately the first like 3 kubectl containers I found (I wasn’t very tenacious) all had ENTRYPOINT set so I just gave up on them, and started from alpine like usual. This means I needed to install Kuberenetes in <project>-infra/.gitlab-ci.yml:

cache:
  paths:
    - binary-cache
# ... other stuff ...
before_script:
  - apk add --no-cache curl ca-certificates
  # create the binary-cache folder if not present
  - test -d binary-cache || mkdir binary-cache
  # install kubectl binary into binary-cache if not already present
  - test -f binary-cache/kubectl || curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl
  - test -f binary-cache/kubectl || mv kubectl binary-cache/kubectl
  # install kubectl binary into /usr/local/bin if not already present
  - test -f /usr/local/bin/kubectl || install -m 770 binary-cache/kubectl /usr/local/bin
# ... other stuff ...
deploy:
  stage: deploy
  only:
    - web # for testing, remove this later
  script:
    - echo $KUBECONFIG
    - echo $KUBE_TOKEN
    - echo $KUBE_URL
    - kubectl get pods -o yaml -n <your-project>

This is in the before_script section, but it could easily go somewhere else (like in the step itself). I’m expecting that just about all my infra related steps will involve kubectl in some way for this project. As you can see I’m also taking advantage of the Gitlab’s build caching as well, with some unix commandline magic to make decisions as steps run.

If you’re able to get this far, you should be able to run the CI pipeline and see the configuration variables you expect output properly as well as a listing of the pods in your project’s namespace.

The configuration you see above is actually “from the future” – it’s the result of working through a few problems, they’re detailed below:

Problem: There is no kubernetes service

Yeah, that’s not a thing – I thought Gitlab had some configuration like:

services:
  - kubernetes

I expected I’d need to put in to use kubectl from my containers or enable the integration or something, but that’s not the case. The fix here is pretty simple… just don’t do this.

Problem: The pre-built kubectl packages don’t work

I kept getting weird errors in Gitlab CI/CD runner output related to sh not being a command in kubectl, and this is where I figured out that containers that have ENTRYPOINT are no good for use with Gitlab’s CI/CD. For the script section, they use sh to execute the lines you provide so if the ENTRYPOINT on the docker container is set, that means that the command will turn into kubectl sh, as far as I understood.

The fix here was to use a container that doesn’t have ENTRYPOINT set (basically I went back to just using bare alpine.

Problem: None of the variables were defined

I figured I might be using the Gitlab Kubernetes integration wrong, since none of the variables that were supposed to be available were – turns out my URL was wrong. I used https://<server IP>, but I don’t run kubernetes on that port (443 for HTTPS), so I needed to replace that with https://<server IP>:<kubernetes port>. While I was sure this was an issue that I had, it didn’t fix the problem and I kept not seeing the expected variables in the CI/CD runner build output, so I kept searching.

It turns out I had another issue – thanks to an SO thread I learned that you need to configure a Gitlab Environment for the Kubernetes integration to work properly.

Problem: Specifying the kubectl configuration

KUBECONFIG is an environment variable that is the file path of a generated kubectl config file that is provided by Gitlab as part of the Kubernetes integration. It took an embarassing amount of time to find the kubectl documentation, and figure out how I could specify the configuration location, only to realise that Gitlab sets KUBECONFIG for you which means you don’t need to spend time configuring kubectl, that’s the ENV variable that kubectl is looking for.

Problem: kubectl not returning any output

This problem was a big head-scratcher – for some reason the kubectl commands I was running seemed to be executing but didn’t return any output (despite returning a zero exit code). It turns out my curl command was wrong (immune to > redirection I think), and the binary was going to the wrong place. The .gitlab-ci.yml config above doesn’t have this issue but I did run into it.

A big helper in debugging this issue was just starting an alpine container locally and going through the steps, making sure they worked.

Problem: HTTPS issue, k8s server cert not recognized

When using kubectl from the Gitlab CI/CD runner, I got an error that noted that my Kubernetes server had an unrecognized HTTPS cert. My immediate reflex was to use something like --insecure-skip-tls-verify=true or something to bypass the problem (at least temporarily), but then I remembered that the integration actually asked for a CA cert for this very reason, so I must have been using it wrong. Looking at a similar kops issue also sealed the deal for this hypothesis, surely I didn’t set up my cluster so wrongly that the CA cert didn’t actually work and I’ve been vulnerable to a MITM this whole time?

It turns out the problem was just that I had the wrong cert file pasted in the Kubernetes integration configuration – a quick change and kubectl cluster-info was returning relevant information.

Step 5: Try a manual deploy and reflect

Here’s the finished full .gitlab-ci.yml for the <project>-infra repo:

image: alpine:3.6

stages:
  - deploy

cache:
  paths:
    - binary-cache

  before_script:
    - apk add --no-cache curl ca-certificates
    # create the binary-cache folder if not present
    - test -d binary-cache || mkdir binary-cache
    # install kubectl binary into binary-cache if not already present
    - test -f binary-cache/kubectl || curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl
    - test -f binary-cache/kubectl || mv kubectl binary-cache/kubectl
    # install kubectl binary into /usr/local/bin if not already present
    - test -f /usr/local/bin/kubectl || install -m 770 binary-cache/kubectl /usr/local/bin

deploy_production:
  stage: deploy
  environment:
    name: production
  ## Uncomment the lines below to run only when the tag `your-project-production` is pushed to the repo
  # only
  #   - your-project-production
  # except
  #   - branches
  script:
    - kubectl apply -f kubernetes/your-project.yaml

To test this, I manually ran the pipeline from the web. For verification you can check the state of your cluster with whichever relevant kubectl commands and see that an apply (or whatever command you used) did occur.

With kubectl commands finally working it was time to reflect on the current flow (if I wanted to stop here):

1. Code changes happen in a particular <project> repo
2. When enough code is written, a new tag for a release it tagged on master branch (ex. v1.2)
3. <project>’s Gitlab CI config builds and pushes a new Docker container to the <project>-infra registry
4. Infrastructure code changes (version bumping in k8s resource configs) get committed (or feature branch merged) to <project>-infra project’s master branch that represents (and basically is) the change to the infrastructure
5. I tag some commit on master with with <project>-production
6. Gitlab runs the deploy_production job in the deploy stage upon seeing the pushed tag, and the appropriate resource file(s) get kubectl apply -f’d.

This is still a little manual (step 4 in particular), as I need to change the <project>-infra repo by hand, but it’s not a bad first step. Why stop here when we can go all the way to automated CD?

This is a good time to recognize that we’re standing on the shoulders of giants standing on the shoulders of other giants. There is a LOT of code coming together to make all this work, along with computers coordinating. Getting this far makes me quite happy I’ve taken the time to become at least competent in the requisite platforms and technologies:

• Linux (alpine has been wonderful to use, shell scripting magic has been fun to write, and very effective)
• Git
• Docker
• Kubernetes

Step 6: Fully automate the deploy

At this point, I’m almost to the goal, now I just need to fully automate the deploy CI/CD stage running in the <project>-repo when a relevant push is made from the <project> repo. What I want is to trigger a CI action in Gitlab, and there’s documentation for that. Gitlab makes it pretty easy to use webhooks to accomplish this, so I basically just need to send a webhook to Gitlab from the Gitlab CI/CD runner. I also found it useful to review the CI build permissions model that Gitlab has adopted.

The last piece of this puzzle is to enable a HTTP request (cURL) to Gitlab’s trigger API from the CI/CD runner running on Gitlab during the deploy stage in <project> repo. Once the <project> repo does what it thinks is a deploy, the appropriate <project>-infra repo CI/CD stage will be triggered and do the ACTUAL Kubernetes-powered deploy. The setup is a little convoluted, but given my penchant for using <project>-infra repositories, it’s as simple as it can get.

Step 6.1: Adding the trigger to the <project>-infa repo

It’s pretty easy to Create a gitlab trigger by navigating to the Settings > CI/CD > Pipeline Trigger area of the UI (see the documentation for triggers), I added one called “deploy-production”

Step 6.2: Add a personal access token for use from CI

Easily done through the Gitlab user settings UI.

Step 6.2: Updating the <project> repo to with a faux “deploy” stage

I call it a faux deploy stage since all it does is signal another repo to do a deploy. Here’s what this looks like for me:

deploy:
  stage: deploy
  image: alpine:3.6
  # Only deploy when a release tag (vX.X.X) is pushed
  only:
    - /v[0-9|\.]+/
  except:
    - branches
  script:
    - apk --no-cache add curl ca-certificates
    # Get the current version out of the Makefile
    - export DEPLOY_VERSION=`awk -F= '/DEPLOY_VERSION=(.*)/{print $2}' Makefile`
    - echo -e "Triggering deploy for version [$DEPLOY_VERSION]..."
    - "curl -X POST -F token=$INFRA_PIPELINE_TOKEN  \"ref=try-deploy-pipeline\" \"$INFRA_PIPELINE_TRIGGER_URL\""

The Gitlab Secret Variables INFRA_PIPELINE_TOKEN and INFRA_PIPELINE_TRIGGER_URL of course need to be set for the <project> repo (basically telling it how to find the <project>-infra repo.

If this seems too easy, that’s because it is. there’s quite a bit of information we need to pass to the -infra repo to make things actually work (for the right version to be deployed and so-on), the real commend looks more like this:

deploy:
  stage: deploy
  image: alpine:3.6
  # Only deploy when a release tag (vX.X.X) is pushed
  only:
    - /v[0-9|\.]+/
  except:
    - branches
  script:
    - apk --no-cache add curl ca-certificates
    # Get the current version out of the Makefile
    - export DEPLOY_VERSION=`awk -F= '/DEPLOY_VERSION=(.*)/{print $2}' Makefile`
    - echo -e "Triggering deploy for version [$DEPLOY_VERSION]..."
    - "curl -X POST -F token=$INFRA_PIPELINE_TOKEN -F \"ref=master\" -F \"variables[TARGET_IMAGE_VERSION]=$DEPLOY_VERSION\" -F \"variables[PROJECT_NAME]=your-project\" -F \"variables[POD_NAME]=your-project-<your-project>\" -F \"variables[IMAGE_NAME]=registry.gitlab.com/<gitlab username>/<your-project>/<your-project>\" \"$INFRA_PIPELINE_TRIGGER_URL\""

I needed to pass a bunch of variables that a more complicated script on the <project>-infra side would use to deploy the right version of the right containers to the right place.

You don’t have to do it exactly the same way I did of course, but (with hindsight) I found I needed at least this number of extra information passed along to get things right. The next thing is to make the script that will do the hard work in the <project>-infra repo.

Step 6.2: Add a script to the infra repo that will help with heavy lifting

In the infra repo, I’ve updated the deploy step to look like this:

deploy_production:
  stage: deploy
  environment:
    name: production
  only:
    - triggers
  script:
    - apk add --update git
    - scripts/rollout-update
    - scripts/commit-tag-push

As you can see, there are two new scripts that have quite a bit of work in them: scripts/rollout-update and scripts/commit-tag-push. As you might have realized, I actually added something to automatically tag stuff for me! Here’s the contents of those scripts:

rollout-update

#!/bin/sh

# Ensure required variables are provided
if [ -z "$PROJECT_NAME" ]; then echo "ERROR: \$PROJECT_NAME ENV variable is unspecified"; exit 1; fi
if [ -z "$POD_NAME" ]; then echo "ERROR: \$POD_NAME ENV variable is unspecified"; exit 1; fi
if [ -z "$IMAGE_NAME" ]; then echo "ERROR: \$IMAGE_NAME ENV variable is unspecified"; exit 1; fi
if [ -z "$TARGET_IMAGE_VERSION" ]; then echo "ERROR: \$TARGET_IMAGE_VERSION ENV variable is unspecified"; exit 1; fi
if [ -z $(which kubectl) ]; then echo "ERROR: \`kubectl\` executable is missing"; exit 1; fi

echo -e "INFO: Updating image ${IMAGE_NAME} to ${TARGET_IMAGE_VERSION} for deployment/${PROJECT_NAME}..."
echo -e "CMD: kubectl set image deployment/${PROJECT_NAME} ${POD_NAME}=${IMAGE_NAME}:${TARGET_IMAGE_VERSION} -n $PROJECT_NAME"
kubectl set image deployment/${PROJECT_NAME} ${POD_NAME}=${IMAGE_NAME}:${TARGET_IMAGE_VERSION} -n $PROJECT_NAME
if [ $? -ne 0 ]; then echo "ERROR: Failed to update image version ('kubectl set image' command failed)"; exit 1; fi

echo "INFO: Checking rollout status..."
echo -e "CMD: kubectl rollout status $PROJECT_NAME -n $PROJECT_NAME"
kubectl rollout status deployment/${PROJECT_NAME} -n $PROJECT_NAME
if [ $? -ne 0 ]; then echo "ERROR: Rollout status check failed ('kubectl rollout status' command failed)"; exit 1; fi

Not that it matters (because I’m in a <project>-infra repo), but this set up is pretty generic, and as long as I give it the right environment variables, it will attempt to do a kubernetes patch and rollout.

commit-tag-push:

#!/bin/sh

# Ensure required variables are provided
if [ -z "$PROJECT_NAME" ]; then echo "ERROR: \$PROJECT_NAME ENV variable is unspecified"; exit 1; fi
if [ -z "$POD_NAME" ]; then echo "ERROR: \$POD_NAME ENV variable is unspecified"; exit 1; fi
if [ -z "$IMAGE_NAME" ]; then echo "ERROR: \$IMAGE_NAME ENV variable is unspecified"; exit 1; fi
if [ -z "$TARGET_IMAGE_VERSION" ]; then echo "ERROR: \$TARGET_IMAGE_VERSION ENV variable is unspecified"; exit 1; fi
if [ -z "$CI_ENVIRONMENT_NAME" ]; then echo "ERROR: \$CI_ENVIRONMENT_NAME Gitlab-CI provided variable is missing"; exit 1; fi
if [ -z "$PROJECT_INFRA_REPO_LOCATION" ]; then echo "ERROR: \$PROJECT_INFRA_REPO_LOCATION ENV variable is unspecified"; exit 1; fi
if [ -z $(which git) ]; then echo "ERROR: \`git\` executable is missing"; exit 1; fi

# NOTE
# At this point it is assumed that a new version (TARGET_IMAGE_VERSION) version has successfully rolled out

# Update the given projects to the given versions
echo "INFO: Automatically commiting update to k8s config for [$PROJECT_NAME]";

git config --global user.email "admin@example.com"
git config --global user.name "Gitlab CI"

# Attempting to replace the image name
# `kubectl patch` would be better for this, but I don't know that it works on static files (and I don't want to pull & save the live config from the cluster)...
echo -e 'CMD: sed -i "s@image\: $IMAGE_NAME\:.*@image\: $IMAGE_NAME:$TARGET_IMAGE_VERSION@g" kubernetes/${PROJECT_NAME}.yaml'
sed -i "s@image\: $IMAGE_NAME\:.*@image\: $IMAGE_NAME:$TARGET_IMAGE_VERSION@g" kubernetes/${PROJECT_NAME}.yaml

# Add a remote that is the HTTPS version, should be possible to push to it using a personal access token + HTTPS
echo -e "\nCMD: git remote add gitlab ..."
git remote add gitlab https://gitlab-ci-token:${CI_PERSONAL_ACCESS_TOKEN}@${PROJECT_INFRA_REPO_LOCATION}
if [ $? -ne 0 ]; then echo "ERROR: Git remote add failed"; exit 1; fi

# commit the change
echo -e "\nCMD: git commit..."
git commit -am "AUTO: Deploying to (${CI_ENVIRONMENT_NAME}) => ${PROJECT_NAME} ${IMAGE_NAME} -> v${TARGET_IMAGE_VERSION}"
if [ $? -ne 0 ]; then echo "ERROR: Git commit failed"; exit 1; fi

# remove the existing deployment tag if present
echo -e "\nCMD: git push --delete gitlab ${PROJECT_NAME}-${CI_ENVIRONMENT_NAME}"
git tag -d ${PROJECT_NAME}-${CI_ENVIRONMENT_NAME}
git push --delete gitlab ${PROJECT_NAME}-${CI_ENVIRONMENT_NAME}

# tag the change
echo -e "\nCMD: git tag ${PROJECT_NAME}-${CI_ENVIRONMENT_NAME}"
git tag ${PROJECT_NAME}-${CI_ENVIRONMENT_NAME}
if [ $? -ne 0 ]; then echo "ERROR: Git tag failed"; exit 1; fi

# push the new commit and new tag, the runner wil be on a detached head
echo -e "\nCMD: git push gitlab HEAD:master --tags"
git push gitlab HEAD:master --tags
if [ $? -ne 0 ]; then echo "ERROR: Version change notification commit  push failed"; exit 1; fi

A bit more complicated, but pretty worth it, I do an automated git push of a new tag with xthe PROJECT_NAME and CI_ENVIRONMENT so that when releases happen, I can tell where in the commit history for the <project>-infra branch they happened. This script was fun to write (and by “fun” I mean challenging/riddled with subtle bugs from my lack of git mastery). It’s also pretty janky because I do a sed replace instead of doing something like kubectl patch. I did it that way because I wasn’t sure how to use kubectl to patch an existing yaml file and being sure whether bad/hard-for-humans-to-read gibberish would come out. I can’t say I even tried very hard, I just reached for sed.

There are at least a few things that could certainly be improved:

• The current setup requires that a project pass along it’s own name, and more information than might be necessary.
• If the scripts are pretty generic, maybe I should be using them at a higher level (not <project>-infra

For now though, it works, and that makes me pretty happy.

TL;DR code dump

So I’m not sure if this is cheating according to the tldr principles, but here’s a code/mind dump of everything I needed to make this work:

Kubernetes resources (things your cluster needs)

• Namespace for your project
• Service account for Gitlab to use

Gitlab resources (things you have to make in gitlab)

.gitlab-ci.yml in repo

Here’s the gitlab config in the actual repo with the relevant steps only (most stages/other config is removed for brevity)

image: alpine:3.6

variables:
  # These vars help gitlab caching by pulling stack stuff into the project directory to enable re-use across builds
  # my builds are still around 10-20 mins though :( (pulling stuff from cache also takes a bit of time, because there is just so much)
  STACK_ROOT: "${CI_PROJECT_DIR}/.stack"
  STACK_OPTS: "--system-ghc"

cache:
  - .stack
  - .stack-work
  - target

publish:
  stage: publish
  image: docker
  services:
    - docker:dind
  # Only publish when a release tag (vX.X.X) is pushed
  only:
    - /v[0-9|\.]+/
  except:
    - branches
  script:
    # build the project
    - apk add --update alpine-sdk git linux-headers ca-certificates gmp-dev zlib-dev ghc make curl
    - curl -sSL https://get.haskellstack.org/ | sh
    - make setup build
    # publish
    - docker login -u gitlab-ci-token --password $CI_BUILD_TOKEN registry.gitlab.com
    - make image publish

deploy:
  stage: deploy
  image: alpine:3.6
  # Only deploy when a release tag (vX.X.X) is pushed
  only:
    - /v[0-9|\.]+/
  except:
    - branches
  script:
    - apk --no-cache add curl ca-certificates
    # Get the current version out of the Makefile
    - export DEPLOY_VERSION=`awk -F= '/DEPLOY_VERSION=(.*)/{print $2}' Makefile`
    - echo -e "Triggering deploy for version [$DEPLOY_VERSION]..."
    - "curl -X POST -F token=$INFRA_PIPELINE_TOKEN -F \"ref=master\" -F \"variables[TARGET_IMAGE_VERSION]=$DEPLOY_VERSION\" -F \"variables[PROJECT_NAME]=<project name>\" -F \"variables[POD_NAME]=<k8s pod name>\" -F \"variables[IMAGE_NAME]=<fully qualified image name>\" -F \"variables[PROJECT_INFRA_REPO_LOCATION]=<path to infra repo for this project>\" \"$INFRA_PIPELINE_TRIGGER_URL\""

.gitlab-ci.yml in -infra repo

This script is much shorter so there’s less taken out.

image: alpine:3.6

stages:
  - deploy

cache:
  paths:
    - binary-cache

before_script:
  - apk add --no-cache curl ca-certificates
  # create the binary-cache folder if not present
  - test -d binary-cache || mkdir binary-cache
  # install kubectl binary into binary-cache if not already present
  - test -f binary-cache/kubectl || curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.7.2/bin/linux/amd64/kubectl
  - test -f binary-cache/kubectl || mv kubectl binary-cache/kubectl
  # install kubectl binary into /usr/local/bin if not already present
  - test -f /usr/local/bin/kubectl || install -m 770 binary-cache/kubectl /usr/local/bin

deploy_production:
  stage: deploy
  environment:
    name: production
  only:
    - triggers
  script:
    - apk add --update git
    - scripts/rollout-update
    - scripts/commit-tag-push

rollout-update script in -infra repo

See earlier in the post

commit-tag-push script in -infra repo

See earlier in the post

Wrapping up

This was a pretty big addition to the flow but it was nice to work through it. Halfway through trying to figure it out, I got a little delirious and the process wasn’t so clean, but it was rewarding to finally get it working. The result is somewhat convoluted, but given my choices/habits that was pretty much self-inflicted pain.

I do have one dirty secret thought – the whole process isn’t 100% automatic. To get the last bit in place, I needed to add a CI stage to automatically tag the master branch in my <project> repository, when a new version was added. How would you know if a version was new? Easy – just check if there’s a tag for the version already (something like git show-ref --tags | grep refs/tags/v$DETECTED_VERSION | wc -l will get you most of the way there, I’ll write a post on that fun hack later). Once everything is said and done, the process becomes this:

1. Write lots of code (features/bugfixes/new bugs)
2. In one of the commits, update the project version (probably a commit by itself)
3. Once that commit hits master, the CI stage which automatically tags master with a version if it doesn’t exist already kicks in, an master is tagged with v$NEW_VERSION.
4. Tagging master with v<some version> triggers the CI docker build and publish steps and a new docker container is built and pushed to the project’s container registry
5. Tagging master with v<some version> also triggers a faux deploy stage that does nothing but trigger the <project>-infra repo’s deploy stage using a cURL to Gitlab’s API
6. The <project>-infra’s deploy pipeline triggers, running the necessary kubectl commands to deploy the project (information passed over HTTPS via query params in 5 helps inform the script)
7. Upon successful rollout, the <project>-infra’s deploy pipeline updates the Kubernetes resource config, commits the update, and updates/creates the <project>-<environment> tag on master to the new commit (which contains the changed k8s resource config)

This could have been lots simpler – you could just make the infra-related details and kubernetes resource config available to the project itself – but I chose this setup because I do separate that information, and some repos/services I use in multiple projects (for example, a standalone login service.

This setup is somewhat fragile, there are lots of things that could go wrong:

• Multiple repos are involved (<project> and <project>-infra)
• Gitlab needs to be up/functioning
• Network access at any given step

However, I appreciate that in the happy path case, I have a properly functioning CI/CD infrastructure. I’ll watch it very closely especially at the beginning (and considering that I basically deploy straight to production), but there are two primary reasons this amount of fragility doesn’t both me:

• Gitlab offers pleasant-to-use instrospection ability into how CI/CD stages run (and the output of those stages)
• I’ve written most scripts and utilities to be fail fast and loudly to ease debug

Thanks for reading – I hope you’ve enjoyed this adventure, as I know I have. This very manual/hands-on tour through integrating Kubernetes with Gitlab and doing automated deployment was a blast.