+

tl;dr - Gitlab deploy tokens in a Kubernetes deployment don’t work using the normal k8s private registry documentation instructions. This post lays out the workaround/hack I used the last time it came up to save people some time. Skim through for the process and to the end for the k8s YAML.

One thing I just recently (literally a few moments ago) found myself doing was trying to figure out deploy tokens for a new project that I’m rolling out. I use <project>-infra repositories to manage deployment related scripts/information, these days mostly Kuberentes (“k8s”) YAML resource config files and secrets, and in the past some ansible stuff. I use GNU Make to orchestrate deployments (I call this the makeinfra pattern and it’s probably a terrible idea but I just haven’t done enough with make to know why).

When I sat down to deploy this new thing which is basically a reconfiguration of a bunch of projects I’d already made (but made generically so they could be reused), I found that after largely replicating most of the infrastructure code and configuration I needed I was dreading the generation of a new Secret that Kubernetes would use to pull the image from my private Gitlab registry. Gitlab is awesome (if you didn’t know it had private Docker registries for free, now you know) on this front in that it provides a deploy token feature, but in the past I’d had problems converting the simple user+password combination for gitlab logins to the form Kubernetes expected, which is an “image pull secret”.

There have been some changes in the way image pull secrets are stored/represented in kubernetes, as well as changes in how docker registries interoperate and it’s made it more than trivial to help kubernetes figure out how to properly authenticate itself so it can pull an image when a Pod is starting.

The Problem

The reason I felt a bit of pain when thinking about how to go about this was because the last time I followed the k8s private registry documentation, it didn’t work. Something about either Gitlab’s docker registry version or the kubernetes.io/dockercfgjson type versus type: kubernetes.io/dockercfg meant that pods using the secrets created by following the documentation didn’t work properly, and I only managed to get it to work by basically doing a docker login on my system, plucking the results from ~/.docker/config.json, and base64 encoding the format to what it should have been myself (the newer dockercfgjson secret type expects an auths top level key and the older dockercfg format is keyed by registry URL or something like that).

Just for my own sanity I tried doing everything again when writing this post, meaning I did the create command similar to how it’s written in the documentation:

$ kubectl create secret docker-registry gitlab --docker-server=registry.gitlab.com --docker-username=gitlab+deploy-token-xxxxx --docker-password=itsasecretbuddychill --docker-email=alsoasecret@example.com -n project
secret "gitlab" created

So as you can see a Secret is being created here of the type docker-registry, which is named gitlab, in a Namespace called project. The pods wouldn’t start wouldn’t start due to 403s from Gitlab’s side:

Events:
Type     Reason     Age                From                                    Message
----     ------     ----               ----                                    -------
Normal   Scheduled  32s                default-scheduler                       Successfully assigned project/project-78bf455b95-l66j9 to ubuntu-1804-bionic-64-minimal
Normal   BackOff    29s                kubelet, ubuntu-1804-bionic-64-minimal  Back-off pulling image "registry.gitlab.com/mrman/project/project:1.0.0"
Warning  Failed     29s                kubelet, ubuntu-1804-bionic-64-minimal  Error: ImagePullBackOff
Normal   Pulling    14s (x2 over 30s)  kubelet, ubuntu-1804-bionic-64-minimal  pulling image "registry.gitlab.com/mrman/project/project:1.0.0"
Warning  Failed     13s (x2 over 29s)  kubelet, ubuntu-1804-bionic-64-minimal  Failed to pull image "registry.gitlab.com/mrman/project/project:1.0.0": rpc error: code = Unknown desc = failed to resolve image "registry.gitlab.com/mrman/project/project:1.0.0": failed to fetch anonymous token: unexpected status: 403 Forbidden
  Warning  Failed     13s (x2 over 29s)  kubelet, ubuntu-1804-bionic-64-minimal  Error: ErrImagePull

I’m not sure exactly what causes this – I think Docker’s more recent registries have changed how they work, it’s entirely possible that I just have to change registry.gitlab.com to registry.gitlab.com/v2 or registry-v2.gitlab.com or something and fix the issue, but, I worked around this last time and so this time I wanted to write down how I did, just in case someone else runs into this and is at their wit’s end with either [Gitlab][gitlab] (which I love BTW, I’ll fight you about it) and/or Kubernetes.

At some point someone should definitely look into the actual issue though. Someone… At some point…

The workaround

Here’s the process that I used to make the pull secret that I can use for deployments:

Step 1: Get a deploy token from Gitlab

The relevant screen is @ Settings > Repository, and it looks something like this (UI might have changed since this post):

You’re probably going to want to create a token that can do nothing other than read from the repository. After you create a deploy token, you’ll be provided some randomly generated credentials (a username and password), and you’re going to want to write those down or save them somewhere safe.

Step 2: Backup your current docker credentials

Since you probably don’t want to clobber whatever credentials docker is using on your system right now, you should make a backup:

# cp ~/.docker/config.json ~/.docker/config.json.bak

Step 3: Do a new docker login and get the deploy token credentials

Now, let’s do a docker with the credentials you got from Step 1 – we’re pretending like we’re the robotic user (your CI system) that the deploy token was made for:

$ docker login registry.gitlab.com --username=gitlab+deploy-token-xxxxx --password=itsasecretbuddy
... messages ...

Login Succeeded

So now you should be able to find your credentials @ ~/.docker/config.json.

Step 4: Re-format & Base64 encode the auth information

Here’s what you should see in ~/.docker/config.json:

{
    "auths": {
        "registry.gitlab.com": {
            "auth": "<b64 encoded stuff>"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/18.06.1-ce (linux)"
    }
}

Here’s the crux of the workaround – if we reformat this configuration slightly (mostly just removing the top level auths key) it works as a pre-installable (if you so choose) Kubernetes Secret. Here’s the format we actually want:

{"registry.gitlab.com":{"username":"gitlab+deploy-token-xxxxx","password":"itsasecretbuddy","email":"you@somewhere.com","auth":"< B64 encoded chunk >"}}

So you can basically fill in the auth value, echo it out, and base64 encode it:

$ echo '{....}' | base64 | xclip -selection clipboard # or pbcopy?

Make sure to use single quotes, as it’s JSON and double quotes are present – once you have it encoded, we can stuff this into a Kubernetes secret (and remove the spaces)!

Step 5: Make a (pre-) installable Kubernetes Secret

Here’s the format for a secret that could be kubectl apply -f’d to a cluster:

---
apiVersion: v1
kind: Secret
type: kubernetes.io/dockercfg
metadata:
  name: gitlab
  namespace: <project>
data:
  .dockercfg: <b64 encoded long string from step 4>

I remove the spaces in the long b64 encoded, so make sure to do that. Once I have this resource configuration, I actually save it inside the repository (in a git-crypt protected folder called secrets), and use it whenever you do deploys. Also, my install of the secret is just a little bit verbose to allow for failure:

gitlab-secret:
    $(KUBECTL) get secret gitlab -n project || $(KUBECTL) apply -f ../secrets/kubernetes/gitlab.secret.yaml

Step 6: Restore your regular docker creds

If all is well you can restore the creds you backed up:

$ cp ~/.docker/config.json.bak ~/.docker/config.json

WARNING: This is a workaround/hack, you probably shouldn’t script/automate this

While it’s tempting to make a script for this (I wanted to just have that be the post), it’s probably not a good idea. Though the official k8s process is broken at this moment with Gitlab’s official registry endpoint, it will likely not be broken forever, and you probably don’t want to obscure another thing that should work.

So does that mean this should just be painful forever? I hope not, but I don’t really know what should be done about it at this point. Maybe it’s fine to script it – YMMV.

Wrapup

There’s lots of ways this process could be made better, or I could go check and figure out what’s actually wrong here, but rather than do all that I just wanted to put this article out so the information could at least be out there. I will make sure to update this article if/when I know more about why this work around is currently necessary or what I was doing wrong.

Happy hacking!