Even faster rust builds in Gitlab CI

Getting even faster builds out of rust on Gitlab CI

vados

9 minute read

Rust logo

tl;dr - I apply a bunch of patterns I’ve adopted to make my Gitlab CI powered builds faster on a rust project I just started to achieve <2min builds.

I recently became a proud rustacean, which is what developers who use the programming language rust call themselves. I’ve been watching rust for a while, and I’ve been itching to find something to use it for and I finally found something! The project I found to start using rust for was postmgr, a supervisor daemon for postfix.

In this day and age, it’s pretty uncommon to even run your own mail server whether for personal or application emails. For those that do though, the best choice has been postfix for a long time and has no signs of changing. Though the SMTP specification is actually ridiculously simple, there don’t actually seem to be many other choices. At the beginning I considered actually creating a competitor to postfix, and actually aiming for production usage, but that seemed like far too much work for too little upside (and way more liability). There’s gotta be a reason postfix has the most support. Very clearly, the even better choice would be to just use a service like AWS’s Simple Email Services (SES) or SendGrid to send emails – I wrote my Mailer component to support swapping out SMTP server addresses for this very eventuality. However, I really want to rely on my own infrastructure (or things I at least partially know how to run) as much as possible.

Up until now I was using a project called mailu, and I considered moving to mailcow, but the problem I ran into was that both were too hard to get running in my Kubernetes environment. They both bit down super hard on trying to run tons of little pieces and it’s a nightmare to try and make the working kubernetes resource definitions. Also, when these solutions were being developed, I think [Docker Compose][docker-compose] was the only choice, as they all have documentation for getting started with Docker Compose but really nothing for Kubernetes. After trying to get a newer version of mailu up and running, I got frustrated when it was hard to configure, and made a lot of assumptions that I didn’t think were reasonable. So I decided to write something myself that minimally manages postfix, and makes it possible to configure it. I’ll be learning about rust as I go so development will likely be slower, but I don’t mind, since I think rust is a language worth investing in for the long term. I’ve already received some great help from rust’s awesome community, so things are looking bright.

This post is about when I got my new project set up with Gitlab CI and applied some build optimizations I’ve applied in the past to get it to build just a bit faster.

Builds in rust are already pretty fast

So one great thing about rust is that the builds are already pretty fast – even including downloading of dependencies. My project isn’t doing much yet, but that means even the naive approach was pretty quick – 4mins 30 seconds overall (~2:15 for build and test steps that both did a full compile).

Here’s what the output looked like:

      Running with gitlab-runner 10.7.0 (7c273476)
        on docker-auto-scale 72989761
      Using Docker executor with image rust:1.23.0 ...
      Pulling docker image rust:1.23.0 ...
      Using docker image sha256:c694858c0c3caf4e51af694f9e81266418c4be333aeea1dc1f3b6e3cd83d02dc for rust:1.23.0 ...
      Running on runner-72989761-project-6275296-concurrent-0 via runner-72989761-srm-1526002711-4193c7ae...
      Cloning repository...
      Cloning into '/builds/postmgr/postmgr'...
      Checking out 88ec2789 as master...
      Skipping Git submodules setup
      $ make build
      cargo build
          Updating registry `https://github.com/rust-lang/crates.io-index`
       Downloading log v0.4.1
       Downloading failure v0.1.1
       Downloading serde_derive v1.0.51
       Downloading toml v0.4.6
       Downloading serde v1.0.51
       Downloading simple_logger v0.5.0
       Downloading cfg-if v0.1.3
       Downloading failure_derive v0.1.1
       Downloading backtrace v0.3.7
       Downloading syn v0.11.11
       Downloading synstructure v0.6.1
       Downloading quote v0.3.15
       Downloading synom v0.11.3
       Downloading unicode-xid v0.0.4
       Downloading rustc-demangle v0.1.8
       Downloading syn v0.13.7
       Downloading proc-macro2 v0.3.8
       Downloading quote v0.5.2
       Downloading unicode-xid v0.1.0
       Downloading time v0.1.39
       Downloading libc v0.2.40
       Downloading backtrace-sys v0.1.16
       Downloading cc v1.0.15
         Compiling unicode-xid v0.1.0
         Compiling unicode-xid v0.0.4
         Compiling cc v1.0.15
         Compiling cfg-if v0.1.3
         Compiling quote v0.3.15
         Compiling serde v1.0.51
         Compiling libc v0.2.40
         Compiling rustc-demangle v0.1.8
         Compiling proc-macro2 v0.3.8
         Compiling synom v0.11.3
         Compiling backtrace-sys v0.1.16
         Compiling log v0.4.1
         Compiling toml v0.4.6
         Compiling time v0.1.39
         Compiling quote v0.5.2
         Compiling syn v0.11.11
         Compiling simple_logger v0.5.0
         Compiling syn v0.13.7
         Compiling synstructure v0.6.1
         Compiling serde_derive v1.0.51
         Compiling failure_derive v0.1.1
         Compiling backtrace v0.3.7
         Compiling failure v0.1.1
         Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
          Finished dev [unoptimized + debuginfo] target(s) in 68.38 secs
    Job succeeded

Of course, it’s not a good idea to do this too naively, wouldn’t want to put undue stress on other people’s servers.

Using the container builder pattern to pattern

A pattern I’ve used a lot for projects (most notably [my other haskell project which I’ve written about][haskell-builder-post]) is having a builder image that basically caches parts of the build I don’t want to replace. As I usually note, the reason I use the builder pattern instead of Docker multi-stage builds is that I can specify the builder container as the starting container inside Gitlab’s .gitlab-ci.yml. The idea is simple, remove the downloading by encoding it straight into the starting docker image layer, and make sure the cache @ CARGO_HOME gets used.

To make the builder image, you can just build some older version of the code in the container, then write over the code directory by COPYing slightly newer code in and doing the build again. This removed the downloading and some of the compiling but not all of it:

     Running with gitlab-runner 10.7.0 (7c273476)
       on docker-auto-scale e11ae361
     Using Docker executor with image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Pulling docker image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Using docker image sha256:679be6ae2f78c188ce6f268adf17ae4f62f683554f467b7af84d62843613c979 for registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Running on runner-e11ae361-project-6275296-concurrent-0 via runner-e11ae361-srm-1526006018-5694816a...
     Cloning repository...
     Cloning into '/builds/postmgr/postmgr'...
     Checking out b16fbb8e as add-builder-image...
     Skipping Git submodules setup
     $ make build
     cargo build
        Compiling unicode-xid v0.1.0
        Compiling unicode-xid v0.0.4
        Compiling rustc-demangle v0.1.8
        Compiling quote v0.3.15
        Compiling serde v1.0.51
        Compiling libc v0.2.40
        Compiling cfg-if v0.1.3
        Compiling cc v1.0.15
        Compiling synom v0.11.3
        Compiling proc-macro2 v0.3.8
        Compiling time v0.1.39
        Compiling log v0.4.1
        Compiling backtrace-sys v0.1.16
        Compiling syn v0.11.11
        Compiling quote v0.5.2
        Compiling toml v0.4.6
        Compiling simple_logger v0.5.0
        Compiling syn v0.13.7
        Compiling synstructure v0.6.1
        Compiling failure_derive v0.1.1
        Compiling backtrace v0.3.7
        Compiling serde_derive v1.0.51
        Compiling failure v0.1.1
        Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
         Finished dev [unoptimized + debuginfo] target(s) in 54.5 secs
     Job succeeded

Here’s what the makefile looks like for the build step

This build took much less time, but still wasn’t as fast as I wanted, as I expected to see basically no compiling at all, since it’s the exact same code.

Gitlab CI caching the build artifacts

Clearly Gitlab CI wasn’t caching the build artifacts I expected it to – to fix this, I needed to add target (the build output folder that rust uses) to the cached paths in .gitlab-ci.yml. cargo is smart enough to recompile when things are different so it’s not an issue.

Here’s what a run of the “test” step looks like after build results were cached in a preceeding “build” step:

     Running with gitlab-runner 10.7.0 (7c273476)
       on docker-auto-scale e11ae361
     Using Docker executor with image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Pulling docker image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Using docker image sha256:679be6ae2f78c188ce6f268adf17ae4f62f683554f467b7af84d62843613c979 for registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Running on runner-e11ae361-project-6275296-concurrent-0 via runner-e11ae361-srm-1526014605-dd91df99...
     Cloning repository...
     Cloning into '/builds/postmgr/postmgr'...
     Checking out 1d826586 as add-builder-image...
     Skipping Git submodules setup
     Checking cache for default...
     Downloading cache.zip from http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
     Successfully extracted cache
     $ make test
     cargo test
        Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
         Finished dev [unoptimized + debuginfo] target(s) in 3.45 secs
          Running target/debug/deps/postmgr-d48865bdf56a2c09

     running 1 test
     test config::test::errors_on_empty_files ... ok

     test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Creating cache default...
     target: found 201 matching files
     Uploading cache.zip to http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
     Created cache
     Job succeeded

Perfect, no extra/unexpected compilation, just the test running.

I also ran the build from the Gitlab CI UI and checked the output:

     Running with gitlab-runner 10.7.0 (7c273476)
       on docker-auto-scale e11ae361
     Using Docker executor with image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Pulling docker image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Using docker image sha256:679be6ae2f78c188ce6f268adf17ae4f62f683554f467b7af84d62843613c979 for registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
     Running on runner-e11ae361-project-6275296-concurrent-0 via runner-e11ae361-srm-1526015056-6713c3c2...
     Cloning repository...
     Cloning into '/builds/postmgr/postmgr'...
     Checking out 1d826586 as add-builder-image...
     Skipping Git submodules setup
     Checking cache for default...
     Downloading cache.zip from http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
     Successfully extracted cache
     $ make build
     cargo build
        Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
         Finished dev [unoptimized + debuginfo] target(s) in 3.14 secs
     Creating cache default...
     target: found 201 matching files
     Uploading cache.zip to http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
     Created cache
     Job succeeded

The build step ran this time in 1 minute 12 seconds, with the total CI run being 2 minutes 22 seconds. The CI build time has been halved, which is a great improvement. A few things are coming together to make this possible:

  • Fast docker image retrieval/caching at the gitlab runner (the gitlab runner I ran on had the image in cache which is why downloading it took less time I think)
  • In-image-layer caching of the downloaded deps, that happen to perfectly match, so new deps needed to be downloaded by cargo (this is only true if you builder image is relatively fresh, you can automate this)
  • Caching of compiled deps via gitlab CI caching of target folder.

Probably too far: combining build and test steps

I figured I could do even better – why not combine the build and test steps? I convinced myself that there wasn’t much value in only doing the build without testing it (if the build passed), and if the build fails we’d know anyway. The longer running tests (integraiton/e2e tests) could still be run in a separate stage so they can be parallelized.

Adding this over-optimization reduces the CI build time from 2:22 to 1:36 (caveat: job was run on a shared runner that had pulled the builder image at least once before).

This could get even faster if I used beefier runners than the Gitlab shared runners (it would also ensure that the docker image you want was cached), and it was tempting to go for <1m builds but for now I’m happy with what I have – <2 minute build+test runs are prety good.

Wrap Up

With a few quick and easy hacks, a build time of of ~4:30 went down to 1:36! I’m pretty happy with the faster builds, now I can get on to actually writing some more rust code!

Did you find this read beneficial? Send me questions/comments/clarifciations.
Want my expertise on your team/project? Send me interesting opportunities!