DDG search drop in powered by ddg.patdryburgh.com
tl;dr - I applied a few patterns I’ve used on other projects to a Gitlab CI-powered rust project to achieve <2min builds. Basically just caching at different layers – caching via the docker image builder pattern at the docker level, aggressive caching with Gitlab CI at the CI runner level, also one more step of combining some build steps (probably unnecessarily).
I recently became a proud rustacean, which is what developers who use the programming language rust call themselves. I’ve been watching rust for a while, and I’ve been itching to find something to use it for and I finally found something! The project I found to start using rust for was postmgr, a supervisor daemon for postfix.
In this day and age, it’s pretty uncommon to even run your own mail server whether for personal or application emails. For those that do though, the best choice has been postfix for a long time and has no signs of changing. Though the SMTP specification is actually ridiculously simple, there don’t actually seem to be many other choices. At the beginning I considered actually creating a competitor to postfix, and actually aiming for production usage, but that seemed like far too much work for too little upside (and way more liability). There’s gotta be a reason postfix has the most support. Very clearly, the even better choice would be to just use a service like AWS’s Simple Email Services (SES) or SendGrid to send emails – I wrote my Mailer component to support swapping out SMTP server addresses for this very eventuality. However, I really want to rely on my own infrastructure (or things I at least partially know how to run) as much as possible.
Up until now I was using a project called mailu, and I considered moving to mailcow, but the problem I ran into was that both were too hard to get (and keep) running in my Kubernetes environment. When these solutions were being developed, I think Docker Compose was the only choice, as they all have documentation for getting started with Docker Compose but really nothing for Kubernetes. I contributed working Kubernetes resource definitions for an earlier version of Mailu (and also updated the ingress controller later), but after trying to get an updated version of mailu up and running, I got frustrated when it was hard to configure, and made a lot of assumptions that I didn’t think were reasonable. So I decided to write something myself that minimally manages postfix, and makes it possible to configure it. I’ll be learning about rust as I go so development will likely be slower, but I don’t mind, since I think rust is a language worth investing in for the long term. I’ve already received some great help from rust’s awesome community, so things are looking bright.
This post is about when I got my new project set up with Gitlab CI and applied some build optimizations I’ve applied in the past to get it to build just a bit faster.
So one great thing about rust is that the builds are already pretty fast – even including downloading of dependencies. My project isn’t doing much yet, but that means even the naive approach was pretty quick – 4mins 30 seconds overall (~2:15 for build and test steps that both did a full compile).
Here’s what the output looked like:
Running with gitlab-runner 10.7.0 (7c273476)
on docker-auto-scale 72989761
Using Docker executor with image rust:1.23.0 ...
Pulling docker image rust:1.23.0 ...
Using docker image sha256:c694858c0c3caf4e51af694f9e81266418c4be333aeea1dc1f3b6e3cd83d02dc for rust:1.23.0 ...
Running on runner-72989761-project-6275296-concurrent-0 via runner-72989761-srm-1526002711-4193c7ae...
Cloning repository...
Cloning into '/builds/postmgr/postmgr'...
Checking out 88ec2789 as master...
Skipping Git submodules setup
$ make build
cargo build
Updating registry `https://github.com/rust-lang/crates.io-index`
Downloading log v0.4.1
Downloading failure v0.1.1
Downloading serde_derive v1.0.51
Downloading toml v0.4.6
Downloading serde v1.0.51
Downloading simple_logger v0.5.0
Downloading cfg-if v0.1.3
Downloading failure_derive v0.1.1
Downloading backtrace v0.3.7
Downloading syn v0.11.11
Downloading synstructure v0.6.1
Downloading quote v0.3.15
Downloading synom v0.11.3
Downloading unicode-xid v0.0.4
Downloading rustc-demangle v0.1.8
Downloading syn v0.13.7
Downloading proc-macro2 v0.3.8
Downloading quote v0.5.2
Downloading unicode-xid v0.1.0
Downloading time v0.1.39
Downloading libc v0.2.40
Downloading backtrace-sys v0.1.16
Downloading cc v1.0.15
Compiling unicode-xid v0.1.0
Compiling unicode-xid v0.0.4
Compiling cc v1.0.15
Compiling cfg-if v0.1.3
Compiling quote v0.3.15
Compiling serde v1.0.51
Compiling libc v0.2.40
Compiling rustc-demangle v0.1.8
Compiling proc-macro2 v0.3.8
Compiling synom v0.11.3
Compiling backtrace-sys v0.1.16
Compiling log v0.4.1
Compiling toml v0.4.6
Compiling time v0.1.39
Compiling quote v0.5.2
Compiling syn v0.11.11
Compiling simple_logger v0.5.0
Compiling syn v0.13.7
Compiling synstructure v0.6.1
Compiling serde_derive v1.0.51
Compiling failure_derive v0.1.1
Compiling backtrace v0.3.7
Compiling failure v0.1.1
Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
Finished dev [unoptimized + debuginfo] target(s) in 68.38 secs
Job succeeded
Of course, it’s not a good idea to do this too naively, wouldn’t want to put undue stress on other people’s servers.
A pattern I’ve used a lot for projects (most notably my other haskell project which I’ve written about) is having a builder image that basically caches parts of the build I don’t want to replace. As I usually note, the reason I use the builder pattern instead of Docker multi-stage builds is that I can specify the builder container as the starting container inside Gitlab’s .gitlab-ci.yml. The idea is simple, remove the downloading by encoding it straight into the starting docker image layer, and make sure the cache @ CARGO_HOME gets used.
To make the builder image, you can just build some older version of the code in the container, then write over the code directory by COPYing slightly newer code in and doing the build again. This removed the downloading and some of the compiling but not all of it:
Running with gitlab-runner 10.7.0 (7c273476)
on docker-auto-scale e11ae361
Using Docker executor with image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Pulling docker image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Using docker image sha256:679be6ae2f78c188ce6f268adf17ae4f62f683554f467b7af84d62843613c979 for registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Running on runner-e11ae361-project-6275296-concurrent-0 via runner-e11ae361-srm-1526006018-5694816a...
Cloning repository...
Cloning into '/builds/postmgr/postmgr'...
Checking out b16fbb8e as add-builder-image...
Skipping Git submodules setup
$ make build
cargo build
Compiling unicode-xid v0.1.0
Compiling unicode-xid v0.0.4
Compiling rustc-demangle v0.1.8
Compiling quote v0.3.15
Compiling serde v1.0.51
Compiling libc v0.2.40
Compiling cfg-if v0.1.3
Compiling cc v1.0.15
Compiling synom v0.11.3
Compiling proc-macro2 v0.3.8
Compiling time v0.1.39
Compiling log v0.4.1
Compiling backtrace-sys v0.1.16
Compiling syn v0.11.11
Compiling quote v0.5.2
Compiling toml v0.4.6
Compiling simple_logger v0.5.0
Compiling syn v0.13.7
Compiling synstructure v0.6.1
Compiling failure_derive v0.1.1
Compiling backtrace v0.3.7
Compiling serde_derive v1.0.51
Compiling failure v0.1.1
Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
Finished dev [unoptimized + debuginfo] target(s) in 54.5 secs
Job succeeded
Here’s what the makefile looks like for the build step
This build took much less time, but still wasn’t as fast as I wanted, as I expected to see basically no compiling at all, since it’s the exact same code.
Clearly Gitlab CI wasn’t caching the build artifacts I expected it to – to fix this, I needed to add target (the build output folder that rust uses) to the cached paths in .gitlab-ci.yml. cargo is smart enough to recompile when things are different so it’s not an issue.
Here’s what a run of the “test” step looks like after build results were cached in a preceeding “build” step:
Running with gitlab-runner 10.7.0 (7c273476)
on docker-auto-scale e11ae361
Using Docker executor with image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Pulling docker image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Using docker image sha256:679be6ae2f78c188ce6f268adf17ae4f62f683554f467b7af84d62843613c979 for registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Running on runner-e11ae361-project-6275296-concurrent-0 via runner-e11ae361-srm-1526014605-dd91df99...
Cloning repository...
Cloning into '/builds/postmgr/postmgr'...
Checking out 1d826586 as add-builder-image...
Skipping Git submodules setup
Checking cache for default...
Downloading cache.zip from http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
Successfully extracted cache
$ make test
cargo test
Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
Finished dev [unoptimized + debuginfo] target(s) in 3.45 secs
Running target/debug/deps/postmgr-d48865bdf56a2c09
running 1 test
test config::test::errors_on_empty_files ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Creating cache default...
target: found 201 matching files
Uploading cache.zip to http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
Created cache
Job succeeded
Perfect, no extra/unexpected compilation, just the test running.
I also ran the build from the Gitlab CI UI and checked the output:
Running with gitlab-runner 10.7.0 (7c273476)
on docker-auto-scale e11ae361
Using Docker executor with image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Pulling docker image registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Using docker image sha256:679be6ae2f78c188ce6f268adf17ae4f62f683554f467b7af84d62843613c979 for registry.gitlab.com/postmgr/postmgr/builder:0.0.1 ...
Running on runner-e11ae361-project-6275296-concurrent-0 via runner-e11ae361-srm-1526015056-6713c3c2...
Cloning repository...
Cloning into '/builds/postmgr/postmgr'...
Checking out 1d826586 as add-builder-image...
Skipping Git submodules setup
Checking cache for default...
Downloading cache.zip from http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
Successfully extracted cache
$ make build
cargo build
Compiling postmgr v0.0.1 (file:///builds/postmgr/postmgr)
Finished dev [unoptimized + debuginfo] target(s) in 3.14 secs
Creating cache default...
target: found 201 matching files
Uploading cache.zip to http://runners-cache-5-internal.gitlab.com:444/runner/project/6275296/default
Created cache
Job succeeded
The build step ran this time in 1 minute 12 seconds, with the total CI run being 2 minutes 22 seconds. The CI build time has been halved, which is a great improvement. A few things are coming together to make this possible:
target folder.I figured I could do even better – why not combine the build and test steps? I convinced myself that there wasn’t much value in only doing the build without testing it (if the build passed), and if the build fails we’d know anyway. The longer running tests (integraiton/e2e tests) could still be run in a separate stage so they can be parallelized.
Adding this over-optimization reduces the CI build time from 2:22 to 1:36 (caveat: job was run on a shared runner that had pulled the builder image at least once before).
This could get even faster if I used beefier runners than the Gitlab shared runners (it would also ensure that the docker image you want was cached), and it was tempting to go for <1m builds but for now I’m happy with what I have – <2 minute build+test runs are prety good.
With a few quick and easy hacks, a build time of of ~4:30 went down to 1:36! I’m pretty happy with the faster builds, now I can get on to actually writing some more rust code!