tl;dr - cert-manager
and moved from Bazel to Make (i.e. Makefile
) based build in 1.8.0. I’m into Makefiles (+ kustomize
) for builds and deploys so I’m taking an undeserved victory lap.
Recently I’ve been upgrading some of the critical services in my cluster and like everyone else, cert-manager
is one of the best parts of my cluster.
The cert-manager
maintainers and contributors have saved what I can only imagine is millions of person years at this point through great effort, silently and without fanfare. Shout out to Venafi as well for acquiring Jetstack and being great stewards of cert-manager
so far.
Anyway, as I was reading lots of release notes (I was quite a few versions back), I noticed that v1.8.0 included a move to Make from Bazel.
The reasoning is reproduced below (thanks for documenting your thoughts, cert-manager
maintainers/contributors!):
From Bazel to Make
A common theme when someone tries to make a change to cert-manager for the first time is that they ask for help with navigating Bazel, which cert-manager used as its build tool. Helping people with Bazel isn’t easy; it’s an incredibly powerful tool, but that power also brings a lot of complications which can seriously get in the way of being able to make even simple changes to the code base. Even developers who are familiar with contributing to open source projects in Go can find it daunting to make changes thanks to Bazel.
The problem isn’t limited to open-source contributors; many of cert-manager’s maintainers also struggle with configuring and changing Bazel, too.
cert-manager 1.8 is the first release which is built and tested using a newly written make-based build system. We believe that this new build system should make it much simpler to understand and change the commands which are being run behind the scenes to build and test cert-manager. In time, we’ll fully document the new build system, ensure it’s at full feature-parity with Bazel and then remove all references to Bazel across the codebase.
A neat side effect of this change is that our build times have significantly improved. Bazel took around 14 minutes to build every cert-manager artifact for every platform during a release, while the new make build system can do the same (and more) in under 5 minutes.
If you’re a regular reader, you know I’m into Makefile
s + kustomize
for deploys to k8s, so I was happy to see this!
make
and Makefile
s are underrated tooling for projects today (whether development or infra), and I’m glad to see them used more! Now I’ll just patiently wait for them to remove Helm… Just kidding – the important drawback with kustomize
is that they don’t do variable substitution (and don’t plan to).
For variable substitution, I use envsubst
in a pinch, but most of the time I make sure I build my apps in a way that doesn’t require variable substitution at the resource level!.
One concrete example of how you might remove the need for variable templating is by using secrets (with the same name for each environment, but differing content) – your different evnironments should be in different namespaces, so there should be no risk of mixing!
Anyway, I wanted to take a victory lap since I see this as validation of my rambling – Hopefully more people move to Make across projects.
This post does not mean to disparage Bazel or imply that Bazel is bad in any way! I don’t even mean this post to be clickbaity or provocative which I do from time to time.
Bazel is awesome, Make is awesome – nothing to see there.
Makefile
tricksWhile you’re here, here’s some cool make
tips and tricks I’ve honed over the last few projects.
AKA the poor man’s configure
– here’s how one might check for and ensure some ENV vars and tools are present:
.PHONY: don't forget to put your non-file/folder based makefile targets here!
DOCKER ?= docker
KUBECTL ?= kubectl
BASE64 ?= base64
ENVSUBST ?= envsubst
PROJECT_NAME ?= waaard
ENVIRONMENT ?= staging
K8S_NAMESPACE ?= $(PROJECT_NAME)-$(ENVIRONMENT)
all: secrets registry-login deploy
check-ENV-ENVIRONMENT:
ifeq (,$(ENVIRONMENT))
$(error "ENVIRONMENT not specified")
endif
check-tool-base64:
ifeq (,$(shell which $(BASE64)))
$(error "base64 binary must be installed")
endif
clean: clean-generated-secrets
docker
credentialsHere’s how you might do per-project (folder) docker
credentials (nice and easy per-project creds for use with docker login
, docker push
, etc):
## EXAMPLE: Per-project (folder) docker credentials
## (this is best used with direnv and git-crypt!)
CONTAINER_REGISTRY ?= registry.gitlab.com
RSS_API_REGISTRY_DOCKER_SECRET_FOLDER ?= ./secrets/docker
RSS_API_REGISTRY_PASSWORD_SECRET_PATH ?= ./secrets/gitlab/rss-api/deploy-tokens/registry-password.secret
RSS_API_REGISTRY_USERNAME_SECRET_PATH ?= ./secrets/gitlab/rss-api/deploy-tokens/registry-username.secret
registry-folders:
@mkdir -p $(RSS_API_REGISTRY_DOCKER_SECRET_FOLDER)
registry-login: secret-folders registry-folders registry-rss-api-login
registry-rss-api-login:
@if [ ! -f "$(RSS_API_REGISTRY_USERNAME_SECRET_PATH)" ] ; then \
echo -e "\n[info] Please enter a username for pulling the rss-api docker container:"; \
read REGISTRY_USERNAME && \
echo "[info] registry username to => [$$REGISTRY_USERNAME]" && \
echo "[info] Writing registry username to relevant secret files..." && \
echo $$REGISTRY_USERNAME | tr -d '[:space:]' > $(RSS_API_REGISTRY_USERNAME_SECRET_PATH); \
fi
@if [ ! -f "$(RSS_API_REGISTRY_PASSWORD_SECRET_PATH)" ] ; then \
echo -e "\n[info] Please enter a password for pulling the rss-api docker container:"; \
read REGISTRY_PASSWORD && \
echo "[info] registry password to => [$$REGISTRY_PASSWORD]" && \
echo "[info] Writing registry password to relevant secret files..." && \
echo $$REGISTRY_PASSWORD | tr -d '[:space:]' > $(RSS_API_REGISTRY_PASSWORD_SECRET_PATH); \
fi
@if [ -f "$(RSS_API_REGISTRY_DOCKER_JSON_PATH)" ] ; then \
echo "[info] file @ [$(RSS_API_REGISTRY_DOCKER_JSON_PATH)] already exists, skipping..."; \
else \
echo "\n[info] logging into docker..."; \
export DOCKER_CONFIG=$(RSS_API_REGISTRY_DOCKER_SECRET_FOLDER) && \
export DOCKER_USERNAME=$$(cat $(RSS_API_REGISTRY_USERNAME_SECRET_PATH)) && \
cat $(RSS_API_REGISTRY_PASSWORD_SECRET_PATH) \
| $(DOCKER) login $(CONTAINER_REGISTRY) -u $$DOCKER_USERNAME --password-stdin; \
cp $(RSS_API_REGISTRY_DOCKER_JSON_PATH) $(RSS_API_REGISTRY_DOCKER_SECRET_FOLDER)/.dockerconfigjson; \
fi
clean-registry-rss-api-login:
@rm ./$(RSS_API_REGISTRY_USERNAME_SECRET_PATH) || true
@rm ./$(RSS_API_REGISTRY_PASSWORD_SECRET_PATH) || true
@rm $(RSS_API_REGISTRY_DOCKER_JSON_PATH) || true
Here’s how you might generate some secrets that are OK to randomize:
## EXAMPLE: Generating random secrets
## Random secret generation!
secret-gen-random-alpha:
@cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1 | tr -d \\n
## Files that can be randomly generated
RANDOMIZED_SECRET_PATHS ?= secrets/$(ENVIRONMENT)/BASIC_AUTH_PASSWORD.secret \
secrets/pulumi/password-encryption.secret \
secrets/$(ENVIRONMENT)/db/users.readysetsaas.PASSWORD.secret \
$(API_POSTGRES_PASSWORD_SECRET_PATH) \
path/to/another/file.secret
clean-secrets-generated:
@echo -e "[info] Removing generated random-ok secrets..."
@for SECRET_PATH in $(RANDOMIZED_SECRET_PATHS) ; do \
[ ! -f $$SECRET_PATH ] && echo -e "[info] file @ [$$SECRET_PATH] does not exist" && continue; \
echo -e "[info] removing random secret @ [$$SECRET_PATH]..."; \
rm -rf $$SECRET_PATH; \
done
secrets-generated: secret-folders
@echo -e "[info] Generating known random-ok secrets..."
@for SECRET_PATH in $(RANDOMIZED_SECRET_PATHS) ; do \
[ -f $$SECRET_PATH ] && echo -e "[info] file @ [$$SECRET_PATH] already exists" && continue; \
echo -e "[info] Generating random secret @ [$$SECRET_PATH]..."; \
$(MAKE) -s --no-print-directory secret-gen-random-alpha > $$SECRET_PATH; \
done
Here’s how you might enable people to enter manual secrets while setting up the project (or a new environment) for the first time:
## Example: Manual secrets that a user needs to enter
secrets-manual: secret-folders check-environment
# Administrator email
@if [ ! -f $(SECRET_ADMIN_EMAIL_ADDRESS_PATH) ] ; then \
echo -e "\n[info] Please enter the enter the email address of the administrator:"; \
read ADMIN_EMAIL && \
echo "[info] Administrator email set to => [$$ADMIN_EMAIL]" && \
echo "[info] Writing email to relevant secret files..." && \
echo $$ADMIN_EMAIL | tr -d '[:space:]' > $(SECRET_ADMIN_EMAIL_ADDRESS_PATH) && \
echo $$ADMIN_EMAIL | tr -d '[:space:]' > secrets/$(ENVIRONMENT)/analytics/FATHOM_ROOT_USER_NAME.secret && \
echo $$ADMIN_EMAIL | tr -d '[:space:]' > secrets/$(ENVIRONMENT)/rss-api/ROOT_ADMIN_EMAIL_ADDRESS.secret; \
fi
These days the cool kids (me?) generate their DKIM secrets and feed them to SES instead of the other way around. Funnily enough, this is how it was done in the old days. Time is a flat circle, etc.
Here’s how you might generate some DKIM secrets.
## Example: generating DKIM secrets
dkim:
@echo -e "\n[info] generating DKIM keys for use with email..."
@if [ ! -f "$(DKIM_PRIVATE_KEY_PATH)" ] ; then \
echo "[info] DKIM private key not found @ [$(DKIM_PRIVATE_KEY_PATH)], creating new key..."; \
$(OPENSSL) genrsa -f4 -out $(DKIM_PRIVATE_KEY_PATH); \
fi
@echo -e "[info] private key @ [$(DKIM_PRIVATE_KEY_PATH)]"
@if [ ! -f "$(DKIM_PUBLIC_KEY_PATH)" ] ; then \
echo "[info] DKIM public key not found @ [$(DKIM_PUBLIC_KEY_PATH)], creating new key..."; \
$(OPENSSL) rsa -in $(DKIM_PRIVATE_KEY_PATH) -outform PEM -pubout -out $(DKIM_PUBLIC_KEY_PATH); \
fi
@echo -e "[info] public key @ [$(DKIM_PUBLIC_KEY_PATH)]"
That’s it for now! I should put out all these tricks someday (maybe compile them all in some sort of Makefile tip site?), but for now I’ll leave this for the curious out there.