tl;dr - There are at least two ways to wait for Kubernetes resources you probably care about: kubectl wait for Pods, initContainers for everything else

One somewhat rarely talked about issue in Kubernetes land is how exactly people wait for stuff to happen. “Stateful workloads” can get hand-waved to using StatefulSets, and most intricate large deployable things (databases, etc) have Operators that you can use. Sometimes you just want to make sure a Service is up before you start a pod that depends on it. I first ran into this while doing my most recent set of tests on storage providers for k8s, IIRC while working on the LINSTOR setup – Turns out there are a bunch of ways to wait for various things and some are hackier than others.

For a certain size of problem, the “right” and “best” solution is probably the operator pattern – a Controller and (optionally) a CRD to really track and manage the aggregate system/group of components you’re trying to create. What I’d like to explore in this article is all the sizes below that – the little things you want to wait for.

Pods: kubectl wait

Turns out there’s a kubectl wait command! This gets the problem solved pretty nicely, if what you need to wait for is a Pod.

You can wait for a pod by name (hopefully not one that is named randomly as part of a Deployment):

$ k wait --for=condition=ready pod maddy-957455599-shj6m --timeout=60s
pod/maddy-957455599-shj6m condition met

This returns near instantly since that is a pod that is already running. You can also wait for one or more pods by label:

$ k wait --for=condition=ready pod -l app=blog --timeout=60s
pod/blog-866b9bf5f5-25zpn condition met
pod/listmonk-666f99cc55-k55zm condition met
... the command is stuck ...

Uh oh, what happened there?

Pitfall: Watch out for non-conforming (but not quite incorrect) pods

We’ve already hit our first pitfall – you have to be careful about your specificity and the possiblity for other pods to be in statuses that don’t match, as they could cause you to eat up the timeout. Here’s what it looks like if I set a lower timeout:

$ k wait --for=condition=ready pod -l app=blog --timeout=5s
pod/blog-866b9bf5f5-25zpn condition met
pod/listmonk-666f99cc55-k55zm condition met
pod/listmonk-pg-6cdbf4d7f6-tt8jf condition met
pod/maddy-957455599-shj6m condition met
timed out waiting for the condition on pods/listmonk-backups-27166200-gw8r5
timed out waiting for the condition on pods/listmonk-backups-27166920-46w8k
timed out waiting for the condition on pods/listmonk-backups-27167640-n97mm

Simply having some orphaned pods that were Completed rather than a recognized Ready condition unexpectedly caused a hangup. Here’s what the full listing looks like:

$ k get pods
NAME                              READY   STATUS      RESTARTS   AGE
blog-866b9bf5f5-25zpn             1/1     Running     0          12d
fathom-658c7cf686-z79jq           1/1     Running     1          71d
listmonk-666f99cc55-k55zm         1/1     Running     0          49d
listmonk-backups-27166200-gw8r5   0/1     Completed   0          24h
listmonk-backups-27166920-46w8k   0/1     Completed   0          12h
listmonk-backups-27167640-n97mm   0/1     Completed   0          52m
listmonk-pg-6cdbf4d7f6-tt8jf      1/1     Running     1          82d
maddy-957455599-shj6m             2/2     Running     0          49d

It’s easy to see with all the context in front of us – kubectl wait got stuck because I have some pods that ran and completed as the result of a CronJob (in this case, one that does backups!):

Services: initContainers and nc (netcat) to the rescue

Of course, Pods aren’t the only thing you might want to wait for – one thing I found myself wanting to wait for was a Service. This is a bit more difficult than Pods since the wait command does not work on Services.

There are at least two ways we might know that a Service is ready:

• try to connect to it on the port we care about
• wait for at least one Endpoint kubernetes object to be present

Well I’m a stickler for simplicity and if what I care about is a Service the most important thing is probably that it can take requests so I mostly go with option #1 above. You can get yourself a container that has nc installed (thanks to StackOverflow and the magic of bash):

      initContainers:
        # Wait for postgres to show up
        - name: wait-for-pg
          image: busybox:latest
          imagePullPolicy: IfNotPresent
          command: ['sh', '-c', 'until nc -vz ${POD_NAME}.${POD_NAMESPACE} 5432; do echo "Waiting for postgres..."; sleep 3; done;']
          env:
            - name: POD_NAME
              value: postgres
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

That particular hack came in handy when writing a Job that ran pgbench. Here’s another:

      initContainers:
      - name: message-bus-probe
        image: busybox:latest
        command: ['sh', '-c', 'until nc -vz nats 4222; do echo "Waiting for message bus..."; sleep 1; done;']

The example above is so neatly/well written that it must not have been me who wrote it, this must be where I got the idea. Anyway, you get the idea at this point.

Everything else: initContainers and kubectl

Just in case you haven’t put it together yet, I’ll leave this as an exercise for the reader – with a well-credentialed ServiceAccount (make sure to disable Service Acount automounting if the workload doesn’t need to talk to the API!), you can hit the Kubernetes API as much as you like to figure out when it’s time to go!

A fly in the ointment: What about DaemonSets ?

DaemonSets are unique because you may generally want to wait for a DaemonSet to be functional on the node the pod you’re checking from is on.

If you know that a certain number of replicas should be available then you can hack around the problem by waiting for them as I proposed in a PR that ultimately didn’t land:

       initContainers:
        ## Wait for at least one controller to be ready -- a controller must be running for satellite to register with
        - name: wait-for-controller
          image: bitnami/kubectl
          imagePullPolicy: IfNotPresent
          command:
            - /bin/sh
            - -exc
            - |
              n=0
              until [ $n -ge 30 ]; do
                  REPLICA_COUNT=$(kubectl get deploy/${CONTROLLER_DEPLOYMENT_NAME} -n ${CONTROLLER_NAMESPACE} -o template --template='{{ .status.availableReplicas }}')
                  if [ "${REPLICA_COUNT}" -gt "0" ] ; then
                      echo "[info] found ${REPLICA_COUNT} available replicas."
                      break
                  fi
                  echo -n "[info] waiting 10 seconds before trying again..."
                  sleep 10
              done              
          env:
            - name: CONTROLLER_DEPLOYMENT_NAME
              value: "{{ $fullName }}-controller"
            - name: CONTROLLER_NAMESPACE
              value: {{ .Release.Namespace }}

You could make this a bit more dynamic by the associated service account some more power to look around and inquire about your cluster and it’s workloads, but that quickly becomes a security hazard.

BONUS: More fun with initContainers – hacking your database initialization

Databases like Postgres and others are really easy to run in containers, but often there’s important configuration you need to do to set the database up. Usually there’s a right way (if the image is robust enough, putting configuration files in a certain place on disk), and then there’s this way:

      initContainers:
        # Init container that creates the API user if not present
        - name: setup-pg
          image: postgres:13.1-alpine
          command:
            - /bin/ash
            - -xc
            - |
              psql $POSTGRES_URL -c "CREATE ROLE $NEW_USER_NAME LOGIN PASSWORD '$NEW_USER_PASSWORD';" || true;
              psql $POSTGRES_URL -c "CREATE DATABASE $NEW_DB_NAME WITH OWNER = '$NEW_DB_OWNER_NAME';" || true;              
          env:
            # Information needed for the script (command)
            - name: NEW_DB_NAME
              value: api
            - name: NEW_DB_OWNER_NAME
              value: api
            - name: NEW_USER_NAME
              value: api
            - name: NEW_USER_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: api-db-secrets
                  key: users.api.PASSWORD.secret
            # For actually connecting to PG instance
            - name: PGHOST
              value: rss-postgres
            - name: PGUSER
              value: readysetsaas
            - name: PGDATABASE
              value: staging
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: rss-postgres-secrets
                  key: users.readysetsaas.PASSWORD.secret

The YAML above definitely works in a pinch and, but is definitely not the right way. To cover my own behind I’ll leave a link to what you should be doing (make sure those initialization scripts are idempotent!).

Update: Another tool for waiting, wait4x

Turns out there’s an awesome tool for exactly this kind of waiting & checking, called wait4x. It’s a great addition to your arsenal for those mornings where you just don’t want to write another shell script.