k8s Container Linux ignition with rkt and kube-router

Categories

This post still working for you?

It's been a while since this was posted. Hopefully the information in here is still useful to you (if it isn't please let me know!). If you want to get the new stuff as soon as it's out though, sign up to the mailing list below.

Join the Mailing list
Kubernetes logo + rkt logo + kube-router logo

I recently wrote a post about about switching back to container linux for my small Kubernetes cluster, in which I outlined everything i needed to do to get it up and running. Even more recently, I decided I wanted to go ahead and run “rktnetes” to try and take advantage of it’s annotation-powered stage1 selection, and figured I should post that up too for any fellow rkt enthusiasts!

Along with using rkt, I’ve also used a newer version of the CNI plugins, along with bringing kube-router back into the fold (over canal). There are also some small script improvements (for example a lock file to ensure the k8s-setup oneshot systemd service doesn’t run unnecessarily).

Without further ado, here’s the updated ignition.yaml:

# This config is meant to be consumed by the config transpiler, which will
# generate the corresponding Ignition config. Do not pass this config directly
# to instances of Container Linux.

# NOTE: This configuration is meant to work with Config Transpiler v0.8.0
# The spec is available at (https://github.com/coreos/container-linux-config-transpiler/blob/v0.8.0/doc/configuration.md)

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa <LONG KEY>
        - ssh-rsa <LONG KEY>

systemd:
  units:
    # Docker will be configured initially but we'll be using rkt exclusively and will disable it after containerd setup
    - name: docker.service
      enabled: true

    ####################
    # services for rkt #
    ####################
    - name: rkt-api.service
      enabled: true
      contents: |
        [Unit]
        Description=rkt api service
        Documentation=http://github.com/rkt/rkt
        After=network.target rkt-api-tcp.socket
        Requires=rkt-api-tcp.socket

        [Service]
        ExecStart=/usr/bin/rkt api-service

        [Install]
        WantedBy=multi-user.target        

    - name: rkt-api-tcp.socket
      enabled: true
      contents: |
        [Unit]
        Description=rkt api service socket
        PartOf=rkt-api.service

        [Socket]
        ListenStream=127.0.0.1:15441
        ListenStream=[::1]:15441
        Service=rkt-api.service
        BindIPv6Only=both

        [Install]
        WantedBy=sockets.target        

    - name: rkt-gc.service
      enabled: true
      contents: |
        [Unit]
        Description=Garbage Collection for rkt

        [Service]
        Environment=GRACE_PERIOD=24h
        Type=oneshot
        ExecStart=/usr/bin/rkt gc --grace-period=${GRACE_PERIOD}        

    - name: rkt-gc.timer
      enabled: true
      contents: |
        [Unit]
        Description=Periodic Garbage Collection for rkt

        [Timer]
        OnActiveSec=0s
        OnUnitActiveSec=12h

        [Install]
        WantedBy=multi-user.target        

    - name: rkt-metadata.service
      enabled: true
      contents: |
        [Unit]
        Description=rkt metadata service
        Documentation=http://github.com/rkt/rkt
        After=network.target rkt-metadata.socket
        Requires=rkt-metadata.socket

        [Service]
        ExecStart=/usr/bin/rkt metadata-service

        [Install]
        WantedBy=multi-user.target        

    - name: rkt-metadata.socket
      enabled: true
      contents: |
        [Unit]
        Description=rkt metadata service socket
        PartOf=rkt-metadata.service

        [Socket]
        ListenStream=/run/rkt/metadata-svc.sock
        SocketMode=0660
        SocketUser=root
        SocketGroup=root
        RemoveOnStop=true

        [Install]
        WantedBy=sockets.target        

    ################################
    # services for install scripts #
    ################################
    - name: k8s-install.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=k8s installation script
        Wants=network-online.target
        After=network.target network-online.target

        [Service]
        Type=oneshot
        ExecStart=/ignition/init/k8s/install.sh        

    - name: cni-install.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=cni plugin installation script
        Requires=k8s-install.service
        After=k8s-install.service

        [Service]
        Type=oneshot
        ExecStart=/ignition/init/cni/install.sh        

    - name: rkt-setup.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=rkt installation script
        Requires=cni-install.service
        After=cni-install.service

        [Service]
        Type=oneshot
        ExecStart=/ignition/init/rkt/setup.sh        

    - name: kubeadm-install.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=kubeadm installation script
        Requires=rkt-setup.service
        After=rkt-setup.service

        [Service]
        Type=oneshot
        Environment="PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin"
        ExecStart=/ignition/init/kubeadm/kubeadm-install.sh        

    - name: k8s-setup.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=kubernetes setup script
        Requires=kubeadm-install.service
        After=kubeadm-install.service

        [Service]
        Type=oneshot
        User=core
        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
        ExecStart=/ignition/init/k8s/setup.sh        

storage:
  filesystems:
    - mount:
        device: /dev/disk/by-label/ROOT
        format: xfs
        wipe_filesystem: true
        label: ROOT

  files:
    - path: /opt/bin/kubeadm
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubeadm
          verification:
            hash:
              function: sha512
              sum: fc96e821fd593a212c632a6c9093143fab5817f6833ba1df1ced2ce4fb82f1ebefde71d9a898e8f9574515e9ba19e40f6ab09a907f6b1b908d7adfcf57b3bf8b

    - path: /opt/bin/kubelet
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubelet
          verification:
            hash:
              function: sha512
              sum: 5cf4bde886d832d1cc48c47aeb43768050f67fe0458a330e4702b8071567665c975ed1fe2296cba5aea95a6de0bec4b731a32525837cac24646fb0158e2c2f64

    - path: /opt/bin/kubectl
      filesystem: root
      mode: 511 # 0777
      contents:
        remote:
          url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubectl
          verification:
            hash:
              function: sha512
              sum: 38a2746ac7b87cf7969cf33ccac177e63a6a0020ac593b7d272d751889ab568ad46a60e436d2f44f3654e2b4b5b196eabf8860b3eb87368f0861e2b3eb545a80

    - path: /etc/systemd/system/kubelet.service
      filesystem: root
      mode: 420 # 0644
      contents:
        remote:
          url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/kubelet.service
          verification:
            hash:
              function: sha512
              sum: b9ca0db34fea67dfd0654e65d3898a72997b1360c1e802cab5adc4288199c1a08423f90751757af4a7f1ff5932bfd81d3e215ce9b9d3f4efa1c04a202228adc8

    - path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
      filesystem: root
      mode: 420 # 0644
      contents:
        remote:
          url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/10-kubeadm.conf
          verification:
            hash:
              function: sha512
              sum: 32cfc8e56ec6e5ba93219852a68ec5eb25938a39c3e360ea4728fc71a14710b6ff85d0d84c2663eb5297d5dc21e1fad6914d6c0a8ce3357283f0b98ad4280ef7

    - path: /ignition/init/cni/cni-plugins-v0.7.1.tgz
      filesystem: root
      mode: 420 # 0644
      contents:
        remote:
          url: https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
          verification:
            hash:
              function: sha512
              sum: b3b0c1cc7b65cea619bddae4c17b8b488e7e13796345c7f75e069af93d1146b90a66322be6334c4c107e8a0ccd7c6d0b859a44a6745f9b85a0239d1be9ad4ccd

    - path: /ignition/init/canal/rbac.yaml
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
          verification:
            hash:
              function: sha512
              sum: e045645a1f37b4974890c3e4f8505a10bbb138ed0723869d7a7bc399c449072dfd2c8c2c482d3baac9bf700b7b0cfdca122cb260e70b437fb495eb86f9f6cccc

    - path: /ignition/init/canal/canal.yaml
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml
          verification:
            hash:
              function: sha512
              sum: 5c4953d74680a2ff84349c730bba48d0d43e28b70217c59cdf736f65e198f362c25b534b686c10ed5370dbb1bca4a1326e42039ddee7eb7b0dd314cc08523b67

    - path: /ignition/init/kube-router/kube-router.yaml
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://raw.githubusercontent.com/cloudnativelabs/kube-router/v0.1.0/daemonset/kubeadm-kuberouter-all-features-hostport.yaml
          verification:
            hash:
              function: sha512
              sum: e47051a2ccef7a3af050cdd4e4b8bc9a78f8ee130962b1d18ebbb50b51eaf6125aaf8f03fff5b3dde6ba4296fa73341656decd6338925fc4ea4a50fb206bfbbf

    - path: /ignition/init/k8s/install.sh
      filesystem: root
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Unzip the kubernetes binaries if not already present
          test -d /opt/bin/kubeadm && echo "k8s binaries (kubeadm) already installed" && exit 0

          # NOTE: If RELEASE is updated, the SHA512 SUMs will need to be as well
          echo -e "=> Installing k8s v1.10.2"

          echo "=> Cusomizing kubelet.service..."
          sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service
          sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

          systemctl daemon-reload
          systemctl enable kubelet
          systemctl start kubelet          

    - filesystem: root
      path: /ignition/init/rkt/setup.sh
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Unzip the kubernetes binaries if not already present
          test -d /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf && echo "rkt systemd drop-ins already installed" && exit 0

          echo "=> Adding dropins for rkt...."
          cat > /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf <<EOF
          [Service]
          Environment="KUBELET_EXTRA_ARGS=--container-runtime=rkt --runtime-request-timeout=15m --volume-plugin-dir=/var/lib/kubelet/volumeplugins --anonymous-auth=false --read-only-port=0 cadvisor-port=0"
          EOF

          echo "=> Triggering systemctl daemon-reload...."
          systemctl daemon-reload
          systemctl enable rkt-api
          systemctl enable rkt-metadata
          systemctl start rkt-api
          systemctl start rkt-metadata
          systemctl start rkt-gc          

    - filesystem: root
      path: /ignition/init/cni/install.sh
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Unzip the kubernetes binaries if not already present
          test -d /opt/cni/bin && echo "CNI binaries already installed" && exit 0

          VERSION=0.7.1
          echo -e "=> Installing CNI (v${VERSION}) binaries to /opt/cni/bin"
          cd /ignition/init/cni
          mkdir -p /opt/cni/bin
          tar -C /opt/cni/bin -k -xzf cni-plugins-v${VERSION}.tgz          

    - filesystem: root
      path: /ignition/init/kubeadm/kubeadm-install.sh
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Ensure kubeadm binary is present
          test -f /opt/bin/kubeadm || (echo "Failed to find kubeadm binary" && exit 1)

          # Exit if kubeadm has already been run (/etc/kubernetes folder would have been created)
          test -d /etc/kubernetes && echo "/etc/kubernetes is present, kubeadm should have already been run once" && exit 0

          echo "=> Running kubeadm init..."
          /opt/bin/kubeadm init --pod-network-cidr "10.244.0.0/16"

          echo "=> Running kubeadm post-install set up for user 'core'"
          mkdir -p /home/core/.kube
          cp -i /etc/kubernetes/admin.conf /home/core/.kube/config
          chown $(id -u core):$(id -g core) /home/core/.kube/config          

    - filesystem: root
      path: /ignition/init/k8s/setup.sh
      mode: 493 # 0755
      contents:
        inline: |
          #!/bin/bash

          # Ensure /etc/kubernetes is present (created by kubeadm)
          test -d /etc/kubernetes || (echo "/etc/kubernetes not present, ensure kubeadm has run properly" && exit 1)

          test -f /home/core/.kubernetes-setup-finished && echo "kubeadm already run, k8s should be set up" && exit 0

          echo "=> Enabling workload running on the master node"
          kubectl taint nodes --all node-role.kubernetes.io/master-

          echo "=> Installing kube-router"
          kubectl apply -f /ignition/init/kube-router/kube-router.yaml

          echo "=> Removing kube-proxy (kube-router will perform routing for the node)"
          kubectl delete daemonset kube-proxy -n kube-system
          docker run --privileged --net=host k8s.gcr.io/kube-proxy-amd64:v1.10.2 kube-proxy --cleanup

          # Disable docker (it won't be used by anything after now)
          sudo systemctl stop docker
          sudo systemctl disable docker

          echo "=> Leaving post-setup lock file @ [/home/core/.kubernetes-setup-finished]"
          touch /home/core/.kubernetes-setup-finished          

    - filesystem: root
      path: /etc/cni/net.d/10-kuberouter.conf
      mode: 493 # 0755
      contents:
        inline: |
          {
            "name":"kubernetes",
            "type":"bridge",
            "bridge":"kube-bridge",
            "isDefaultGateway":true,
            "ipam": {
              "type":"host-local"
            }
          }          

    - filesystem: root
      path: /ignition/init/kube-router/kube-router.yaml
      mode: 493 # 0755
      contents:
        inline: |
          ---
          apiVersion: extensions/v1beta1
          kind: DaemonSet
          metadata:
            labels:
              k8s-app: kube-router
              tier: node
            name: kube-router
            namespace: kube-system
          spec:
            template:
              metadata:
                labels:
                  k8s-app: kube-router
                  tier: node
                annotations:
                  scheduler.alpha.kubernetes.io/critical-pod: ''
              spec:
                serviceAccountName: kube-router
                serviceAccount: kube-router
                containers:
                - name: kube-router
                  image: cloudnativelabs/kube-router:v0.2.0-beta.6
                  imagePullPolicy: Always
                  args:
                  - --run-router=true
                  - --run-firewall=true
                  - --run-service-proxy=true
                  - --kubeconfig=/var/lib/kube-router/kubeconfig
                  env:
                  - name: NODE_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: spec.nodeName
                  livenessProbe:
                    httpGet:
                      path: /healthz
                      port: 20244
                    initialDelaySeconds: 10
                    periodSeconds: 3
                  resources:
                    requests:
                      cpu: 250m
                      memory: 250Mi
                  securityContext:
                    privileged: true
                  volumeMounts:
                  - name: lib-modules
                    mountPath: /lib/modules
                    readOnly: true
                  - name: cni-conf-dir
                    mountPath: /etc/cni/net.d
                  - name: kubeconfig
                    mountPath: /var/lib/kube-router
                    readOnly: true
                hostNetwork: true
                tolerations:
                - key: CriticalAddonsOnly
                  operator: Exists
                - effect: NoSchedule
                  key: node-role.kubernetes.io/master
                  operator: Exists
                volumes:
                - name: lib-modules
                  hostPath:
                    path: /lib/modules
                - name: cni-conf-dir
                  hostPath:
                    path: /etc/cni/net.d
                - name: kubeconfig
                  configMap:
                    name: kube-proxy
                    items:
                    - key: kubeconfig.conf
                      path: kubeconfig
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: kube-router
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1beta1
          metadata:
            name: kube-router
            namespace: kube-system
          rules:
            - apiGroups:
              - ""
              resources:
                - namespaces
                - pods
                - services
                - nodes
                - endpoints
              verbs:
                - list
                - get
                - watch
            - apiGroups:
              - "networking.k8s.io"
              resources:
                - networkpolicies
              verbs:
                - list
                - get
                - watch
            - apiGroups:
              - extensions
              resources:
                - networkpolicies
              verbs:
                - get
                - list
                - watch
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1beta1
          metadata:
            name: kube-router
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: kube-router
          subjects:
          - kind: ServiceAccount
            name: kube-router
            namespace: kube-system          

Some things to note:

  • The canal stuff still gets downloaded, just in case you need it or want to switch to it.
  • I had to remove the initContainer that kube-router was using, it was crashing in init.
  • The version is also arbitrarily pinned to the most recent latest (that worked), you should probably change that at your leisure.
  • Almost unbearably hacky, but I had to include the whole resource definition in here because the initContainer is bugged in rktnetes, I think.

Like what you're reading? Get it in your inbox