+ +

I recently wrote a post about about switching back to container linux for my small Kubernetes cluster, in which I outlined everything i needed to do to get it up and running. Even more recently, I decided I wanted to go ahead and run “rktnetes” to try and take advantage of it’s annotation-powered stage1 selection, and figured I should post that up too for any fellow rkt enthusiasts!

Along with using rkt, I’ve also used a newer version of the CNI plugins, along with bringing kube-router back into the fold (over canal). There are also some small script improvements (for example a lock file to ensure the k8s-setup oneshot systemd service doesn’t run unnecessarily).

Without further ado, here’s the updated ignition.yaml:

# This config is meant to be consumed by the config transpiler, which will
# generate the corresponding Ignition config. Do not pass this config directly
# to instances of Container Linux.

# NOTE: This configuration is meant to work with Config Transpiler v0.8.0
# The spec is available at (https://github.com/coreos/container-linux-config-transpiler/blob/v0.8.0/doc/configuration.md)

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa <LONG KEY>
        - ssh-rsa <LONG KEY>

systemd:
  units:
    # Docker will be configured initially but we'll be using rkt exclusively and will disable it after containerd setup
    - name: docker.service
      enabled: true

    ####################
    # services for rkt #
    ####################
    - name: rkt-api.service
      enabled: true
      contents: |
        [Unit]
        Description=rkt api service
        Documentation=http://github.com/rkt/rkt
        After=network.target rkt-api-tcp.socket
        Requires=rkt-api-tcp.socket

        [Service]
        ExecStart=/usr/bin/rkt api-service

        [Install]
        WantedBy=multi-user.target

    - name: rkt-api-tcp.socket
      enabled: true
      contents: |
        [Unit]
        Description=rkt api service socket
        PartOf=rkt-api.service

        [Socket]
        ListenStream=127.0.0.1:15441
        ListenStream=[::1]:15441
        Service=rkt-api.service
        BindIPv6Only=both

        [Install]
        WantedBy=sockets.target

    - name: rkt-gc.service
      enabled: true
      contents: |
        [Unit]
        Description=Garbage Collection for rkt

        [Service]
        Environment=GRACE_PERIOD=24h
        Type=oneshot
        ExecStart=/usr/bin/rkt gc --grace-period=${GRACE_PERIOD}

    - name: rkt-gc.timer
      enabled: true
      contents: |
        [Unit]
        Description=Periodic Garbage Collection for rkt

        [Timer]
        OnActiveSec=0s
        OnUnitActiveSec=12h

        [Install]
        WantedBy=multi-user.target

    - name: rkt-metadata.service
      enabled: true
      contents: |
        [Unit]
        Description=rkt metadata service
        Documentation=http://github.com/rkt/rkt
        After=network.target rkt-metadata.socket
        Requires=rkt-metadata.socket

        [Service]
        ExecStart=/usr/bin/rkt metadata-service

        [Install]
        WantedBy=multi-user.target

    - name: rkt-metadata.socket
      enabled: true
      contents: |
        [Unit]
        Description=rkt metadata service socket
        PartOf=rkt-metadata.service

        [Socket]
        ListenStream=/run/rkt/metadata-svc.sock
        SocketMode=0660
        SocketUser=root
        SocketGroup=root
        RemoveOnStop=true

        [Install]
        WantedBy=sockets.target

    ################################
    # services for install scripts #
    ################################
    - name: k8s-install.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=k8s installation script
        Wants=network-online.target
        After=network.target network-online.target

        [Service]
        Type=oneshot
        ExecStart=/ignition/init/k8s/install.sh

    - name: cni-install.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=cni plugin installation script
        Requires=k8s-install.service
        After=k8s-install.service

        [Service]
        Type=oneshot
        ExecStart=/ignition/init/cni/install.sh

    - name: rkt-setup.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=rkt installation script
        Requires=cni-install.service
        After=cni-install.service

        [Service]
        Type=oneshot
        ExecStart=/ignition/init/rkt/setup.sh

    - name: kubeadm-install.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=kubeadm installation script
        Requires=rkt-setup.service
        After=rkt-setup.service

        [Service]
        Type=oneshot
        Environment="PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin"
        ExecStart=/ignition/init/kubeadm/kubeadm-install.sh

    - name: k8s-setup.service
      enabled: true
      contents: |
        [Install]
        WantedBy=multi-user.target

        [Unit]
        Description=kubernetes setup script
        Requires=kubeadm-install.service
        After=kubeadm-install.service

        [Service]
        Type=oneshot
        User=core
        Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
        ExecStart=/ignition/init/k8s/setup.sh

storage:
  filesystems:
    - mount:
        device: /dev/disk/by-label/ROOT
        format: xfs
        wipe_filesystem: true
        label: ROOT

  files:
    - path: /opt/bin/kubeadm
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubeadm
          verification:
            hash:
              function: sha512
              sum: fc96e821fd593a212c632a6c9093143fab5817f6833ba1df1ced2ce4fb82f1ebefde71d9a898e8f9574515e9ba19e40f6ab09a907f6b1b908d7adfcf57b3bf8b

    - path: /opt/bin/kubelet
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubelet
          verification:
            hash:
              function: sha512
              sum: 5cf4bde886d832d1cc48c47aeb43768050f67fe0458a330e4702b8071567665c975ed1fe2296cba5aea95a6de0bec4b731a32525837cac24646fb0158e2c2f64

    - path: /opt/bin/kubectl
      filesystem: root
      mode: 511 # 0777
      contents:
        remote:
          url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubectl
          verification:
            hash:
              function: sha512
              sum: 38a2746ac7b87cf7969cf33ccac177e63a6a0020ac593b7d272d751889ab568ad46a60e436d2f44f3654e2b4b5b196eabf8860b3eb87368f0861e2b3eb545a80

    - path: /etc/systemd/system/kubelet.service
      filesystem: root
      mode: 420 # 0644
      contents:
        remote:
          url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/kubelet.service
          verification:
            hash:
              function: sha512
              sum: b9ca0db34fea67dfd0654e65d3898a72997b1360c1e802cab5adc4288199c1a08423f90751757af4a7f1ff5932bfd81d3e215ce9b9d3f4efa1c04a202228adc8

    - path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
      filesystem: root
      mode: 420 # 0644
      contents:
        remote:
          url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/10-kubeadm.conf
          verification:
            hash:
              function: sha512
              sum: 32cfc8e56ec6e5ba93219852a68ec5eb25938a39c3e360ea4728fc71a14710b6ff85d0d84c2663eb5297d5dc21e1fad6914d6c0a8ce3357283f0b98ad4280ef7

    - path: /ignition/init/cni/cni-plugins-v0.7.1.tgz
      filesystem: root
      mode: 420 # 0644
      contents:
        remote:
          url: https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
          verification:
            hash:
              function: sha512
              sum: b3b0c1cc7b65cea619bddae4c17b8b488e7e13796345c7f75e069af93d1146b90a66322be6334c4c107e8a0ccd7c6d0b859a44a6745f9b85a0239d1be9ad4ccd

    - path: /ignition/init/canal/rbac.yaml
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
          verification:
            hash:
              function: sha512
              sum: e045645a1f37b4974890c3e4f8505a10bbb138ed0723869d7a7bc399c449072dfd2c8c2c482d3baac9bf700b7b0cfdca122cb260e70b437fb495eb86f9f6cccc

    - path: /ignition/init/canal/canal.yaml
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml
          verification:
            hash:
              function: sha512
              sum: 5c4953d74680a2ff84349c730bba48d0d43e28b70217c59cdf736f65e198f362c25b534b686c10ed5370dbb1bca4a1326e42039ddee7eb7b0dd314cc08523b67

    - path: /ignition/init/kube-router/kube-router.yaml
      filesystem: root
      mode: 493 # 0755
      contents:
        remote:
          url: https://raw.githubusercontent.com/cloudnativelabs/kube-router/v0.1.0/daemonset/kubeadm-kuberouter-all-features-hostport.yaml
          verification:
            hash:
              function: sha512
              sum: e47051a2ccef7a3af050cdd4e4b8bc9a78f8ee130962b1d18ebbb50b51eaf6125aaf8f03fff5b3dde6ba4296fa73341656decd6338925fc4ea4a50fb206bfbbf

    - path: /ignition/init/k8s/install.sh
      filesystem: root
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Unzip the kubernetes binaries if not already present
          test -d /opt/bin/kubeadm && echo "k8s binaries (kubeadm) already installed" && exit 0

          # NOTE: If RELEASE is updated, the SHA512 SUMs will need to be as well
          echo -e "=> Installing k8s v1.10.2"

          echo "=> Cusomizing kubelet.service..."
          sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service
          sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

          systemctl daemon-reload
          systemctl enable kubelet
          systemctl start kubelet

    - filesystem: root
      path: /ignition/init/rkt/setup.sh
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Unzip the kubernetes binaries if not already present
          test -d /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf && echo "rkt systemd drop-ins already installed" && exit 0

          echo "=> Adding dropins for rkt...."
          cat > /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf <<EOF
          [Service]
          Environment="KUBELET_EXTRA_ARGS=--container-runtime=rkt --runtime-request-timeout=15m --volume-plugin-dir=/var/lib/kubelet/volumeplugins --anonymous-auth=false --read-only-port=0 cadvisor-port=0"
          EOF

          echo "=> Triggering systemctl daemon-reload...."
          systemctl daemon-reload
          systemctl enable rkt-api
          systemctl enable rkt-metadata
          systemctl start rkt-api
          systemctl start rkt-metadata
          systemctl start rkt-gc

    - filesystem: root
      path: /ignition/init/cni/install.sh
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Unzip the kubernetes binaries if not already present
          test -d /opt/cni/bin && echo "CNI binaries already installed" && exit 0

          VERSION=0.7.1
          echo -e "=> Installing CNI (v${VERSION}) binaries to /opt/cni/bin"
          cd /ignition/init/cni
          mkdir -p /opt/cni/bin
          tar -C /opt/cni/bin -k -xzf cni-plugins-v${VERSION}.tgz

    - filesystem: root
      path: /ignition/init/kubeadm/kubeadm-install.sh
      mode: 480 # 740
      contents:
        inline: |
          #!/bin/bash

          # Ensure kubeadm binary is present
          test -f /opt/bin/kubeadm || (echo "Failed to find kubeadm binary" && exit 1)

          # Exit if kubeadm has already been run (/etc/kubernetes folder would have been created)
          test -d /etc/kubernetes && echo "/etc/kubernetes is present, kubeadm should have already been run once" && exit 0

          echo "=> Running kubeadm init..."
          /opt/bin/kubeadm init --pod-network-cidr "10.244.0.0/16"

          echo "=> Running kubeadm post-install set up for user 'core'"
          mkdir -p /home/core/.kube
          cp -i /etc/kubernetes/admin.conf /home/core/.kube/config
          chown $(id -u core):$(id -g core) /home/core/.kube/config

    - filesystem: root
      path: /ignition/init/k8s/setup.sh
      mode: 493 # 0755
      contents:
        inline: |
          #!/bin/bash

          # Ensure /etc/kubernetes is present (created by kubeadm)
          test -d /etc/kubernetes || (echo "/etc/kubernetes not present, ensure kubeadm has run properly" && exit 1)

          test -f /home/core/.kubernetes-setup-finished && echo "kubeadm already run, k8s should be set up" && exit 0

          echo "=> Enabling workload running on the master node"
          kubectl taint nodes --all node-role.kubernetes.io/master-

          echo "=> Installing kube-router"
          kubectl apply -f /ignition/init/kube-router/kube-router.yaml

          echo "=> Removing kube-proxy (kube-router will perform routing for the node)"
          kubectl delete daemonset kube-proxy -n kube-system
          docker run --privileged --net=host k8s.gcr.io/kube-proxy-amd64:v1.10.2 kube-proxy --cleanup

          # Disable docker (it won't be used by anything after now)
          sudo systemctl stop docker
          sudo systemctl disable docker

          echo "=> Leaving post-setup lock file @ [/home/core/.kubernetes-setup-finished]"
          touch /home/core/.kubernetes-setup-finished

    - filesystem: root
      path: /etc/cni/net.d/10-kuberouter.conf
      mode: 493 # 0755
      contents:
        inline: |
          {
            "name":"kubernetes",
            "type":"bridge",
            "bridge":"kube-bridge",
            "isDefaultGateway":true,
            "ipam": {
              "type":"host-local"
            }
          }

    - filesystem: root
      path: /ignition/init/kube-router/kube-router.yaml
      mode: 493 # 0755
      contents:
        inline: |
          ---
          apiVersion: extensions/v1beta1
          kind: DaemonSet
          metadata:
            labels:
              k8s-app: kube-router
              tier: node
            name: kube-router
            namespace: kube-system
          spec:
            template:
              metadata:
                labels:
                  k8s-app: kube-router
                  tier: node
                annotations:
                  scheduler.alpha.kubernetes.io/critical-pod: ''
              spec:
                serviceAccountName: kube-router
                serviceAccount: kube-router
                containers:
                - name: kube-router
                  image: cloudnativelabs/kube-router:v0.2.0-beta.6
                  imagePullPolicy: Always
                  args:
                  - --run-router=true
                  - --run-firewall=true
                  - --run-service-proxy=true
                  - --kubeconfig=/var/lib/kube-router/kubeconfig
                  env:
                  - name: NODE_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: spec.nodeName
                  livenessProbe:
                    httpGet:
                      path: /healthz
                      port: 20244
                    initialDelaySeconds: 10
                    periodSeconds: 3
                  resources:
                    requests:
                      cpu: 250m
                      memory: 250Mi
                  securityContext:
                    privileged: true
                  volumeMounts:
                  - name: lib-modules
                    mountPath: /lib/modules
                    readOnly: true
                  - name: cni-conf-dir
                    mountPath: /etc/cni/net.d
                  - name: kubeconfig
                    mountPath: /var/lib/kube-router
                    readOnly: true
                hostNetwork: true
                tolerations:
                - key: CriticalAddonsOnly
                  operator: Exists
                - effect: NoSchedule
                  key: node-role.kubernetes.io/master
                  operator: Exists
                volumes:
                - name: lib-modules
                  hostPath:
                    path: /lib/modules
                - name: cni-conf-dir
                  hostPath:
                    path: /etc/cni/net.d
                - name: kubeconfig
                  configMap:
                    name: kube-proxy
                    items:
                    - key: kubeconfig.conf
                      path: kubeconfig
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: kube-router
            namespace: kube-system
          ---
          kind: ClusterRole
          apiVersion: rbac.authorization.k8s.io/v1beta1
          metadata:
            name: kube-router
            namespace: kube-system
          rules:
            - apiGroups:
              - ""
              resources:
                - namespaces
                - pods
                - services
                - nodes
                - endpoints
              verbs:
                - list
                - get
                - watch
            - apiGroups:
              - "networking.k8s.io"
              resources:
                - networkpolicies
              verbs:
                - list
                - get
                - watch
            - apiGroups:
              - extensions
              resources:
                - networkpolicies
              verbs:
                - get
                - list
                - watch
          ---
          kind: ClusterRoleBinding
          apiVersion: rbac.authorization.k8s.io/v1beta1
          metadata:
            name: kube-router
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: kube-router
          subjects:
          - kind: ServiceAccount
            name: kube-router
            namespace: kube-system

Some things to note:

• The canal stuff still gets downloaded, just in case you need it or want to switch to it.
• I had to remove the initContainer that kube-router was using, it was crashing in init.
• The version is also arbitrarily pinned to the most recent latest (that worked), you should probably change that at your leisure.
• Almost unbearably hacky, but I had to include the whole resource definition in here because the initContainer is bugged in rktnetes, I think.