+
+
I recently wrote a post about about switching back to container linux for my small Kubernetes cluster, in which I outlined everything i needed to do to get it up and running. Even more recently, I decided I wanted to go ahead and run “rktnetes” to try and take advantage of it’s annotation-powered stage1 selection, and figured I should post that up too for any fellow rkt enthusiasts!
Along with using rkt, I’ve also used a newer version of the CNI plugins, along with bringing kube-router back into the fold (over canal). There are also some small script improvements (for example a lock file to ensure the k8s-setup oneshot systemd service doesn’t run unnecessarily).
Without further ado, here’s the updated ignition.yaml:
# This config is meant to be consumed by the config transpiler, which will
# generate the corresponding Ignition config. Do not pass this config directly
# to instances of Container Linux.
# NOTE: This configuration is meant to work with Config Transpiler v0.8.0
# The spec is available at (https://github.com/coreos/container-linux-config-transpiler/blob/v0.8.0/doc/configuration.md)
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa <LONG KEY>
- ssh-rsa <LONG KEY>
systemd:
units:
# Docker will be configured initially but we'll be using rkt exclusively and will disable it after containerd setup
- name: docker.service
enabled: true
####################
# services for rkt #
####################
- name: rkt-api.service
enabled: true
contents: |
[Unit]
Description=rkt api service
Documentation=http://github.com/rkt/rkt
After=network.target rkt-api-tcp.socket
Requires=rkt-api-tcp.socket
[Service]
ExecStart=/usr/bin/rkt api-service
[Install]
WantedBy=multi-user.target
- name: rkt-api-tcp.socket
enabled: true
contents: |
[Unit]
Description=rkt api service socket
PartOf=rkt-api.service
[Socket]
ListenStream=127.0.0.1:15441
ListenStream=[::1]:15441
Service=rkt-api.service
BindIPv6Only=both
[Install]
WantedBy=sockets.target
- name: rkt-gc.service
enabled: true
contents: |
[Unit]
Description=Garbage Collection for rkt
[Service]
Environment=GRACE_PERIOD=24h
Type=oneshot
ExecStart=/usr/bin/rkt gc --grace-period=${GRACE_PERIOD}
- name: rkt-gc.timer
enabled: true
contents: |
[Unit]
Description=Periodic Garbage Collection for rkt
[Timer]
OnActiveSec=0s
OnUnitActiveSec=12h
[Install]
WantedBy=multi-user.target
- name: rkt-metadata.service
enabled: true
contents: |
[Unit]
Description=rkt metadata service
Documentation=http://github.com/rkt/rkt
After=network.target rkt-metadata.socket
Requires=rkt-metadata.socket
[Service]
ExecStart=/usr/bin/rkt metadata-service
[Install]
WantedBy=multi-user.target
- name: rkt-metadata.socket
enabled: true
contents: |
[Unit]
Description=rkt metadata service socket
PartOf=rkt-metadata.service
[Socket]
ListenStream=/run/rkt/metadata-svc.sock
SocketMode=0660
SocketUser=root
SocketGroup=root
RemoveOnStop=true
[Install]
WantedBy=sockets.target
################################
# services for install scripts #
################################
- name: k8s-install.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=k8s installation script
Wants=network-online.target
After=network.target network-online.target
[Service]
Type=oneshot
ExecStart=/ignition/init/k8s/install.sh
- name: cni-install.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=cni plugin installation script
Requires=k8s-install.service
After=k8s-install.service
[Service]
Type=oneshot
ExecStart=/ignition/init/cni/install.sh
- name: rkt-setup.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=rkt installation script
Requires=cni-install.service
After=cni-install.service
[Service]
Type=oneshot
ExecStart=/ignition/init/rkt/setup.sh
- name: kubeadm-install.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=kubeadm installation script
Requires=rkt-setup.service
After=rkt-setup.service
[Service]
Type=oneshot
Environment="PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin"
ExecStart=/ignition/init/kubeadm/kubeadm-install.sh
- name: k8s-setup.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=kubernetes setup script
Requires=kubeadm-install.service
After=kubeadm-install.service
[Service]
Type=oneshot
User=core
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
ExecStart=/ignition/init/k8s/setup.sh
storage:
filesystems:
- mount:
device: /dev/disk/by-label/ROOT
format: xfs
wipe_filesystem: true
label: ROOT
files:
- path: /opt/bin/kubeadm
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubeadm
verification:
hash:
function: sha512
sum: fc96e821fd593a212c632a6c9093143fab5817f6833ba1df1ced2ce4fb82f1ebefde71d9a898e8f9574515e9ba19e40f6ab09a907f6b1b908d7adfcf57b3bf8b
- path: /opt/bin/kubelet
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubelet
verification:
hash:
function: sha512
sum: 5cf4bde886d832d1cc48c47aeb43768050f67fe0458a330e4702b8071567665c975ed1fe2296cba5aea95a6de0bec4b731a32525837cac24646fb0158e2c2f64
- path: /opt/bin/kubectl
filesystem: root
mode: 511 # 0777
contents:
remote:
url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubectl
verification:
hash:
function: sha512
sum: 38a2746ac7b87cf7969cf33ccac177e63a6a0020ac593b7d272d751889ab568ad46a60e436d2f44f3654e2b4b5b196eabf8860b3eb87368f0861e2b3eb545a80
- path: /etc/systemd/system/kubelet.service
filesystem: root
mode: 420 # 0644
contents:
remote:
url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/kubelet.service
verification:
hash:
function: sha512
sum: b9ca0db34fea67dfd0654e65d3898a72997b1360c1e802cab5adc4288199c1a08423f90751757af4a7f1ff5932bfd81d3e215ce9b9d3f4efa1c04a202228adc8
- path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
filesystem: root
mode: 420 # 0644
contents:
remote:
url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/10-kubeadm.conf
verification:
hash:
function: sha512
sum: 32cfc8e56ec6e5ba93219852a68ec5eb25938a39c3e360ea4728fc71a14710b6ff85d0d84c2663eb5297d5dc21e1fad6914d6c0a8ce3357283f0b98ad4280ef7
- path: /ignition/init/cni/cni-plugins-v0.7.1.tgz
filesystem: root
mode: 420 # 0644
contents:
remote:
url: https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
verification:
hash:
function: sha512
sum: b3b0c1cc7b65cea619bddae4c17b8b488e7e13796345c7f75e069af93d1146b90a66322be6334c4c107e8a0ccd7c6d0b859a44a6745f9b85a0239d1be9ad4ccd
- path: /ignition/init/canal/rbac.yaml
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
verification:
hash:
function: sha512
sum: e045645a1f37b4974890c3e4f8505a10bbb138ed0723869d7a7bc399c449072dfd2c8c2c482d3baac9bf700b7b0cfdca122cb260e70b437fb495eb86f9f6cccc
- path: /ignition/init/canal/canal.yaml
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml
verification:
hash:
function: sha512
sum: 5c4953d74680a2ff84349c730bba48d0d43e28b70217c59cdf736f65e198f362c25b534b686c10ed5370dbb1bca4a1326e42039ddee7eb7b0dd314cc08523b67
- path: /ignition/init/kube-router/kube-router.yaml
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://raw.githubusercontent.com/cloudnativelabs/kube-router/v0.1.0/daemonset/kubeadm-kuberouter-all-features-hostport.yaml
verification:
hash:
function: sha512
sum: e47051a2ccef7a3af050cdd4e4b8bc9a78f8ee130962b1d18ebbb50b51eaf6125aaf8f03fff5b3dde6ba4296fa73341656decd6338925fc4ea4a50fb206bfbbf
- path: /ignition/init/k8s/install.sh
filesystem: root
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Unzip the kubernetes binaries if not already present
test -d /opt/bin/kubeadm && echo "k8s binaries (kubeadm) already installed" && exit 0
# NOTE: If RELEASE is updated, the SHA512 SUMs will need to be as well
echo -e "=> Installing k8s v1.10.2"
echo "=> Cusomizing kubelet.service..."
sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service
sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
- filesystem: root
path: /ignition/init/rkt/setup.sh
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Unzip the kubernetes binaries if not already present
test -d /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf && echo "rkt systemd drop-ins already installed" && exit 0
echo "=> Adding dropins for rkt...."
cat > /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=rkt --runtime-request-timeout=15m --volume-plugin-dir=/var/lib/kubelet/volumeplugins --anonymous-auth=false --read-only-port=0 cadvisor-port=0"
EOF
echo "=> Triggering systemctl daemon-reload...."
systemctl daemon-reload
systemctl enable rkt-api
systemctl enable rkt-metadata
systemctl start rkt-api
systemctl start rkt-metadata
systemctl start rkt-gc
- filesystem: root
path: /ignition/init/cni/install.sh
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Unzip the kubernetes binaries if not already present
test -d /opt/cni/bin && echo "CNI binaries already installed" && exit 0
VERSION=0.7.1
echo -e "=> Installing CNI (v${VERSION}) binaries to /opt/cni/bin"
cd /ignition/init/cni
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -k -xzf cni-plugins-v${VERSION}.tgz
- filesystem: root
path: /ignition/init/kubeadm/kubeadm-install.sh
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Ensure kubeadm binary is present
test -f /opt/bin/kubeadm || (echo "Failed to find kubeadm binary" && exit 1)
# Exit if kubeadm has already been run (/etc/kubernetes folder would have been created)
test -d /etc/kubernetes && echo "/etc/kubernetes is present, kubeadm should have already been run once" && exit 0
echo "=> Running kubeadm init..."
/opt/bin/kubeadm init --pod-network-cidr "10.244.0.0/16"
echo "=> Running kubeadm post-install set up for user 'core'"
mkdir -p /home/core/.kube
cp -i /etc/kubernetes/admin.conf /home/core/.kube/config
chown $(id -u core):$(id -g core) /home/core/.kube/config
- filesystem: root
path: /ignition/init/k8s/setup.sh
mode: 493 # 0755
contents:
inline: |
#!/bin/bash
# Ensure /etc/kubernetes is present (created by kubeadm)
test -d /etc/kubernetes || (echo "/etc/kubernetes not present, ensure kubeadm has run properly" && exit 1)
test -f /home/core/.kubernetes-setup-finished && echo "kubeadm already run, k8s should be set up" && exit 0
echo "=> Enabling workload running on the master node"
kubectl taint nodes --all node-role.kubernetes.io/master-
echo "=> Installing kube-router"
kubectl apply -f /ignition/init/kube-router/kube-router.yaml
echo "=> Removing kube-proxy (kube-router will perform routing for the node)"
kubectl delete daemonset kube-proxy -n kube-system
docker run --privileged --net=host k8s.gcr.io/kube-proxy-amd64:v1.10.2 kube-proxy --cleanup
# Disable docker (it won't be used by anything after now)
sudo systemctl stop docker
sudo systemctl disable docker
echo "=> Leaving post-setup lock file @ [/home/core/.kubernetes-setup-finished]"
touch /home/core/.kubernetes-setup-finished
- filesystem: root
path: /etc/cni/net.d/10-kuberouter.conf
mode: 493 # 0755
contents:
inline: |
{
"name":"kubernetes",
"type":"bridge",
"bridge":"kube-bridge",
"isDefaultGateway":true,
"ipam": {
"type":"host-local"
}
}
- filesystem: root
path: /ignition/init/kube-router/kube-router.yaml
mode: 493 # 0755
contents:
inline: |
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
k8s-app: kube-router
tier: node
name: kube-router
namespace: kube-system
spec:
template:
metadata:
labels:
k8s-app: kube-router
tier: node
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: kube-router
serviceAccount: kube-router
containers:
- name: kube-router
image: cloudnativelabs/kube-router:v0.2.0-beta.6
imagePullPolicy: Always
args:
- --run-router=true
- --run-firewall=true
- --run-service-proxy=true
- --kubeconfig=/var/lib/kube-router/kubeconfig
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
livenessProbe:
httpGet:
path: /healthz
port: 20244
initialDelaySeconds: 10
periodSeconds: 3
resources:
requests:
cpu: 250m
memory: 250Mi
securityContext:
privileged: true
volumeMounts:
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: cni-conf-dir
mountPath: /etc/cni/net.d
- name: kubeconfig
mountPath: /var/lib/kube-router
readOnly: true
hostNetwork: true
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
volumes:
- name: lib-modules
hostPath:
path: /lib/modules
- name: cni-conf-dir
hostPath:
path: /etc/cni/net.d
- name: kubeconfig
configMap:
name: kube-proxy
items:
- key: kubeconfig.conf
path: kubeconfig
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-router
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-router
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
- services
- nodes
- endpoints
verbs:
- list
- get
- watch
- apiGroups:
- "networking.k8s.io"
resources:
- networkpolicies
verbs:
- list
- get
- watch
- apiGroups:
- extensions
resources:
- networkpolicies
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-router
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-router
subjects:
- kind: ServiceAccount
name: kube-router
namespace: kube-system
Some things to note:
initContainer that kube-router was using, it was crashing in init.initContainer is bugged in rktnetes, I think.