It's been a while since this was posted. Hopefully the information in here is still useful to you (if it isn't please let me know!). If you want to get the new stuff as soon as it's out though, sign up to the mailing list below.
Join the Mailing list
+
+
I recently wrote a post about about switching back to container linux for my small Kubernetes cluster, in which I outlined everything i needed to do to get it up and running. Even more recently, I decided I wanted to go ahead and run “rktnetes” to try and take advantage of it’s annotation-powered stage1 selection, and figured I should post that up too for any fellow rkt enthusiasts!
Along with using rkt, I’ve also used a newer version of the CNI plugins, along with bringing kube-router back into the fold (over canal). There are also some small script improvements (for example a lock file to ensure the k8s-setup oneshot systemd service doesn’t run unnecessarily).
Without further ado, here’s the updated ignition.yaml:
# This config is meant to be consumed by the config transpiler, which will
# generate the corresponding Ignition config. Do not pass this config directly
# to instances of Container Linux.
# NOTE: This configuration is meant to work with Config Transpiler v0.8.0
# The spec is available at (https://github.com/coreos/container-linux-config-transpiler/blob/v0.8.0/doc/configuration.md)
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa <LONG KEY>
- ssh-rsa <LONG KEY>
systemd:
units:
# Docker will be configured initially but we'll be using rkt exclusively and will disable it after containerd setup
- name: docker.service
enabled: true
####################
# services for rkt #
####################
- name: rkt-api.service
enabled: true
contents: |
[Unit]
Description=rkt api service
Documentation=http://github.com/rkt/rkt
After=network.target rkt-api-tcp.socket
Requires=rkt-api-tcp.socket
[Service]
ExecStart=/usr/bin/rkt api-service
[Install]
WantedBy=multi-user.target
- name: rkt-api-tcp.socket
enabled: true
contents: |
[Unit]
Description=rkt api service socket
PartOf=rkt-api.service
[Socket]
ListenStream=127.0.0.1:15441
ListenStream=[::1]:15441
Service=rkt-api.service
BindIPv6Only=both
[Install]
WantedBy=sockets.target
- name: rkt-gc.service
enabled: true
contents: |
[Unit]
Description=Garbage Collection for rkt
[Service]
Environment=GRACE_PERIOD=24h
Type=oneshot
ExecStart=/usr/bin/rkt gc --grace-period=${GRACE_PERIOD}
- name: rkt-gc.timer
enabled: true
contents: |
[Unit]
Description=Periodic Garbage Collection for rkt
[Timer]
OnActiveSec=0s
OnUnitActiveSec=12h
[Install]
WantedBy=multi-user.target
- name: rkt-metadata.service
enabled: true
contents: |
[Unit]
Description=rkt metadata service
Documentation=http://github.com/rkt/rkt
After=network.target rkt-metadata.socket
Requires=rkt-metadata.socket
[Service]
ExecStart=/usr/bin/rkt metadata-service
[Install]
WantedBy=multi-user.target
- name: rkt-metadata.socket
enabled: true
contents: |
[Unit]
Description=rkt metadata service socket
PartOf=rkt-metadata.service
[Socket]
ListenStream=/run/rkt/metadata-svc.sock
SocketMode=0660
SocketUser=root
SocketGroup=root
RemoveOnStop=true
[Install]
WantedBy=sockets.target
################################
# services for install scripts #
################################
- name: k8s-install.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=k8s installation script
Wants=network-online.target
After=network.target network-online.target
[Service]
Type=oneshot
ExecStart=/ignition/init/k8s/install.sh
- name: cni-install.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=cni plugin installation script
Requires=k8s-install.service
After=k8s-install.service
[Service]
Type=oneshot
ExecStart=/ignition/init/cni/install.sh
- name: rkt-setup.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=rkt installation script
Requires=cni-install.service
After=cni-install.service
[Service]
Type=oneshot
ExecStart=/ignition/init/rkt/setup.sh
- name: kubeadm-install.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=kubeadm installation script
Requires=rkt-setup.service
After=rkt-setup.service
[Service]
Type=oneshot
Environment="PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/bin"
ExecStart=/ignition/init/kubeadm/kubeadm-install.sh
- name: k8s-setup.service
enabled: true
contents: |
[Install]
WantedBy=multi-user.target
[Unit]
Description=kubernetes setup script
Requires=kubeadm-install.service
After=kubeadm-install.service
[Service]
Type=oneshot
User=core
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin"
ExecStart=/ignition/init/k8s/setup.sh
storage:
filesystems:
- mount:
device: /dev/disk/by-label/ROOT
format: xfs
wipe_filesystem: true
label: ROOT
files:
- path: /opt/bin/kubeadm
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubeadm
verification:
hash:
function: sha512
sum: fc96e821fd593a212c632a6c9093143fab5817f6833ba1df1ced2ce4fb82f1ebefde71d9a898e8f9574515e9ba19e40f6ab09a907f6b1b908d7adfcf57b3bf8b
- path: /opt/bin/kubelet
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubelet
verification:
hash:
function: sha512
sum: 5cf4bde886d832d1cc48c47aeb43768050f67fe0458a330e4702b8071567665c975ed1fe2296cba5aea95a6de0bec4b731a32525837cac24646fb0158e2c2f64
- path: /opt/bin/kubectl
filesystem: root
mode: 511 # 0777
contents:
remote:
url: https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubectl
verification:
hash:
function: sha512
sum: 38a2746ac7b87cf7969cf33ccac177e63a6a0020ac593b7d272d751889ab568ad46a60e436d2f44f3654e2b4b5b196eabf8860b3eb87368f0861e2b3eb545a80
- path: /etc/systemd/system/kubelet.service
filesystem: root
mode: 420 # 0644
contents:
remote:
url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/kubelet.service
verification:
hash:
function: sha512
sum: b9ca0db34fea67dfd0654e65d3898a72997b1360c1e802cab5adc4288199c1a08423f90751757af4a7f1ff5932bfd81d3e215ce9b9d3f4efa1c04a202228adc8
- path: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
filesystem: root
mode: 420 # 0644
contents:
remote:
url: https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.2/build/debs/10-kubeadm.conf
verification:
hash:
function: sha512
sum: 32cfc8e56ec6e5ba93219852a68ec5eb25938a39c3e360ea4728fc71a14710b6ff85d0d84c2663eb5297d5dc21e1fad6914d6c0a8ce3357283f0b98ad4280ef7
- path: /ignition/init/cni/cni-plugins-v0.7.1.tgz
filesystem: root
mode: 420 # 0644
contents:
remote:
url: https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
verification:
hash:
function: sha512
sum: b3b0c1cc7b65cea619bddae4c17b8b488e7e13796345c7f75e069af93d1146b90a66322be6334c4c107e8a0ccd7c6d0b859a44a6745f9b85a0239d1be9ad4ccd
- path: /ignition/init/canal/rbac.yaml
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
verification:
hash:
function: sha512
sum: e045645a1f37b4974890c3e4f8505a10bbb138ed0723869d7a7bc399c449072dfd2c8c2c482d3baac9bf700b7b0cfdca122cb260e70b437fb495eb86f9f6cccc
- path: /ignition/init/canal/canal.yaml
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml
verification:
hash:
function: sha512
sum: 5c4953d74680a2ff84349c730bba48d0d43e28b70217c59cdf736f65e198f362c25b534b686c10ed5370dbb1bca4a1326e42039ddee7eb7b0dd314cc08523b67
- path: /ignition/init/kube-router/kube-router.yaml
filesystem: root
mode: 493 # 0755
contents:
remote:
url: https://raw.githubusercontent.com/cloudnativelabs/kube-router/v0.1.0/daemonset/kubeadm-kuberouter-all-features-hostport.yaml
verification:
hash:
function: sha512
sum: e47051a2ccef7a3af050cdd4e4b8bc9a78f8ee130962b1d18ebbb50b51eaf6125aaf8f03fff5b3dde6ba4296fa73341656decd6338925fc4ea4a50fb206bfbbf
- path: /ignition/init/k8s/install.sh
filesystem: root
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Unzip the kubernetes binaries if not already present
test -d /opt/bin/kubeadm && echo "k8s binaries (kubeadm) already installed" && exit 0
# NOTE: If RELEASE is updated, the SHA512 SUMs will need to be as well
echo -e "=> Installing k8s v1.10.2"
echo "=> Cusomizing kubelet.service..."
sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service
sed -i "s:/usr/bin:/opt/bin:g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
- filesystem: root
path: /ignition/init/rkt/setup.sh
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Unzip the kubernetes binaries if not already present
test -d /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf && echo "rkt systemd drop-ins already installed" && exit 0
echo "=> Adding dropins for rkt...."
cat > /etc/systemd/system/kubelet.service.d/0-kubelet-extra-args.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=rkt --runtime-request-timeout=15m --volume-plugin-dir=/var/lib/kubelet/volumeplugins --anonymous-auth=false --read-only-port=0 cadvisor-port=0"
EOF
echo "=> Triggering systemctl daemon-reload...."
systemctl daemon-reload
systemctl enable rkt-api
systemctl enable rkt-metadata
systemctl start rkt-api
systemctl start rkt-metadata
systemctl start rkt-gc
- filesystem: root
path: /ignition/init/cni/install.sh
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Unzip the kubernetes binaries if not already present
test -d /opt/cni/bin && echo "CNI binaries already installed" && exit 0
VERSION=0.7.1
echo -e "=> Installing CNI (v${VERSION}) binaries to /opt/cni/bin"
cd /ignition/init/cni
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -k -xzf cni-plugins-v${VERSION}.tgz
- filesystem: root
path: /ignition/init/kubeadm/kubeadm-install.sh
mode: 480 # 740
contents:
inline: |
#!/bin/bash
# Ensure kubeadm binary is present
test -f /opt/bin/kubeadm || (echo "Failed to find kubeadm binary" && exit 1)
# Exit if kubeadm has already been run (/etc/kubernetes folder would have been created)
test -d /etc/kubernetes && echo "/etc/kubernetes is present, kubeadm should have already been run once" && exit 0
echo "=> Running kubeadm init..."
/opt/bin/kubeadm init --pod-network-cidr "10.244.0.0/16"
echo "=> Running kubeadm post-install set up for user 'core'"
mkdir -p /home/core/.kube
cp -i /etc/kubernetes/admin.conf /home/core/.kube/config
chown $(id -u core):$(id -g core) /home/core/.kube/config
- filesystem: root
path: /ignition/init/k8s/setup.sh
mode: 493 # 0755
contents:
inline: |
#!/bin/bash
# Ensure /etc/kubernetes is present (created by kubeadm)
test -d /etc/kubernetes || (echo "/etc/kubernetes not present, ensure kubeadm has run properly" && exit 1)
test -f /home/core/.kubernetes-setup-finished && echo "kubeadm already run, k8s should be set up" && exit 0
echo "=> Enabling workload running on the master node"
kubectl taint nodes --all node-role.kubernetes.io/master-
echo "=> Installing kube-router"
kubectl apply -f /ignition/init/kube-router/kube-router.yaml
echo "=> Removing kube-proxy (kube-router will perform routing for the node)"
kubectl delete daemonset kube-proxy -n kube-system
docker run --privileged --net=host k8s.gcr.io/kube-proxy-amd64:v1.10.2 kube-proxy --cleanup
# Disable docker (it won't be used by anything after now)
sudo systemctl stop docker
sudo systemctl disable docker
echo "=> Leaving post-setup lock file @ [/home/core/.kubernetes-setup-finished]"
touch /home/core/.kubernetes-setup-finished
- filesystem: root
path: /etc/cni/net.d/10-kuberouter.conf
mode: 493 # 0755
contents:
inline: |
{
"name":"kubernetes",
"type":"bridge",
"bridge":"kube-bridge",
"isDefaultGateway":true,
"ipam": {
"type":"host-local"
}
}
- filesystem: root
path: /ignition/init/kube-router/kube-router.yaml
mode: 493 # 0755
contents:
inline: |
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
k8s-app: kube-router
tier: node
name: kube-router
namespace: kube-system
spec:
template:
metadata:
labels:
k8s-app: kube-router
tier: node
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: kube-router
serviceAccount: kube-router
containers:
- name: kube-router
image: cloudnativelabs/kube-router:v0.2.0-beta.6
imagePullPolicy: Always
args:
- --run-router=true
- --run-firewall=true
- --run-service-proxy=true
- --kubeconfig=/var/lib/kube-router/kubeconfig
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
livenessProbe:
httpGet:
path: /healthz
port: 20244
initialDelaySeconds: 10
periodSeconds: 3
resources:
requests:
cpu: 250m
memory: 250Mi
securityContext:
privileged: true
volumeMounts:
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: cni-conf-dir
mountPath: /etc/cni/net.d
- name: kubeconfig
mountPath: /var/lib/kube-router
readOnly: true
hostNetwork: true
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
volumes:
- name: lib-modules
hostPath:
path: /lib/modules
- name: cni-conf-dir
hostPath:
path: /etc/cni/net.d
- name: kubeconfig
configMap:
name: kube-proxy
items:
- key: kubeconfig.conf
path: kubeconfig
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-router
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-router
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
- services
- nodes
- endpoints
verbs:
- list
- get
- watch
- apiGroups:
- "networking.k8s.io"
resources:
- networkpolicies
verbs:
- list
- get
- watch
- apiGroups:
- extensions
resources:
- networkpolicies
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: kube-router
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-router
subjects:
- kind: ServiceAccount
name: kube-router
namespace: kube-system
Some things to note:
initContainer that kube-router was using, it was crashing in init.initContainer is bugged in rktnetes, I think.