DDG search drop in powered by ddg.patdryburgh.com
+
tl;dr - The kata containers 3.0.2 documentation on custom kernels is great, but I ran into some issues, so I rehash exactly what I did below
I recently spent some time exploring how to build custom kernels to work with kata, and it’s way easier than it seems, though the docs are a bit stale/don’t work as expected.
Generally there are a few reasons you might want to do this, but usually it’s to enable kernel flags/features that weren’t enabled before. During a normal kernel build this is the configuration you’d do with make menuconfig.
Here’s a quick guide from the notes I took while doing it (and re-produced for this post):
If you’re already running kata-containers, you can check the versions you’re using:
pgrep qemu -la
You’ll see the invocation options of qemu, and should be able to confirm the kernel versions running (look for --kernel or vmlinux)
Clone the [kata-containers repository][kata-repo] and navigate to tools/packaging/kernel:
git clone https://github.com/kata-containers/kata-containers.git
cd kata-containers/tools/packaging/kernel
There’s a helpful script in that repository called build-kernel.sh, which we can use to pull a kernel that we want.
There are quite a few options to the script, but we’ll be using only one to specify the kernel version we want:
./build-kernel.sh -v 5.15.26 setup
NOTE You can add -d to enable debugging
Here’s the part you’ve been waiting for, you need to configure the kernel!
I find it easiest to modify the files in configs/fragments/<platform> (ex. configs/fragments/x86_64/base.conf) to my liking.
Let’s change something trivial like the number of CPUs allowed:
CONFIG_X86=y
CONFIG_X86_CPUID=y
CONFIG_X86_MSR=y
CONFIG_X86_X2APIC=y
CONFIG_X86_VERBOSE_BOOTUP=y
# Configs around linux guest support and optimizations.
CONFIG_HYPERVISOR_GUEST=y
CONFIG_KVM_GUEST=y
CONFIG_PARAVIRT=y
# Use the maximum number of CPUs supported by KVM (240)
CONFIG_NR_CPUS=9001 # <--- here's our change
# For security
CONFIG_LEGACY_VSYSCALL_NONE=y
CONFIG_SPECULATION_MITIGATIONS=y
CONFIG_RETPOLINE=y
# Boot directly into the uncompressed kernel
# Reduce memory footprint
CONFIG_PVH=y
CONFIG_NO_HZ_FULL=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_RANDOMIZE_BASE=y
Now that we’ve made some changes, we can re-run setup to build the kernel configuration again:
./build-kernel.sh -v 5.15.26 -f setup
What you’ll likely find though, is an error/failure:
❯ ./build-kernel.sh -v 5.15.26 -f setup
INFO: Config version: 111
INFO: Kernel version: 5.15.26
INFO: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111 already exist
INFO: Force generate config due to '-f'
INFO: Apply patches from /path/to/kata-containers/tools/packaging/kernel/patches/5.15.x
INFO: Found 0 patches
grep: warning: stray \ before !
INFO: Constructing config from fragments: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
INFO: Some CONFIG elements failed to make the final .config:
INFO: Value requested for CONFIG_NR_CPUS not in final .config
INFO: Generated config file can be found in /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
ERROR: Failed to construct requested .config file
ERROR: failed to find default config
NOTE You might want to go with a regular stable release, or a different LTS version, etc.
The values from the fragment that we changed did not make it into the .config file that was generated for the kernel.
Turns out the value we used was too high (it was over 9000!), you can change it to something lower:
CONFIG_NR_CPUS=120
Once you have, you can re-run the configuration and confirm the change in the .config file:
➜ ./build-kernel.sh -v 5.15.26 -f setup
INFO: Config version: 111
INFO: Kernel version: 5.15.26
INFO: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111 already exist
INFO: Force generate config due to '-f'
INFO: Apply patches from /path/to/kata-containers/tools/packaging/kernel/patches/5.15.x
INFO: Found 0 patches
grep: warning: stray \ before !
INFO: Constructing config from fragments: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
INFO: Copying config file from: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
#
# configuration written to .config
#
Kernel source ready: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111
And if you confirm the change in the .config file:
➜ grep '120' /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
355:CONFIG_NR_CPUS=120
NOTE Yes, you could have done this first before running setup.
Now we’re ready to actually build the kernel:
./build-kernel.sh -v 5.15.26 build
After a bunch of compiling and waiting, you should see:
...
Kernel: arch/x86/boot/bzImage is ready (#1)
This is a great time to back up the kernels that your kata VMs are currently using.
By default, kata uses /usr/share/kata-containers, though my distribution/installation of k8s uses /opt/kata/share/kata-containers.
As kata-containers picks up kernels from a specified location on-disk, we can run the install script (which writes to that place on disk) and use it to install there:
# ./build-kernel.sh -v 5.15.26 install
INFO: Config version: 110
INFO: Kernel version: 5.15.26
lrwxrwxrwx 1 root root 19 Jul 19 01:24 /usr/share/kata-containers/vmlinux.container -> vmlinux-5.15.26-110
lrwxrwxrwx 1 root root 19 Jul 19 01:24 /usr/share/kata-containers/vmlinuz.container -> vmlinuz-5.15.26-110
At this point you should be able to start a pod (or any other kind of workload) using runtimeClassName: kata and confirm the qemuprocess using the newer versions of linux.
For example, you can grep the output of pgrep for -kernel and in my cas I see something like this:
-kernel /opt/kata/share/kata-containers/vmlinux-5.15.26-110
Thanks to kata-containers GitHub issue #3790 which noted some install options that I was missing originally.