tl;dr - The kata containers 3.0.2 documentation on custom kernels is great, but I ran into some issues, so I rehash exactly what I did below
I recently spent some time exploring how to build custom kernels to work with kata, and it’s way easier than it seems, though the docs are a bit stale/don’t work as expected.
Generally there are a few reasons you might want to do this, but usually it’s to enable kernel flags/features that weren’t enabled before. During a normal kernel build this is the configuration you’d do with
Here’s a quick guide from the notes I took while doing it (and re-produced for this post):
If you’re already running kata-containers, you can check the versions you’re using:
pgrep qemu -la
You’ll see the invocation options of
qemu, and should be able to confirm the kernel versions running (look for
Clone the [
kata-containers repository][kata-repo] and navigate to
git clone https://github.com/kata-containers/kata-containers.git cd kata-containers/tools/packaging/kernel
There’s a helpful script in that repository called
build-kernel.sh, which we can use to pull a kernel that we want.
There are quite a few options to the script, but we’ll be using only one to specify the kernel version we want:
./build-kernel.sh -v 5.15.26 setup
NOTE You can add
-d to enable debugging
Here’s the part you’ve been waiting for, you need to configure the kernel!
I find it easiest to modify the files in
configs/fragments/x86_64/base.conf) to my liking.
Let’s change something trivial like the number of CPUs allowed:
CONFIG_X86=y CONFIG_X86_CPUID=y CONFIG_X86_MSR=y CONFIG_X86_X2APIC=y CONFIG_X86_VERBOSE_BOOTUP=y # Configs around linux guest support and optimizations. CONFIG_HYPERVISOR_GUEST=y CONFIG_KVM_GUEST=y CONFIG_PARAVIRT=y # Use the maximum number of CPUs supported by KVM (240) CONFIG_NR_CPUS=9001 # <--- here's our change # For security CONFIG_LEGACY_VSYSCALL_NONE=y CONFIG_SPECULATION_MITIGATIONS=y CONFIG_RETPOLINE=y # Boot directly into the uncompressed kernel # Reduce memory footprint CONFIG_PVH=y CONFIG_NO_HZ_FULL=y CONFIG_GENERIC_MSI_IRQ_DOMAIN=y CONFIG_RANDOMIZE_BASE=y
Now that we’ve made some changes, we can re-run setup to build the kernel configuration again:
./build-kernel.sh -v 5.15.26 -f setup
What you’ll likely find though, is an error/failure:
❯ ./build-kernel.sh -v 5.15.26 -f setup INFO: Config version: 111 INFO: Kernel version: 5.15.26 INFO: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111 already exist INFO: Force generate config due to '-f' INFO: Apply patches from /path/to/kata-containers/tools/packaging/kernel/patches/5.15.x INFO: Found 0 patches grep: warning: stray \ before ! INFO: Constructing config from fragments: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config INFO: Some CONFIG elements failed to make the final .config: INFO: Value requested for CONFIG_NR_CPUS not in final .config INFO: Generated config file can be found in /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config ERROR: Failed to construct requested .config file ERROR: failed to find default config
NOTE You might want to go with a regular stable release, or a different LTS version, etc.
The values from the fragment that we changed did not make it into the
.config file that was generated for the kernel.
Turns out the value we used was too high (it was over 9000!), you can change it to something lower:
Once you have, you can re-run the configuration and confirm the change in the
➜ ./build-kernel.sh -v 5.15.26 -f setup INFO: Config version: 111 INFO: Kernel version: 5.15.26 INFO: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111 already exist INFO: Force generate config due to '-f' INFO: Apply patches from /path/to/kata-containers/tools/packaging/kernel/patches/5.15.x INFO: Found 0 patches grep: warning: stray \ before ! INFO: Constructing config from fragments: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config INFO: Copying config file from: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config # # configuration written to .config # Kernel source ready: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111
And if you confirm the change in the
➜ grep '120' /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config 355:CONFIG_NR_CPUS=120
NOTE Yes, you could have done this first before running setup.
Now we’re ready to actually build the kernel:
./build-kernel.sh -v 5.15.26 build
After a bunch of compiling and waiting, you should see:
... Kernel: arch/x86/boot/bzImage is ready (#1)
This is a great time to back up the kernels that your kata VMs are currently using.
By default, kata uses
/usr/share/kata-containers, though my distribution/installation of k8s uses
As kata-containers picks up kernels from a specified location on-disk, we can run the install script (which writes to that place on disk) and use it to install there:
# ./build-kernel.sh -v 5.15.26 install INFO: Config version: 110 INFO: Kernel version: 5.15.26 lrwxrwxrwx 1 root root 19 Jul 19 01:24 /usr/share/kata-containers/vmlinux.container -> vmlinux-5.15.26-110 lrwxrwxrwx 1 root root 19 Jul 19 01:24 /usr/share/kata-containers/vmlinuz.container -> vmlinuz-5.15.26-110
At this point you should be able to start a pod (or any other kind of workload) using
runtimeClassName: kata and confirm the
qemuprocess using the newer versions of linux.
For example, you can
grep the output of
-kernel and in my cas I see something like this:
kata-containers GitHub issue #3790 which noted some install options that I was missing originally.