Awesome FOSS Logo
Discover awesome open source software
Launched 🚀🧑‍🚀

Building Custom Kernels for Kata Containers

Kubernetes logo + Kata containers project logo

tl;dr - The kata containers 3.0.2 documentation on custom kernels is great, but I ran into some issues, so I rehash exactly what I did below

I recently spent some time exploring how to build custom kernels to work with kata, and it’s way easier than it seems, though the docs are a bit stale/don’t work as expected.

Generally there are a few reasons you might want to do this, but usually it’s to enable kernel flags/features that weren’t enabled before. During a normal kernel build this is the configuration you’d do with make menuconfig.

Here’s a quick guide from the notes I took while doing it (and re-produced for this post):

0. Check which versions you’re currently using

If you’re already running kata-containers, you can check the versions you’re using:

pgrep qemu -la

You’ll see the invocation options of qemu, and should be able to confirm the kernel versions running (look for --kernel or vmlinux)

1. Clone the Kata source repository

Clone the [kata-containers repository][kata-repo] and navigate to tools/packaging/kernel:

git clone
cd kata-containers/tools/packaging/kernel

2. Run the helper script to download kernel source code

There’s a helpful script in that repository called, which we can use to pull a kernel that we want.

There are quite a few options to the script, but we’ll be using only one to specify the kernel version we want:

./ -v 5.15.26 setup

NOTE You can add -d to enable debugging

3. Configure the kernel

Here’s the part you’ve been waiting for, you need to configure the kernel!

I find it easiest to modify the files in configs/fragments/<platform> (ex. configs/fragments/x86_64/base.conf) to my liking.

Let’s change something trivial like the number of CPUs allowed:


# Configs around linux guest support and optimizations.


# Use the maximum number of CPUs supported by KVM (240)
CONFIG_NR_CPUS=9001 # <--- here's our change

# For security

# Boot directly into the uncompressed kernel
# Reduce memory footprint


4. Run setup again to build the kernel config

Now that we’ve made some changes, we can re-run setup to build the kernel configuration again:

./ -v 5.15.26 -f setup

What you’ll likely find though, is an error/failure:

❯ ./ -v 5.15.26 -f setup
INFO: Config version: 111
INFO: Kernel version: 5.15.26
INFO: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111 already exist
INFO: Force generate config due to '-f'
INFO: Apply patches from /path/to/kata-containers/tools/packaging/kernel/patches/5.15.x
INFO: Found 0 patches
grep: warning: stray \ before !
INFO: Constructing config from fragments: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
INFO: Some CONFIG elements failed to make the final .config:
INFO: Value requested for CONFIG_NR_CPUS not in final .config
INFO: Generated config file can be found in /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
ERROR: Failed to construct requested .config file
ERROR: failed to find default config

NOTE You might want to go with a regular stable release, or a different LTS version, etc.

The values from the fragment that we changed did not make it into the .config file that was generated for the kernel.

Turns out the value we used was too high (it was over 9000!), you can change it to something lower:


Once you have, you can re-run the configuration and confirm the change in the .config file:

➜ ./ -v 5.15.26 -f setup
INFO: Config version: 111
INFO: Kernel version: 5.15.26
INFO: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111 already exist
INFO: Force generate config due to '-f'
INFO: Apply patches from /path/to/kata-containers/tools/packaging/kernel/patches/5.15.x
INFO: Found 0 patches
grep: warning: stray \ before !
INFO: Constructing config from fragments: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
INFO: Copying config file from: /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config
# configuration written to .config
Kernel source ready: /path/to/kata-containers/tools/packaging/kernel/kata-linux-5.15.26-111

And if you confirm the change in the .config file:

➜ grep '120' /path/to/kata-containers/tools/packaging/kernel/configs/fragments/x86_64/.config

NOTE Yes, you could have done this first before running setup.

5. Build the kernel

Now we’re ready to actually build the kernel:

./ -v 5.15.26 build

After a bunch of compiling and waiting, you should see:

Kernel: arch/x86/boot/bzImage is ready  (#1)

6. Back up your currently used kernel

This is a great time to back up the kernels that your kata VMs are currently using.

By default, kata uses /usr/share/kata-containers, though my distribution/installation of k8s uses /opt/kata/share/kata-containers.

7. Install the new kernel to somewhere kata will pick it up

As kata-containers picks up kernels from a specified location on-disk, we can run the install script (which writes to that place on disk) and use it to install there:

# ./ -v 5.15.26 install
INFO: Config version: 110
INFO: Kernel version: 5.15.26
lrwxrwxrwx 1 root root 19 Jul 19 01:24 /usr/share/kata-containers/vmlinux.container -> vmlinux-5.15.26-110
lrwxrwxrwx 1 root root 19 Jul 19 01:24 /usr/share/kata-containers/vmlinuz.container -> vmlinuz-5.15.26-110

8. Start a new pod and check the results

At this point you should be able to start a pod (or any other kind of workload) using runtimeClassName: kata and confirm the qemuprocess using the newer versions of linux.

For example, you can grep the output of pgrep for -kernel and in my cas I see something like this:

-kernel /opt/kata/share/kata-containers/vmlinux-5.15.26-110


Thanks to kata-containers GitHub issue #3790 which noted some install options that I was missing originally.