+
tl;dr - Bluetooth died after a recent upgrade to Linux kernel 5.9.1 on Arch which people noticed nVidia didn’t support – I upgraded anyway, graphics were fine, but Bluetooth was not.
So very recently a release of NVIDIA’s drivers was confirmed not to work with Linux kernel 5.9.x. It’s been discussed widely on the internet (by those affected) so I won’t go into it here:
Read those sources if you want more context, but the gist as I understand is:
nvidia_uvm (?) and other modules that will fix/re-enable thingsI’m not NVIDIA’s greatest fan nor have I been greatly wronged by NVIDIA, but I do run (and am very happy with) a Oryx Pro so that’s what’s in my laptop, so I’m affected. If I was building a desktop today I’d pick a DAN-style Mini-ITX case, a Ryzen Threadripper (what does htop even look like with that many cores??) and a Radeon
Turns out I can actually play CS:GO (the only game I play these days) just fine, but i my case, graphics weren’t the problem – Bluetooth was.
bluez?user@oryxpro $ systemctl status bluetooth
● bluetooth.service - Bluetooth service
Loaded: loaded (/usr/lib/systemd/system/bluetooth.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-10-23 20:04:42 JST; 14h ago
Docs: man:bluetoothd(8)
Main PID: 82921 (bluetoothd)
Status: "Running"
Tasks: 1 (limit: 38142)
Memory: 896.0K
CPU: 17ms
CGroup: /system.slice/bluetooth.service
└─82921 /usr/lib/bluetooth/bluetoothd
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEMinCon>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEMaxCon>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEConnec>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEConnec>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEAutoco>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Starting SDP server
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Bluetooth management interface 1.18 initialized
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Endpoint registered: sender=:1.37 path=/MediaEndpoint/A2DPSink/sbc
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Endpoint registered: sender=:1.37 path=/MediaEndpoint/A2DPSource/sbc
Oct 23 20:05:03 oryxpro bluetoothd[82921]: Failed to set mode: Failed (0x03)
If you try and run bluetoothctl and do things like power on or connect <device> you’re going to get a bunch of errors telling you that bluez (the Linux bluetooth stack) is gone/wonky. I did a small amount of searching and didn’t find too much:
So I think it might be kernel problem (since that was the last big change).For now I’m going to have to downgrade to kernel 5.8.9 – although not for the reasons everyone else is.
The excellent Arch Wiki has you covered as usual, read there first before some random blog on the internet (where you are now).
For the impatient, the basic gist of the relevant sections of that wiki is:
/var/cache/pacman/pkgpacman -Ulinux and linux-headers, but watch out for other packages that depended on the newer/older kernel (HINT FROM THE FUTURE: I’m about to run head first into this)I ran into a snag when I tried to simply downgrade linux and linux-headers though:
user@oryxpro $ cd /var/cache/pacman/pkg/
user@oryxpro $ sudo pacman -U linux-5.8.9.arch2-1-x86_64.pkg.tar.zst linux-headers-5.8.9.arch2-1-x86_64.pkg.tar.zst
loading packages...
warning: downgrading package linux (5.9.1.arch1-1 => 5.8.9.arch2-1)
warning: downgrading package linux-headers (5.9.1.arch1-1 => 5.8.9.arch2-1)
resolving dependencies...
looking for conflicting packages...
Packages (2) linux-5.8.9.arch2-1 linux-headers-5.8.9.arch2-1
Total Installed Size: 199.79 MiB
Net Upgrade Size: -2.51 MiB
:: Proceed with installation? [Y/n] Y
(2/2) checking keys in keyring [######################################################] 100%
(2/2) checking package integrity [######################################################] 100%
(2/2) loading package files [######################################################] 100%
(2/2) checking for file conflicts [######################################################] 100%
(2/2) checking available disk space [######################################################] 100%
warning: could not get file information for usr/lib/modules/5.9.1-arch1-1/kernel/drivers/platform/x86/system76_acpi.ko.xz
:: Running pre-transaction hooks...
(1/2) Removing linux initcpios...
(2/2) Remove DKMS modules
==> dkms remove --no-depmod -m system76-acpi -v 1.0.1 -k 5.9.1-arch1-1
==> dkms remove --no-depmod -m system76-io -v 1.0.1 -k 5.9.1-arch1-1
==> dkms remove --no-depmod -m system76 -v 1.0.9 -k 5.9.1-arch1-1
==> depmod 5.9.1-arch1-1
:: Processing package changes...
(1/2) downgrading linux [######################################################] 100%
(2/2) downgrading linux-headers [######################################################] 100%
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating module dependencies...
(3/4) Install DKMS modules
==> dkms install --no-depmod -m system76-acpi -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76-io -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76 -v 1.0.9 -k 5.8.9-arch2-1
==> depmod 5.8.9-arch2-1
(4/4) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> ERROR: module not found: `nvidia'
==> ERROR: module not found: `nvidia_modeset'
==> ERROR: module not found: `nvidia_uvm'
==> ERROR: module not found: `nvidia_drm'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> ERROR: module not found: `nvidia'
==> ERROR: module not found: `nvidia_modeset'
==> ERROR: module not found: `nvidia_uvm'
==> ERROR: module not found: `nvidia_drm'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
Well of course, nothing ever works the first time, but it is nice to see that the system76-* drivers build just fine with the older kernel and get loaded & unloaded (I sure would have not liked to restart and have nothing supported properly), but there’s a problem with the nvidia ones!
So I probably need to downgrade the nvidia drivers first right?
NARRATOR: He didn’t.
WARNING FROM THE FUTURE: In case it wasn’t obvious from the previous section’s ending, this wasn’t actually necessary,
nvidia-dkmsis what was needed since it’s the reason that nvidia didn’t do a build during the downgrade phase.
I don’t use nvidia-dkms (I don’t have it installed) but when I search /var/cache/pacman/pkg for "nvidia" I see the combination of nvidia-utils and nvidia packages. As you might have guessed nvidia package version numbers don’t map to kernel numbers, but after a little searching I found a thread that makes me think maybe I should try the 450 series of patches (maybe those were the ones that worked with 4.8.9 kernels?
user@oryxpro $ cd /var/cache/pacman/pkg/
user@oryxpro $ sudo pacman -U lib32-nvidia-utils-450.66-1-x86_64.pkg.tar.zst nvidia-utils-450.66-1-x86_64.pkg.tar.zst nvidia-450.66-22-x86_64.pkg.tar.zst
loading packages...
warning: downgrading package lib32-nvidia-utils (455.28-1 => 450.66-1)
warning: downgrading package nvidia-utils (455.28-1 => 450.66-1)
warning: downgrading package nvidia (455.28-4 => 450.66-22)
resolving dependencies...
looking for conflicting packages...
Packages (3) lib32-nvidia-utils-450.66-1 nvidia-450.66-22 nvidia-utils-450.66-1
Total Installed Size: 362.86 MiB
Net Upgrade Size: -71.25 MiB
:: Proceed with installation? [Y/n] Y
(3/3) checking keys in keyring [##########################################################] 100%
(3/3) checking package integrity [##########################################################] 100%
(3/3) loading package files [##########################################################] 100%
(3/3) checking for file conflicts [##########################################################] 100%
(3/3) checking available disk space [##########################################################] 100%
:: Processing package changes...
(1/3) downgrading nvidia-utils [##########################################################] 100%
If you run into trouble with CUDA not being available, run nvidia-modprobe first.
(2/3) downgrading lib32-nvidia-utils [##########################################################] 100%
(3/3) downgrading nvidia [##########################################################] 100%
:: Running post-transaction hooks...
(1/4) Creating system user accounts...
(2/4) Reloading system manager configuration...
(3/4) Arming ConditionNeedsUpdate...
(4/4) Updating module dependencies...
Well that worked out just fine, maybe I’m good to go now?
NARRATOR: He wasn’t.
So now that I have nvidia downgraded, hopefully kernel downgrade will go smoothly… aaaand it didn’t, same error.
At this point, there must be a misunderstanding in my knowledge – in particular what nvidia-dkms actually is/does and whether I should have it. A friendly forum poster shed some light on it for me (without diving into the actual Arch Wiki).
So what nvidia-dkms will do is enable me to rebuild relevant nvidia driver(s) every time a new custom kernel (which mine is indeed somewhat customized) is built. This is exactly what I need, I need the nvidia driver to get rebuilt just like System76’s drivers.
At first try though, it looks like it has a problem with the downgraded nvidia drivers:
user@oryxpro $ sudo pacman -S nvidia-dkms
[sudo] password for user:
resolving dependencies...
looking for conflicting packages...
error: failed to prepare transaction (could not satisfy dependencies)
:: installing nvidia-utils (455.28-1) breaks dependency 'nvidia-utils=450.66' required by nvidia
So lets… upgrade to downgrade? I mean technically the 455 series didn’t produce a problem with kernels < 5.9 AFAIK, so maybe I could just use the 455 without downgrading anyway.
user@oryxpro $ sudo pacman -S nvidia
<successful, so output redacted>
user@oryxpro $ sudo pacman -S nvidia-dkms
resolving dependencies...
looking for conflicting packages...
Packages (1) nvidia-dkms-455.28-1
Total Download Size: 18.56 MiB
Total Installed Size: 40.99 MiB
:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
nvidia-dkms-455.28-1-x86_64 18.6 MiB 4.94 MiB/s 00:04 [##########################################################] 100%
(1/1) checking keys in keyring [##########################################################] 100%
^N(1/1) checking package integrity [##########################################################] 100%
(1/1) loading package files [##########################################################] 100%
(1/1) checking for file conflicts [##########################################################] 100%
(1/1) checking available disk space [##########################################################] 100%
:: Processing package changes...
(1/1) installing nvidia-dkms [##########################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Install DKMS modules
==> dkms install --no-depmod -m nvidia -v 455.28 -k 5.8.9-arch2-1
==> depmod 5.8.9-arch2-1
OK so now what I’m expecting is that when I try to downgrade the kernel, the nvidia module will get rebuilt right on the spot, let’s see it happen:
user@oryxpro $ sudo pacman -U linux-5.8.9.arch2-1-x86_64.pkg.tar.zst linux-headers-5.8.9.arch2-1-x86_64.pkg.tar.zst
loading packages...
warning: linux-5.8.9.arch2-1 is up to date -- reinstalling
warning: linux-headers-5.8.9.arch2-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...
Packages (2) linux-5.8.9.arch2-1 linux-headers-5.8.9.arch2-1
Total Installed Size: 199.79 MiB
Net Upgrade Size: 0.00 MiB
:: Proceed with installation? [Y/n] Y
(2/2) checking keys in keyring [##########################################################] 100%
(2/2) checking package integrity [##########################################################] 100%
(2/2) loading package files [##########################################################] 100%
(2/2) checking for file conflicts [##########################################################] 100%
(2/2) checking available disk space [##########################################################] 100%
:: Running pre-transaction hooks...
(1/1) Remove upgraded DKMS modules
==> dkms remove --no-depmod -m system76-acpi -v 1.0.1 -k 5.8.9-arch2-1
==> dkms remove --no-depmod -m system76-io -v 1.0.1 -k 5.8.9-arch2-1
==> dkms remove --no-depmod -m system76 -v 1.0.9 -k 5.8.9-arch2-1
==> dkms remove --no-depmod -m nvidia -v 455.28 -k 5.8.9-arch2-1
:: Processing package changes...
(1/2) reinstalling linux [##########################################################] 100%
(2/2) reinstalling linux-headers [##########################################################] 100%
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating module dependencies...
(3/4) Install DKMS modules
==> dkms install --no-depmod -m system76-acpi -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76-io -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76 -v 1.0.9 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m nvidia -v 455.28 -k 5.8.9-arch2-1
==> depmod 5.8.9-arch2-1
(4/4) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful
And now a restart with fingers crossed… It works!
Looking back on all of this, the funny thing is I saw all the signs of people having problems on r/archlinux and the wiki, but upgraded anyway, only to get bit in the butt by a completely unexpected issue. People were reporting that what was broken was CUDA and OpenCL, so I wasn’t too worried, and I didn’t have a reason to be, games worked just fine. I definitely didn’t expect is that 5.9 might break my bluetooth driver…
This actually reminds me of a comment on Hacker News about linux viability in consumer computing, specifically this quote:
I just literally lived this person’s experience.
Obviously, I’m going to keep cracking on with Arch though, 90% of the time things just work in this day and age (and with the hard work of MANY selfless contributors) I’ve found it to be awesome to run (it’s very stable), and I run it everywhere now. I even briefly ran it on servers, but had some issues (which I think was around the inability to update grub, because of Hetzner’s in-built software RAID assumptions but that’s a story for another time).