It's been a while since this was posted. Hopefully the information in here is still useful to you (if it isn't please let me know!). If you want to get the new stuff as soon as it's out though, sign up to the mailing list below.
Join the Mailing list
+
tl;dr - Bluetooth died after a recent upgrade to Linux kernel 5.9.1 on Arch which people noticed nVidia didn’t support – I upgraded anyway, graphics were fine, but Bluetooth was not.
So very recently a release of NVIDIA’s drivers was confirmed not to work with Linux kernel 5.9.x. It’s been discussed widely on the internet (by those affected) so I won’t go into it here:
Read those sources if you want more context, but the gist as I understand is:
nvidia_uvm (?) and other modules that will fix/re-enable thingsI’m not NVIDIA’s greatest fan nor have I been greatly wronged by NVIDIA, but I do run (and am very happy with) a Oryx Pro so that’s what’s in my laptop, so I’m affected. If I was building a desktop today I’d pick a DAN-style Mini-ITX case, a Ryzen Threadripper (what does htop even look like with that many cores??) and a Radeon
Turns out I can actually play CS:GO (the only game I play these days) just fine, but i my case, graphics weren’t the problem – Bluetooth was.
bluez?user@oryxpro $ systemctl status bluetooth
● bluetooth.service - Bluetooth service
Loaded: loaded (/usr/lib/systemd/system/bluetooth.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2020-10-23 20:04:42 JST; 14h ago
Docs: man:bluetoothd(8)
Main PID: 82921 (bluetoothd)
Status: "Running"
Tasks: 1 (limit: 38142)
Memory: 896.0K
CPU: 17ms
CGroup: /system.slice/bluetooth.service
└─82921 /usr/lib/bluetooth/bluetoothd
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEMinCon>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEMaxCon>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEConnec>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEConnec>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: src/main.c:parse_controller_config() Key file does not have key “LEAutoco>
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Starting SDP server
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Bluetooth management interface 1.18 initialized
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Endpoint registered: sender=:1.37 path=/MediaEndpoint/A2DPSink/sbc
Oct 23 20:04:42 oryxpro bluetoothd[82921]: Endpoint registered: sender=:1.37 path=/MediaEndpoint/A2DPSource/sbc
Oct 23 20:05:03 oryxpro bluetoothd[82921]: Failed to set mode: Failed (0x03)
If you try and run bluetoothctl and do things like power on or connect <device> you’re going to get a bunch of errors telling you that bluez (the Linux bluetooth stack) is gone/wonky. I did a small amount of searching and didn’t find too much:
So I think it might be kernel problem (since that was the last big change).For now I’m going to have to downgrade to kernel 5.8.9 – although not for the reasons everyone else is.
The excellent Arch Wiki has you covered as usual, read there first before some random blog on the internet (where you are now).
For the impatient, the basic gist of the relevant sections of that wiki is:
/var/cache/pacman/pkgpacman -Ulinux and linux-headers, but watch out for other packages that depended on the newer/older kernel (HINT FROM THE FUTURE: I’m about to run head first into this)I ran into a snag when I tried to simply downgrade linux and linux-headers though:
user@oryxpro $ cd /var/cache/pacman/pkg/
user@oryxpro $ sudo pacman -U linux-5.8.9.arch2-1-x86_64.pkg.tar.zst linux-headers-5.8.9.arch2-1-x86_64.pkg.tar.zst
loading packages...
warning: downgrading package linux (5.9.1.arch1-1 => 5.8.9.arch2-1)
warning: downgrading package linux-headers (5.9.1.arch1-1 => 5.8.9.arch2-1)
resolving dependencies...
looking for conflicting packages...
Packages (2) linux-5.8.9.arch2-1 linux-headers-5.8.9.arch2-1
Total Installed Size: 199.79 MiB
Net Upgrade Size: -2.51 MiB
:: Proceed with installation? [Y/n] Y
(2/2) checking keys in keyring [######################################################] 100%
(2/2) checking package integrity [######################################################] 100%
(2/2) loading package files [######################################################] 100%
(2/2) checking for file conflicts [######################################################] 100%
(2/2) checking available disk space [######################################################] 100%
warning: could not get file information for usr/lib/modules/5.9.1-arch1-1/kernel/drivers/platform/x86/system76_acpi.ko.xz
:: Running pre-transaction hooks...
(1/2) Removing linux initcpios...
(2/2) Remove DKMS modules
==> dkms remove --no-depmod -m system76-acpi -v 1.0.1 -k 5.9.1-arch1-1
==> dkms remove --no-depmod -m system76-io -v 1.0.1 -k 5.9.1-arch1-1
==> dkms remove --no-depmod -m system76 -v 1.0.9 -k 5.9.1-arch1-1
==> depmod 5.9.1-arch1-1
:: Processing package changes...
(1/2) downgrading linux [######################################################] 100%
(2/2) downgrading linux-headers [######################################################] 100%
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating module dependencies...
(3/4) Install DKMS modules
==> dkms install --no-depmod -m system76-acpi -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76-io -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76 -v 1.0.9 -k 5.8.9-arch2-1
==> depmod 5.8.9-arch2-1
(4/4) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> ERROR: module not found: `nvidia'
==> ERROR: module not found: `nvidia_modeset'
==> ERROR: module not found: `nvidia_uvm'
==> ERROR: module not found: `nvidia_drm'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> WARNING: errors were encountered during the build. The image may not be complete.
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> ERROR: module not found: `nvidia'
==> ERROR: module not found: `nvidia_modeset'
==> ERROR: module not found: `nvidia_uvm'
==> ERROR: module not found: `nvidia_drm'
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> WARNING: errors were encountered during the build. The image may not be complete.
error: command failed to execute correctly
Well of course, nothing ever works the first time, but it is nice to see that the system76-* drivers build just fine with the older kernel and get loaded & unloaded (I sure would have not liked to restart and have nothing supported properly), but there’s a problem with the nvidia ones!
So I probably need to downgrade the nvidia drivers first right?
NARRATOR: He didn’t.
WARNING FROM THE FUTURE: In case it wasn’t obvious from the previous section’s ending, this wasn’t actually necessary,
nvidia-dkmsis what was needed since it’s the reason that nvidia didn’t do a build during the downgrade phase.
I don’t use nvidia-dkms (I don’t have it installed) but when I search /var/cache/pacman/pkg for "nvidia" I see the combination of nvidia-utils and nvidia packages. As you might have guessed nvidia package version numbers don’t map to kernel numbers, but after a little searching I found a thread that makes me think maybe I should try the 450 series of patches (maybe those were the ones that worked with 4.8.9 kernels?
user@oryxpro $ cd /var/cache/pacman/pkg/
user@oryxpro $ sudo pacman -U lib32-nvidia-utils-450.66-1-x86_64.pkg.tar.zst nvidia-utils-450.66-1-x86_64.pkg.tar.zst nvidia-450.66-22-x86_64.pkg.tar.zst
loading packages...
warning: downgrading package lib32-nvidia-utils (455.28-1 => 450.66-1)
warning: downgrading package nvidia-utils (455.28-1 => 450.66-1)
warning: downgrading package nvidia (455.28-4 => 450.66-22)
resolving dependencies...
looking for conflicting packages...
Packages (3) lib32-nvidia-utils-450.66-1 nvidia-450.66-22 nvidia-utils-450.66-1
Total Installed Size: 362.86 MiB
Net Upgrade Size: -71.25 MiB
:: Proceed with installation? [Y/n] Y
(3/3) checking keys in keyring [##########################################################] 100%
(3/3) checking package integrity [##########################################################] 100%
(3/3) loading package files [##########################################################] 100%
(3/3) checking for file conflicts [##########################################################] 100%
(3/3) checking available disk space [##########################################################] 100%
:: Processing package changes...
(1/3) downgrading nvidia-utils [##########################################################] 100%
If you run into trouble with CUDA not being available, run nvidia-modprobe first.
(2/3) downgrading lib32-nvidia-utils [##########################################################] 100%
(3/3) downgrading nvidia [##########################################################] 100%
:: Running post-transaction hooks...
(1/4) Creating system user accounts...
(2/4) Reloading system manager configuration...
(3/4) Arming ConditionNeedsUpdate...
(4/4) Updating module dependencies...
Well that worked out just fine, maybe I’m good to go now?
NARRATOR: He wasn’t.
So now that I have nvidia downgraded, hopefully kernel downgrade will go smoothly… aaaand it didn’t, same error.
At this point, there must be a misunderstanding in my knowledge – in particular what nvidia-dkms actually is/does and whether I should have it. A friendly forum poster shed some light on it for me (without diving into the actual Arch Wiki).
So what nvidia-dkms will do is enable me to rebuild relevant nvidia driver(s) every time a new custom kernel (which mine is indeed somewhat customized) is built. This is exactly what I need, I need the nvidia driver to get rebuilt just like System76’s drivers.
At first try though, it looks like it has a problem with the downgraded nvidia drivers:
user@oryxpro $ sudo pacman -S nvidia-dkms
[sudo] password for user:
resolving dependencies...
looking for conflicting packages...
error: failed to prepare transaction (could not satisfy dependencies)
:: installing nvidia-utils (455.28-1) breaks dependency 'nvidia-utils=450.66' required by nvidia
So lets… upgrade to downgrade? I mean technically the 455 series didn’t produce a problem with kernels < 5.9 AFAIK, so maybe I could just use the 455 without downgrading anyway.
user@oryxpro $ sudo pacman -S nvidia
<successful, so output redacted>
user@oryxpro $ sudo pacman -S nvidia-dkms
resolving dependencies...
looking for conflicting packages...
Packages (1) nvidia-dkms-455.28-1
Total Download Size: 18.56 MiB
Total Installed Size: 40.99 MiB
:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
nvidia-dkms-455.28-1-x86_64 18.6 MiB 4.94 MiB/s 00:04 [##########################################################] 100%
(1/1) checking keys in keyring [##########################################################] 100%
^N(1/1) checking package integrity [##########################################################] 100%
(1/1) loading package files [##########################################################] 100%
(1/1) checking for file conflicts [##########################################################] 100%
(1/1) checking available disk space [##########################################################] 100%
:: Processing package changes...
(1/1) installing nvidia-dkms [##########################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Install DKMS modules
==> dkms install --no-depmod -m nvidia -v 455.28 -k 5.8.9-arch2-1
==> depmod 5.8.9-arch2-1
OK so now what I’m expecting is that when I try to downgrade the kernel, the nvidia module will get rebuilt right on the spot, let’s see it happen:
user@oryxpro $ sudo pacman -U linux-5.8.9.arch2-1-x86_64.pkg.tar.zst linux-headers-5.8.9.arch2-1-x86_64.pkg.tar.zst
loading packages...
warning: linux-5.8.9.arch2-1 is up to date -- reinstalling
warning: linux-headers-5.8.9.arch2-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...
Packages (2) linux-5.8.9.arch2-1 linux-headers-5.8.9.arch2-1
Total Installed Size: 199.79 MiB
Net Upgrade Size: 0.00 MiB
:: Proceed with installation? [Y/n] Y
(2/2) checking keys in keyring [##########################################################] 100%
(2/2) checking package integrity [##########################################################] 100%
(2/2) loading package files [##########################################################] 100%
(2/2) checking for file conflicts [##########################################################] 100%
(2/2) checking available disk space [##########################################################] 100%
:: Running pre-transaction hooks...
(1/1) Remove upgraded DKMS modules
==> dkms remove --no-depmod -m system76-acpi -v 1.0.1 -k 5.8.9-arch2-1
==> dkms remove --no-depmod -m system76-io -v 1.0.1 -k 5.8.9-arch2-1
==> dkms remove --no-depmod -m system76 -v 1.0.9 -k 5.8.9-arch2-1
==> dkms remove --no-depmod -m nvidia -v 455.28 -k 5.8.9-arch2-1
:: Processing package changes...
(1/2) reinstalling linux [##########################################################] 100%
(2/2) reinstalling linux-headers [##########################################################] 100%
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating module dependencies...
(3/4) Install DKMS modules
==> dkms install --no-depmod -m system76-acpi -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76-io -v 1.0.1 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m system76 -v 1.0.9 -k 5.8.9-arch2-1
==> dkms install --no-depmod -m nvidia -v 455.28 -k 5.8.9-arch2-1
==> depmod 5.8.9-arch2-1
(4/4) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 5.8.9-arch2-1
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [modconf]
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: xhci_pci
-> Running build hook: [filesystems]
-> Running build hook: [keyboard]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful
And now a restart with fingers crossed… It works!
Looking back on all of this, the funny thing is I saw all the signs of people having problems on r/archlinux and the wiki, but upgraded anyway, only to get bit in the butt by a completely unexpected issue. People were reporting that what was broken was CUDA and OpenCL, so I wasn’t too worried, and I didn’t have a reason to be, games worked just fine. I definitely didn’t expect is that 5.9 might break my bluetooth driver…
This actually reminds me of a comment on Hacker News about linux viability in consumer computing, specifically this quote:
Do we have to get cool people to use Linux?
Every time this comes up I am here to offer a perspective perhaps few have.
I started using Linux before it was 1.0. It was so long ago, quite a few ppl here weren’t even alive (and then again, I am sure there a few here who started in ‘91 or ‘92). For a while, only at university department level servers. Then dual booting it at home, with Windows 2000 (maybe even Win98 at the tail end of that, I don’t remember) and having Linux as my daily driver first on desktop then on laptop 2004-end of 2017.
After this long, I must say, anyone using Linux on a laptop has Stockholm syndrome. You can choose between Ubuntu where infrequent OS updates shatter your system so throughly you can’t work for days or you can use Arch Linux which rolls and breaks smaller parts of your system from time to time, mostly scanner, printer, Bluetooth. That’s how it goes. https://xkcd.com/619/ this is 1000% true. Oh yes, I use Linux on servers to this day, it’s great.
Sure, if your usage is plain home wifi and browsing the web, basically something just a tiny bit beyond a Chromebook, it might work but heaven forbid you’d try to connect to some funny enterprise VPN or even worse, enterprise wifi or as I mentioned, have a Bluetooth speaker or an MFD.
And there’s no help. None. Nobody knows any more what’s going on. We had a pretty good handle on how the various parts of the system fit together because it was damn simple but post-systemd noone understands it any more or if they do, they don’t share. I posted this more than once, my favorite example here is the DisplayLink adapters (which are awful but they are often a necessary awful) and people said, read the Arch wiki page, well, back then, even like three years after I did, the meat of that page was what I could scrape together from mailing list and blog posts of the relevant driver authors but that’s it. I certainly didn’t understand it and people blindly copypasting out of the wiki didn’t either and if you hit a problem, bam, who is there to help? Or to the previous point and this no help together: the IT lead at my workplace asked some years ago how to make the F5 VPN w/ two factor work on Linux and the deafening silence is the answer. I managed to make it work by running an old enough Firefox as root (!) which could run the F5 NPAPI extension. It’s just plain nuts. Yes, F5 should have provided a native Linux client which supported 2FA on Linux but they didn’t. It doesn’t matter whose fault it is but at the end of the day, I am out of work if I can’t log into the VPN.
I am running Windows 10 since 2018 January. Not a problem. I ran O&o shutup, I use Classic Shell, so I never even see the Win 10 Start menu, I use WSL and – everything just works. I have no problems with the system installing security updates at 5am, I set my system once so it only installs those, big updates are postponed by six month or forever and the whole thing just. works. Yes, Windows Subsystem For Linux, because the Linux command line tool are just great. It’s everything else that is less great.
https://www.reddit.com/r/linuxquestions/comments/iej5y6/netf… look at this and despair. Once again: I don’t care, I have translated Doctorow’s speech to Hungarian to spread it, to help fight DRM, no matter how little, even that was 16 years ago, I fought yeah, but the reality is, it’s here and I don’t want to bother with this. Look here, Linux people say, it-works-or-at-least-ought-to-with-this-workaround this is what I am saying: people will bend over backwards to defend their OS… I am not sure whether I am able to get my point across. sigh.
I just literally lived this person’s experience.
Obviously, I’m going to keep cracking on with Arch though, 90% of the time things just work in this day and age (and with the hard work of MANY selfless contributors) I’ve found it to be awesome to run (it’s very stable), and I run it everywhere now. I even briefly ran it on servers, but had some issues (which I think was around the inability to update grub, because of Hetzner’s in-built software RAID assumptions but that’s a story for another time).