Docker on Arch Linux - docker0 just doesn't seem to want it's IPv4 address

Recent instability running Docker on Arch -- docker0 just doesn't want to keep it's IPv4 address.

vados

4 minute read

tl;dr - My setup of Docker on Arch Linux is having some issues, around docker0 not properly holding on to it’s IPV4 addresses (listed as inet in ip addr output). I originally though it was a problem with Alpine CDNs, but it was actually docker0 throwing up repeatedly. Short term work around I’ve found is to just create the missing link again, w/ sudo ip addr add 172.17.0.1/16 dev docker0.

I’ve recently run into a bit of a problem working with on docker containers on Arch Linux, the first symptoms that I would see would be that network requests that went out from inside containers during build would stop working (when they are perfectly fine running on my host computer).

At first, I was fully convinced that maybe the Alpine CDN was down, but it absolutely was not. Running the commands from my computer (and wgeting some APKINDEX.tar.gz files as well) was enough to convince me that hunch was wrong.

The next idea I had was that maybe it was just the docker service being weird, so I restarted it (sudo systemctl restart docker), and the first external request that the container made seemed to work, but the second and third would go back to failing.

After lots of looking around the internet and coming upon some Docker Arch forum users, as well as some Arch users on the forums, I wondered if I had somehow done something wrong – surely these problems were already fixed?

It turns out that the real problem is that the docker0 network interface (created by docker) just didn’t want to hold on to it’s inet link. Check out this output:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 30:5a:3a:7a:20:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.85/24 brd 192.168.1.255 scope global enp3s0
       valid_lft forever preferred_lft forever
    inet6 fe80::df26:31f6:9408:ee75/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:bb:2c:8b:60 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:bbff:fe2c:8b60/64 scope link
       valid_lft forever preferred_lft forever
41: veth5ebab90@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 86:ad:fe:6b:f7:48 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::84ad:feff:fe6b:f748/64 scope link
       valid_lft forever preferred_lft forever

I ran ip addr while some other commands in the container were running (apk update inside an alpine container), and notices the virtual connections (veth<whatever>) coming and going, but also noticed that after a while, under docker0, the inet link would dissappear (and not reappear). I noticed that after that happened

Of course, my first thought was did I read/properly follow the manual for Docker on Arch? It actually has a section on the docker0 bridge getting no IP/internet access. While setting up Docker for Arch the first time, I did absolutely check the documentation and follow it, but double checking wouldn’t hurt, so I did – I even used the manually-defined network instructions since it was “known-good”.

The dirty dirty workaround

While I still haven’t fully tracked down the issue (assuming it’s something I’m even capable of tracking down), my current work around is this.. I just create the link that goes down:

sudo ip addr add 172.17.0.1/16 dev docker0

This is a terrible workaround (I literally have to watch -n 1 ip addr and run that command when the link inevitably goes down), but works in the best straight-forward almost neanderthalic fashion.

I also explored whether the problem might be related to UFW & docker, but it doesn’t seem likely (and turning off UFW didn’t help). Even still, I ensured that the UFW DEFAULT_FORWARD_POLICY was set to “ACCEPT”.

I have no idea when this will get fixed, or if I’ll find a new more acceptable work-around/local fix but just wanted to put this out there for anyone wondering why their containers suddenly aren’t building anymore and are running Arch.

Did you find this read beneficial? Send me questions/comments/clarifciations.
Want my expertise on your team/project? Send me interesting opportunities!