DDG search drop in powered by ddg.patdryburgh.com
tl;dr - My setup of Docker on Arch Linux is having some issues, around docker0 not properly holding on to it’s IPV4 addresses (listed as inet in ip addr output). I originally though it was a problem with Alpine CDNs, but it was actually docker0 throwing up repeatedly. Short term work around I’ve found is to just create the missing link again, w/ sudo ip addr add 172.17.0.1/16 dev docker0.
I got an email from a reader which mentioned that this blog post actually does not have the fix spelled out. Apologies to those that might have stumbled across this and not been able to figure out how I fixed it! The fix is actually in a follow up post to this one.
Thanks to the help of Garrett, it looks like there might be a long-term fix (along with a good explanation) of what's actually wrong. Basically what's happening is that `systemd-networkd` is trying to manage the `docker0` interface. You can confirm this by running `networkctl`:
<code>
networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier configured 2 en0 ether routable configured 3 docker0 ether no-carrier configuring
Double check that you networkctl configuration is not trying to manage the docker0 interface, so the output looks more like:
$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
.... other links ....
3 docker0 ether no-carrier unmanaged
Thanks again to Garettt for figuring out what was wrong with his setup and then emailing me so I could update the blog post!
</p>
This problem is still around somehow, here's a band-aid:
<code>
sudo watch -n1 'ip addr show docker0 | grep "inet " || ip addr add 172.17.0.1/16 dev docker0'
</code>
This script will check every second to ensure that the IPv4 address on `docker0`is available, if not it will make it. Just run this before you do `docker build` stuff and it should be good
</p>
The original post follows below
I’ve recently run into a bit of a problem working with on docker containers on Arch Linux, the first symptoms that I would see would be that network requests that went out from inside containers during build would stop working (when they are perfectly fine running on my host computer).
At first, I was fully convinced that maybe the Alpine CDN was down, but it absolutely was not. Running the commands from my computer (and wgeting some APKINDEX.tar.gz files as well) was enough to convince me that hunch was wrong.
The next idea I had was that maybe it was just the docker service being weird, so I restarted it (sudo systemctl restart docker), and the first external request that the container made seemed to work, but the second and third would go back to failing.
After lots of looking around the internet and coming upon some Docker Arch forum users, as well as some Arch users on the forums, I wondered if I had somehow done something wrong – surely these problems were already fixed?
It turns out that the real problem is that the docker0 network interface (created by docker) just didn’t want to hold on to it’s inet link. Check out this output:
$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 30:5a:3a:7a:20:c9 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.85/24 brd 192.168.1.255 scope global enp3s0
valid_lft forever preferred_lft forever
inet6 fe80::df26:31f6:9408:ee75/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:bb:2c:8b:60 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:bbff:fe2c:8b60/64 scope link
valid_lft forever preferred_lft forever
41: veth5ebab90@if40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 86:ad:fe:6b:f7:48 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::84ad:feff:fe6b:f748/64 scope link
valid_lft forever preferred_lft forever
I ran ip addr while some other commands in the container were running (apk update inside an alpine container), and notices the virtual connections (veth<whatever>) coming and going, but also noticed that after a while, under docker0, the inet link would dissappear (and not reappear). I noticed that after that happened
Of course, my first thought was did I read/properly follow the manual for Docker on Arch? It actually has a section on the docker0 bridge getting no IP/internet access. While setting up Docker for Arch the first time, I did absolutely check the documentation and follow it, but double checking wouldn’t hurt, so I did – I even used the manually-defined network instructions since it was “known-good”.
While I still haven’t fully tracked down the issue (assuming it’s something I’m even capable of tracking down), my current work around is this.. I just create the link that goes down:
sudo ip addr add 172.17.0.1/16 dev docker0
This is a terrible workaround (I literally have to watch -n 1 ip addr and run that command when the link inevitably goes down), but works in the best straight-forward almost neanderthalic fashion.
I also explored whether the problem might be related to UFW & docker, but it doesn’t seem likely (and turning off UFW didn’t help). Even still, I ensured that the UFW DEFAULT_FORWARD_POLICY was set to “ACCEPT”.
I have no idea when this will get fixed, or if I’ll find a new more acceptable work-around/local fix but just wanted to put this out there for anyone wondering why their containers suddenly aren’t building anymore and are running Arch.