tl;dr - Adding a rebuilt node with a different IP under an already-used node name did not sit well with Calico in my setup. I could see the problem (mismatched IP address information in Node resources), but I couldn’t fix it – ultimately I gave the new node a new never-before-used name (i.e.. worker-< n+1 > instead of worker-2).

Recently during an expansion and some shifting of my k8s cluster, I found out (the hard way, of course) that if you add a new node that has the same name (let’s say worker-1 or worker-1.your-k8s-cluster), but a different IP address from one that previously existed, It’s possible that Calico may get a bit confused and fail to set up networking between other nodes and that node. As the CRDs that are created by Calicio can persist in a cluster, I’m fairly certain that was the problem I ran into.

In general, ifyou’re going to add a new node that has the same name but a different IP address as an old node, you’re going to want to check and see if Calico hasn’t become misconfigured when you do.

Symptoms

In general, none of the workloads scheduled on the new node will have properly working network access via Calico.

A specific example is Pods scheduled on the new node not being able to successfully query DNS (internal or external) – they won’t be able to reach the CoreDNS (kubedns) instance as no inter-node traffic will be working work.

Diagnosis

Throughout this diagnosis we’re going to assume that my Node (the Kubernetes Node resource) is called node-2.eu-central-1. This isn’t the Node this issue happened on, but it will serve for demonstrative purposes.

Check the calico-node DaemonSet output for the new node

To make sure that you’ve fixed the issue, you’re going to want to check the logs in the calico-node for the new node. Check the kube-system namespace for pods owned by the calico-node DaemonSet:

$ k get pods -n kube-system
NAME                                       READY   STATUS      RESTARTS   AGE
calico-kube-controllers-78659cc5bf-mn58n   1/1     Running     6          10d
calico-node-4qs6d                          1/1     Running     7          10d
calico-node-vbc9k                          1/1     Running     0          10d
... more pods ...

The logs in there should look much better and contain no errors

Here is what the logs for calico-node look like on a node (I can’t remember if this was the node or another node) that is having this issue (I’ve added some spaces for emphasis):

2021-09-20 14:29:38.733 [INFO][80] felix/conntrack.go 90: Removing conntrack flows ip=10.244.128.10
2021-09-20 14:29:38.740 [INFO][80] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"email/haraka-4nzzc", EndpointId:"eth0"} ipVersion=0x4 status="up"
2021-09-20 14:29:38.740 [INFO][80] felix/status_combiner.go 98: Reporting combined status. id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"email/haraka-4nzzc", EndpointId:"eth0"} status="up"
2021-09-20 14:29:38.937 [INFO][80] felix/int_dataplane.go 1430: Received *proto.WireguardEndpointUpdate update from calculation graph msg=hostname:"node-4.eu-central-1" public_key:"<redacted>" interface_ipv4_addr:"10.244.128.1"
2021-09-20 14:29:39.046 [INFO][80] felix/wireguard.go 570: Public key out of sync or updated ourPublicKey=<redacted>
2021-09-20 14:29:39.067 [INFO][78] tunnel-ip-allocator/node.go 88: Error updating Node resource error=Operation cannot be fulfilled on nodes "node-4.eu-central-1": the object has been modified; please apply your changes to the latest version and try again
2021-09-20 14:29:39.067 [INFO][78] tunnel-ip-allocator/allocateip.go 563: Error updating node node-4.eu-central-1: %!s(<nil>). Retrying. type="ipipTunnelAddress"


2021-09-20 14:29:39.244 [WARNING][80] felix/vxlan_mgr.go 354: Failed configure VXLAN tunnel device, retrying... error=Unable to find parent interface with address YYY.YYY.YYY.YYY


2021-09-20 14:29:39.276 [INFO][80] felix/int_dataplane.go 1430: Received *proto.WireguardEndpointUpdate update from calculation graph msg=hostname:"node-4.eu-central-1" public_key:"<redacted>" interface_ipv4_addr:"10.244.128.1"
2021-09-20 14:29:39.301 [INFO][80] felix/int_dataplane.go 1430: Received *proto.WireguardEndpointUpdate update from calculation graph msg=hostname:"node-4.eu-central-1" public_key:"<redacted>" interface_ipv4_addr:"10.244.128.1"
2021-09-20 14:29:39.363 [INFO][80] felix/wireguard.go 570: Public key out of sync or updated ourPublicKey=<redacted>

It’s a surprisingly small indicator, but that WARNING is the smoking gun – not being able to find the node with the OLD IP address (YYY.YYY.YYY.YYY) means the VXLAN tunnel couldn’t be built, and cluster networking didn’t quite work for the given node.

Confirming Kubernetes node information with kubectl

To futher confirm if you’re seeing the specific issue I was, you’re going to want to cross-reference the Node information maintained by two systems – Kubernetes itself and Calico. Check out the Kubernetes native Node object:

$ kubectl get node <the new node> -o yaml

What you want to confirm is that status.addresses[0] (the type is likely InternalIP). Here’s what that looks like:

apiVersion: v1
kind: Node
metadata:
  annotations:
    # ... a bunch of annotations ...
    projectcalico.org/IPv4Address: XXX.XXX.XXX.XXX # redacted
    projectcalico.org/IPv4VXLANTunnelAddr: 10.244.192.64
    projectcalico.org/IPv4WireguardInterfaceAddr: 10.244.192.65
    projectcalico.org/WireguardPublicKey: <redacted>
  creationTimestamp: "2021-10-24T14:33:08Z"
  labels:
    # ... a bunch of labels ...
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: node-2.eu-central-1
    kubernetes.io/os: linux
  name: node-2.eu-central-1
  resourceVersion: "141344346"
  uid: 84a2ccb4-3cee-4758-8da0-5db6601cc5c4
spec:
  podCIDR: 10.244.6.0/24
  podCIDRs:
  - 10.244.6.0/24
status:
  addresses:
  - address: XXX.XXX.XXX.XXX # here's the IP address (should match the calico annotation above)!
    type: InternalIP
  - address: node-2.eu-central-1 # here's your node's name
    type: Hostname
  # ... lots more information

Note that a few things – you’ve got your node status, what calico thinks your node’s IPv4 Address is, where it’s doing VXLAN tunneling, etc.

Confirming the Calico node information with calicoctl

On the Calico side, we’re going to take a look at Calico’s Node resource and see what it says for the node in question. Download calicoctl (I prefer to use it as a kubectl plugin) and use it to check your node to confirm the information:

$ calicoctl get node <your node name> -o yaml

If you have the problem I’m describing in this post, you’ll see that the information output by calicoctl outputs are the old node’s information:

apiVersion: projectcalico.org/v3 # <----- this is a *calico* Node resource
kind: Node
metadata:
  annotations:
    projectcalico.org/kube-labels: '{.... JSON object holding node labels ...}' # redacted
  creationTimestamp: "2021-10-24T14:33:08Z"
  labels:
    # ... labels for the node ...
    kubernetes.io/hostname: node-2.eu-central-1
    kubernetes.io/os: linux
  name: node-2.eu-central-1
  resourceVersion: "141344346"
  uid: 84a2ccb4-3cee-4758-8da0-5db6601cc5c4
spec:
  addresses:
  - address: XXX.XXX.XXX.XXX/32
  - address: XXX.XXX.XXX.XXX
  bgp:
    ipv4Address: XXX.XXX.XXX.XXX/32
  ipv4VXLANTunnelAddr: 10.244.192.64
  orchRefs:
  - nodeName: node-2.eu-central-1
    orchestrator: k8s
  wireguard:
    interfaceIPv4Address: 10.244.192.65
status:
  podCIDRs:
  - 10.244.6.0/24
  wireguardPublicKey: <redacted> # yeah it's the public key but anyway

Drastically less information here, but much easier to parse/understand! This resoruce shows us what Calico thinks are the addresses this node can be accessed at, the address it uses for BGP, wireguard interface IP address, etc.

Here’s the crux of the diagnosis – I was finding that the Calico-created-and-managed Node Resource was differing from the Kubernetes-created-and-managed Node resource. They were not agreeing on IP address and this meant that commmunication between other nodes and this node just wasn’t working.

Things I tried that I thought would fix it

Here are some things I tried:

• Change the InternalIP on the node itself (--node-ip kubelet worker arg)
  • Update etc/hosts or any other resolution details that would convince your master nodes to pick a different IP
    • For me this ment editing /etc/hosts
• Double check that both k8s and calico report the right IP
• Try to use the calico docs to manually configure the node

All of those things sound like great ideas, and I tried them, but in the end I just couldn’t get Kubernetes and Calico to agree on the IP for the stale-named new node.

The real fix: Rebuild and rename the new node.

Ultimately, none of the things I tried worked, and I simply rebuilt & renamed the node and re-added it to the cluster.

Validating the fix

Just a reminder – the fix that worked for me was changing the node name. It’s possible that I was just barely off with some of the other remediations, but I didn’t want to dig too far deeper into Calico’s code to find exactly what was causing the wrong values to be propagated, when the situation is fairly rare and the remediation is so easy.

To make sure that you’ve fixed the issue, you’re going to want to:

• Check the logs in the calico-node for the new node – you should have no warnings or errors (obviously, especially failures to configure VXLAN tunnels)
• Ensure that you can access the network from a workload runnig on the new node:

  $ kubectl run -it --rm \
      test \
      --image=alpine \
      --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"<node name>"}}}'
  

Wrapup

This outline has been sitting around for a while but I’m glad I was able to write it up finally – whenever I add new nodes to a cluster or rebuild nodes I always give them a new ID at this point. It’s not quite the same as completely treating them as cattle (I do still write down the names and the mappings are fairly static/hand-updated), but it at least helps me avoid this issue.

Thanks for reading, hopefully this helps someone out there. If you’ve found that you were able to get past this problem and correctly configure Calico (or know where I went wrong in trying to reconfigure it), please drop me a line!