Intermittent problem with MetalLB

[stblassitude]

I'm running a two-node cluster on Debian buster, using MetalLB in Layer 2 mode. Things appear to be working, but every other request fails on the network level.

For example running curl to a whoami pod with a one second delay:

~$ for i in $(jot 60); do curl whoami.app.example.net >/dev/null 2>&1; echo $?; sleep 1; done
0
0
7
0
0
7
7
...

Before I upgraded to Buster about three months ago, everything was working as expected.

The controller log shows these in one second intervals:

$ kubectl -n metallb-system logs -f controller-7dcc8764f4-s9wth | tail -10
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"kube-system/traefik","ts":"2022-02-06T18:45:12.40093726Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"172.27.64.20","level":"info","msg":"IP address assigned by controller","service":"kube-system/traefik","ts":"2022-02-06T18:45:13.037387997Z"}
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"kube-system/traefik","ts":"2022-02-06T18:45:13.04176137Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"172.27.64.20","level":"info","msg":"IP address assigned by controller","service":"kube-system/traefik","ts":"2022-02-06T18:45:14.305359392Z"}
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"kube-system/traefik","ts":"2022-02-06T18:45:14.309417284Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"172.27.64.20","level":"info","msg":"IP address assigned by controller","service":"kube-system/traefik","ts":"2022-02-06T18:45:14.946401738Z"}
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"kube-system/traefik","ts":"2022-02-06T18:45:15.017604435Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"172.27.64.20","level":"info","msg":"IP address assigned by controller","service":"kube-system/traefik","ts":"2022-02-06T18:45:16.204060606Z"}
{"caller":"level.go:63","event":"serviceUpdated","level":"info","msg":"updated service object","service":"kube-system/traefik","ts":"2022-02-06T18:45:16.21365406Z"}
{"caller":"level.go:63","event":"ipAllocated","ip":"172.27.64.20","level":"info","msg":"IP address assigned by controller","service":"kube-system/traefik","ts":"2022-02-06T18:45:16.836676587Z"}

But there are no error messages (also not on startup of the controller pod), and the two speakers only show occasional messages like

W0206 19:05:56.047519       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
{"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-02-06T19:10:24.651543483Z"}
W0206 19:14:46.049488       1 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
{"caller":"level.go:63","level":"info","msg":"triggering discovery","op":"memberDiscovery","ts":"2022-02-06T19:15:25.827233569Z"}

I'm running v1.21.5+k3s2 with MetalLB quay.io/metallb/controller:v0.11.0

Any suggestions?

[stblassitude]

I've set up the cluster with--flannel-backend=host-gw.

[stblassitude]

It turns out I lost two important lines in the systemd unit when I upgraded. I'm running my own copy of traefik, but on the upgrade, the bundled version was re-enabled, and the service for traefik was somehow being pummled. Making sure to disable traefik and servicelb in /etc/systemd/system/k3s.service fixed it:

ExecStart=/usr/local/bin/k3s \
    server \
        '--flannel-backend=host-gw' \
        --no-deploy=traefik \
        --no-deploy=servicelb

[iwilltry42]

Hi @stblassitude , glad you solved this! :)
One question though: was this related to k3d? Given that you're using the service file here I guess you used plain K3s? 🤔

[stblassitude]

k3s, not k3d. I was slightly confused about the right name.