tl;dr:Consumer routers can mess with your Linux bridge interface. ebtables can eliminate unwanted packets before they’re polluting the bridge’s “rx dropped” stats.
Netdata: automagic monitoring
On my home server/NAS my LXD container usage has grown considerably over the last year. It’s come to the point that I felt the need to have a monitoring solution for this “datacenter”-in-a-box: The 1-2 dozen containers that encapsulate various server daemons all have their own virtual network interface(s) and LAN-valid IP addresses allocated by my LAN’s DHCP server which itself is also running happily inside one of the containers...
After a quick sudo apt install netdata (on Ubuntu 18.04 LTS) I was presented with a beautiful and almost overwhelmingly complete web-interface that tracked and visualized my rather complex setup in real-time (even including the UPS’s status), all that without one iota of config-editing. Nice.
Except that...
My bridge is dropping packets
... at an alarming rate: 600 per 10-minute-interval. I wasn’t aware of any unexpected behaviour, but somehow the (software) bridge interface br0 that connects all my LXD containers’ NICs with the host’s physical NIC is making Netdata very nervous, i.e. it raises critical alarms. That’s no wonder though, its threshold for dropped incoming packets is a mere 5 per 10 minutes.
I was particularly intrigued by the fact that the rate of those dropped RX packets was a perfectly periodic 1 drop per second. This almost immediately ruled out the most serious reason for dropped packets, namely that the system is truly overloaded and simply can’t keep up with the deluge of packets it’s receiving.
Reasons for dropped packets
Some searching turned up the following alternate explanations:
- The softnet backlog full.
- Bad VLAN tags.
- Packets received with unknown or unregistered protocols.
- IPv6 frames when the server is configured only for ipv4.
(source: RedHat)
Confusingly, Netdata also complained about softnet backlog, but that’s unrelated in this case, since it can’t possibly explain the periodic nature of the packet dropping. (Solution: increase net.core.netdev_budget_usecs to 6000.)
Wireshark & tcpdump to the rescue, as always
A Wireshark capture executed on my laptop quickly turned up suspicious broadcast packets that would fit the “unknown protocols” mentioned above. Both packet types seem to be related to the smart home-capabilities of my ISP-provided cable modem, an AVM FritzBox 6490:
The first type, emitted every two seconds, has protocol type “Homeplug AV”, a.k.a. Powerline. The second, also emitted every two seconds, has an even obscurer protocol (protocol number 0x8912). I didn’t bother to further investigate that one. All I care about is that, taken together, they’d perfectly explain the 600 packets per 10 minute-period!
And indeed, tcpdump confirms that those packets are present on br0, but they’re not visible on any of the hosts (i.e. containers) connected to the bridge! Here’s how they show up on the bridge:
# tcpdump -i br0 ether proto 0x8912
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:32:13.241257 cc:ce:1e:a8:43:30 (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 60:
0x0000: 0170 a000 0000 1f84 dca3 97a2 5553 bef1 .p..........US..
0x0010: fcf9 796b 5214 13e9 e200 0000 0000 0000 ..ykR...........
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
21:32:15.241328 cc:ce:1e:a8:43:30 (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 60:
0x0000: 0170 a000 0000 1f84 dca3 97a2 5553 bef1 .p..........US..
0x0010: fcf9 796b 5214 13e9 e200 0000 0000 0000 ..ykR...........
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
Dropping packets using ebtables
The final piece of the puzzle is to explicitly drop the unwanted packets before the bridge is dropping them in a noisier fashion.
$ sudo ebtables -A INPUT -p 8912 -j DROP $ sudo ebtables -A INPUT -p 88e1 -j DROP
In order to persist these rules across reboots I’m using the netfilter-peristent package which comes with Debian/Ubuntu. Unfortunately it does not (yet) support saving ebtables, so I’ve added a third-party ebtables plugin for netfilter-persistent (Thanks, Thomas Tsiakalakis!)
$ sudo apt install netfilter-persistent $ EBT=/usr/share/netfilter-persistent/plugins.d/35-ebtables $ sudo wget -O $EBT https://git.zeyel.net/snippets/30/raw?inline=false $ sudo chmod +x $EBT $ sudo $EBT save
That’s all for today. If you’ve found this post useful, feel free to let me know.