For a few years now I’ve had a weird broadcast spam issue when I run multiple installs of Home Assistant, especially if these installations would be on a different VLAN or subnet, but I think I’ve finally found it!
Running Multiple Home Assistant installs
Because of my YouTube channel and well natural curiosity I play around a lot with sensors and other devices. I have a “production” Home Assistant VM that runs all the things I have automated in my house, including all my lights using QuinLED modules but also temperature sensors or automations to automate my full home ventilation system.
Next to that I generally have one or two other virtual machines also running a install of Home Assistant to do my testing with.
But I’ve been having A LOT of issues doing so….
Multicast/Broadcast storm
Whenever I’d run two Home Assistant installs, especially if those would be on 2 different subnets, my whole network would get flooded with a massive broadcast storm which would immediately go away as soon as there was only a single Home Assistant install running.
I’ve been batteling this issue over about 2 years now and up until recently, I had no clue what was going on, or how to block it even (firewall rules, bridge filters, etc. didn’t help at all).
Network topology
My network topology is likely a bit more complex then the generic home setup, but in short it looks like this:
- Main router is a Mikrotik RB4011
- Main “backbone” switch is a 10Gbit Mikrotik CRS308
- RB4011 and all switches connected with 10Gbit
- 2x Switch Mikrotik CRS226
- Some ports run through 12x Passive POE injector for AP and IP cameras
My main router is my internet gateway but also does a lot of inter-VLAN and subnet routing. I have a few VLANs but mostly I have a lot of subnets configured in a single VLAN.
For instance, I have a subnet for my IP cameras. A lot of people talk about putting your IP cameras on a separate VLAN but generally I think they mean you should put them in their own subnet. That way you can configure firewall rules for that subnet such as “no internet for you” while still being easily able to route to them. But ok, that is not the subject of this article.
The issue
As mentioned above, the issue arises when I start a second Home Assistant install somewhere in my network but it was especially apparent when there was 2 Home Assistant installations running in 2 different subnets. For some reason, ALL physical ports on the network would receive a steady stream of between 8Mbit to 10Mbit of traffic.
Digging into it further, it was clear the Home Assistant installations where sending this traffic and it was directed towards 224.0.0.251 which is the multicast address for mDNS.
Within RouterOS it seemed almost impossible to block this too.
So for some reason, if 2 Home Assistant installations where running on 2 subnets, they would go crazy in spamming the whole network, flooding it completely, sometimes so much so that pings would increase or even start failing at some point.
Casued by the “Unknown Multicast Flood” option
Well, after 2 years I think I have finally found it. It turns out the cause might be a feature Mikrotik has enabled by default called “unknown Multicast Flood”.
Basically what that does, is that if a Multicast comes in, it gets replicated to all ports on the same bridge that have this enabled, which it is by default.
But if you have a fully interconnected network like I have, somehow it decides that it’s just going to spam this to *ALL* ports you have on the network. And normally I guess this would be ok, that’s how mDNS tries to find devices, if it all worked correctly, mDNS would send out a “ping” to all devices on the network and then a device that does mDNS can respond back saying it’s there and what IP it has.
But in my Mikrotik network at least, things go completely haywire and forms into some kind of feedback loop between the 2 Home Assistant installations and brings down most of the network until you shut one of them off!
Turn it off, fixed?!
Now, I’m sure there are use cases where you’d want all unknown Multicasts to be broadcast to all your ports. But personally I don’t use mDNS since it’s not compatible with my network setup (in theory it doesn’t work over multiple VLANs and subnets, unless your router has an mDNS helper, which Mikrotik does not).
But turning this feature off, especially on my main backbone switch, instantly fixed the issue, no more broadcast storm, all good!
Now this will likely interfere with some auto-scanning functionality of Home Assistant to be able to find compatible devices on your network. But as said, that wasn’t really working anyway because my production Home Assistant server runs in a different subnet then my IOT wireless devices for instance.
This setting is present per port in a bridge in RouterOS:
Final World
Now, I’m really glad I have finally been able to resolve this issue. But let me know down in the comments if you have ever experienced something like this. Maybe with Mikrotik equipment or a completely different setup? It certainly had me stumped for quite a while!
It’s always most inspiring to read your posts, Quindor. Thanks for giving us insight in your world.
I’d really like to get to know a bit more about your network topology, the basics above woke my interest since I’m looking for something alike for my house.
Being a noop in sfp I’m struggling with the very basics like the cable. I always thought DAC is digital to analog converter but how does that make sense in this field? It appears DAC is Direct Attach Twinax Cable 🙂
Quindor, I can’t believe I finally found someone else who came across this issue.
It’s still a problem! I’ve had this problem for years and it’s very frustrating. I have UniFi kit.
Unfortunately I heavily depend on mDNS and the reflector.
I think you are right that the 2 instances are somehow causing a never ending mDNS tennis match.
I’ve tried to explain the issue in the Home Assistant Discord. But rarely get past people understanding why I need to run 3 instances of HA each month. (For comparing releases) They seem to lose interest when realising it’s not a simple issue to get to the root cause of.
Perhaps I’ll try 3 instances just on the same VLAN and see if the problem persists or goes away.
Hy,
when you Disable the Unknown Multicast Flood, the IPv6 Address advertising is still not working. I don´t know the reason in in the deep details, I have testing with current Releases 7.16, 7.17 after disable no IPv6 adresses is recived from the Clients
Poh, I haven’t gotten into IPv6 too much so I haven’t noticed any issues in that regard myself!