Running Proxmox with multiple network connections
I have a Proxmox cluster running with 3 nodes and I have 2x 10Gbe interfaces per machine, one for my internal backend network for the cluster communications and storage replication and one for my normal LAN traffic.
But I want both sides to be accessible from several other subnets and VLANs but normally you can only have a single gateway, how to configure this?
My personal take on this!
This is my personal take on this configuration, you might not agree or might need a different setup in your configuration, that’s up to you! In my case I mainly want to prevent traffic from running through my router if that’s not really required since the machine has a network card in 2 subnets.
Normally you’d say “but then you don’t need a gateway”, yes that’s true but I have several other networks and VLANs that I also wish to access the machine using one of the 2 interfaces and then it needs to understand how it can talk back, this config accomplishes that!
This article is as much personal documentation as a public explanation! đ
My setup
As I said each of my node has 2x 10Gbe network cards, enp4s0 and enp6s0.
enp4s0 is my normal network, 10.10.128.0/24
enp6s0 is my backend network, 10.10.10.0/24
Each node will have the same I number at the end (11, 12, 13) for each of the subnets.
The need for 2 routing tables
Each interface needs to keep it’s own routing table, to accomplish this we need to edit /etc/iproute2/rt_tables and change that file to add 2 new routing tables:
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 1 vmbr1 2 vmbr0
The order of these matter, in my case I want the machine to use vmbr1 first and then vmbr0 so I have added the 2 new routing tables in that order. You can use any name you wish, in my case I found it easy to have them have the same name as the bridge I use them with.
My /etc/network/interfaces file
Easiest is to just share my configuration file for node1 and then I will give some clarification. Make sure to add this configuration to all nodes that are applicable and change the values with their IPs.
auto lo iface lo inet loopback iface enp4s0 inet manual iface enp6s0 inet manual iface enp7s0 inet manual auto vmbr0 iface vmbr0 inet static address 10.10.10.11/24 bridge-ports enp6s0 bridge-stp off bridge-fd 0 post-up ip route add 10.10.10.0/24 dev vmbr0 src 10.10.10.11 table vmbr0 post-up ip route add default via 10.10.10.254 dev vmbr0 table vmbr0 post-up ip rule add from 10.10.10.11/32 table vmbr0 post-up ip rule add to 10.10.10.11/32 table vmbr0 #Backend network auto vmbr1 iface vmbr1 inet static address 10.10.128.11/24 bridge-ports enp4s0 bridge-stp off bridge-fd 0 post-up ip route add default via 10.10.128.254 dev vmbr1 post-up ip route add 10.10.128.0/24 dev vmbr1 src 10.10.128.11 table vmbr1 post-up ip route add default via 10.10.128.254 dev vmbr1 table vmbr1 post-up ip rule add from 10.10.128.11/32 table vmbr1 post-up ip rule add to 10.10.128.11/32 table vmbr1 #LAN network
What’s happening?
First we enable the interfaces and then we add 2 linux bridges, vmbr0 and vmbr1
Since we want each to have it’s own routing we basically add some “post-up” commands to add the routing destinations (in my case “default” as in 0.0.0.0/0 to the gateway on that network.
Since the OS itself also needs to know what interface to use by default (even though both interfaces have their own default gateway) we need to add a default gateway to one of the interfaces without the routing table too. In the above config that’s done in the vmbr1 bridge using the first route add.
Results
The end results can be checked using
ip route default via 10.10.128.254 dev vmbr1 10.10.10.0/24 dev vmbr0 proto kernel scope link src 10.10.10.11 10.10.128.0/24 dev vmbr1 proto kernel scope link src 10.10.128.11
but we can also check the interface/bridge specific routing rules
ip route list table vmbr0 default via 10.10.10.254 dev vmbr0 10.10.10.0/24 dev vmbr0 scope link src 10.10.10.11
ip route list table vmbr1 default via 10.10.128.254 dev vmbr1 10.10.128.0/24 dev vmbr1 scope link src 10.10.128.11
and also check the routing rules
ip rule show 0: from all lookup local 32762: from all to 10.10.128.11 lookup vmbr1 32763: from 10.10.128.11 lookup vmbr1 32764: from all to 10.10.10.11 lookup vmbr0 32765: from 10.10.10.11 lookup vmbr0 32766: from all lookup main 32767: from all lookup default
Bonus, fix DNS
If you want to make sure the machine knows it is available in 2 subnets and what DNS names it has there, make sure to update /etc/hosts accordingly.
I have it configured as follows:
127.0.0.1 localhost.localdomain localhost 10.10.10.11 node1.quinmanage.lan node1 10.10.128.11 node1.quindorian.lan node1 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts
The end result
The end result is an OS that by default uses vmbr1 to access anything beyond that’s available without routing on the vmbr0 and vmbr1 interface.
That’s how it works by default of course but with the above changes clients outside of either subnet can also access the machine and have a route back, back through the same interface! You can of course also set this up with dedicated routing rules for each subnet, but in a complex network that’s a lot more work.
Again, this is specific to my needs and normally this shouldn’t be required, especially if you don’t have any routed subnets or VLANs!
Credits
This article and this forum post was helpful in piecing the above together!