I’ve been working with a lot of VMs lately and was using the “new” macvtap + bridge mode for the VM networking as this is supposed to offer great performance, while being simpler to configure. A key point is that each macvtap has its own mac address on the same Ethernet segment – think of this as an Ethernet sub-interface. The physical interface to which the macvlan is attached is often referred to as “the lower device”, with the macvtap itself referred to as the “upper device”.
macvtap has a number of different modes that it can be run in;
- Bridge: this works almost like a traditional bridge, in that data received on a macvlan in bridge mode and destined for another macvlan of the same lower device is sent directly to the target (if the target macvlan is also in bridge mode), rather than being sent outside. Basically no learning, and no STP – simple and fast
- VEPA: Virtual Ethernet Port Aggregator is the default mode. If the lower device receives data from a macvlan in VEPA mode, this data is always sent “out” to the upstream switch or bridge, even if it’s destined for another macvlan in the same lower device. The benefit to this is that the traffic between VMs becomes visible externally, the downside being that the external switch must support hair-pin mode (a.k.a reflective relay).
- Private: this is essentially like VEPA mode, but with the added feature that no macvlans on the same lower device can communicate, regardless of where the packets come from – basically think of it as acting like a private VLAN.
- Passthrough: No idea yet about this mode
There is one big disadvantage to macvlan though – by design it is not possible for the host to connect directly with the guests.
There is however a workaround – we add a new macvtap interface to the host, and route all VM communications through that bridge. Basic steps are;
Create a new macvlan bridge and bring it up
sudo ip link add link eth0 macvlan0 type macvlan mode bridge
sudo ifconfig macvlan0 up
sudo ifconfig macvlan0
macvlan0 Link encap:Ethernet HWaddr fa:ab:14:97:fd:ec
inet6 addr: fe80::f8ab:14ff:fe97:fdec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:508 (508.0 B)
Assign an IP address to the macvtap interface (192.168.1.3), and add a static host route towards the VM IP (192.168.1.10)
sudo ifconfig macvlan0 192.168.1.3 netmask 255.255.255.0
sudo ip route add 192.168.1.10/32 via 192.168.1.3 dev macvlan0
And there you go – host-guest communications via macvtap interfaces
ping 192.168.1.10
PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.
64 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.282 ms
64 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.134 ms
The downside to this being;
- You lose 1 extra IP address on the host, so it can network to the guest
- You have to add a lot of static routes unless you further subnet your network between VM and non-VM hosts, and route the VM subnet through the macvap interface
