Attaching a Guest Directly to a Virtualization Host Network Interface with a macvtap Driver

External

• https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-Virtual_Networking-Directly_attaching_to_physical_interface.html

Internal

• Linux Virtualization Networking Concepts
• Linux Virtualization Operations
• virt-install | Network Post-Configuration

Overview

This procedure assumes the guest wants to have two network interfaces: one that connects the guest to an internal virtual network, used by guests to communicate with each other (eth0), and the second network interface, enabled by a macvtap driver, attached directly to a visualization host network interface (ens8).

The procedure leaves the virtualization host network interface that will be used for direct attachment unconfigured (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.

After assignment, and after the guest boots, the corresponding virtualization host interface shows up as follows:

3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
[...]
11: macvtap1@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 500
    link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff

The macvtap label index may vary, even between boots. It is probably possible to request specific labels when configuring the interface on the guest.

The same interface shows up as ens9 (or similar) on the guest:

3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff
    inet 104.50.201.84/29 brd 104.50.201.87 scope global noprefixroute ens9
       valid_lft forever preferred_lft forever

The virtualization host macvtap interface (macvtap4@em2) and the guest interface have the same MAC address. In the example above is 52:54:00:5d:9c:33.

Note that multiple macvtap guest interfaces (ensX) can be set at the same time, binding the guest to the hardware of different network interfaces on the virtualization host.

At the same time, multiple independent macvtap interfaces can be set on the same virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the Configure the Virtual Device on Guest section, below.

Procedure

Configure the Physical Network Interface on the Virtualization Host

Leave the network interface unconfigured on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em2 should show the following (only essential configuration is shown):

NAME=em2
DEVICE=em2
TYPE=Ethernet
ONBOOT=no # No network interface should be started at boot, the hardware will be shared by guests
BOOTPROTO=no
DEFROUTE=no # Does not matter, will not be used by the virtualization host
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=85e14c6b-f9d3-49a0-b566-f122d47001c6

After reboot, it should look like this:

3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:dc41:3940:1a66:daff:fe9f:96d7/64 scope global mngtmpaddr dynamic
       valid_lft 3543sec preferred_lft 3543sec
    inet6 fe80::1a66:daff:fe9f:96d7/64 scope link
       valid_lft forever preferred_lft forever

Configure the Virtual Device on Guest

Follow the guest network reconfiguration procedure and update the network configuration as follows:

...
<interface type='direct'>
   <source dev='em2' mode='private'/>
</interface>
...

where "em2" is the virtualization host network interface to be directly exposed to the guest.

Multiple guests can attach to the same virtualization host physical interface, and in "private" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("VEPA", "bridge", etc.) are available.

Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows:

virsh dumpxml <guest-name>

...
<interface type='direct'>
  <mac address='52:54:00:5d:9c:33'/>
  <source dev='em2' mode='private'/>
  <target dev='macvtap1'/>
  <model type='rtl8139'/>
  <alias name='net2'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
...

After guest reboot, the MAC address reported by the configuration must corresponds to the MAC address of the virtualization host macvtap...@em3 interface on the virtualization host, and it must be different from the MAC address of the virtualization host em3 interface.

Optionally, a specific MAC address could be configured on the libvirt guest interface, by specifying

<interface type='direct'>
   ...
   <mac address='...'/>
   ...
</interface>
...

in the guest configuration.

The new network interface shows up on the guest, alongside the default network interface eth1:

ip addr
...
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:e4:e6:8a brd ff:ff:ff:ff:ff:ff

The MAC address reported by the guest mavtap ens8 interface must be the same as the MAC address of the corresponding mavtap...@em3 on the virtualization host.

Primary Interface Name Instability

A certain primary interface name instability has been observed after adding the second virtual network interface in the domain definition file. The name of the primary network interface would be assigned non-deterministically to "eth0" and "eth1", respectively, upon successive reboots. An attempt to configure "device" in the domain name interface definition was ineffective. The issue was worked around - without a good explanation of why it is happening - by adding both /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1. As a side effect of the instability, NetworkManager was not able to properly detect the interfaces at boot, so it was disabled: Disable NetworkManager and Configure the Interfaces Manually.

Configure the Network Interface on Guest

eth0 Manual Configuration

Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains:

DEFROUTE=no

The default route will be provided by the directly attached interface.

ens8 Manual Configuration

After the first boot, figure out the name of the new network interface and add a /etc/sysconfig/network-scripts/ifcfg-<interface-name> configuration file, with a content similar to the one shown below.

Make sure to configure:

DEFROUTE=yes

For more than one interface, configure the one connected to the internal network to serve the default route.

Also make sure UUID is unique, you can generate a new UUID with uuidgen.

Configuration for a network interface that connects to an internal network.

NAME=ens8
DEVICE=ens8
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
PREFIX=24
GATEWAY=192.168.1.1
DEFROUTE=yes
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=7af78337-479d-47e2-8b16-a04dc42227c8
DNS1=8.8.8.8

Configuration for a network interface that connects to an external network.

NAME=ens8
DEVICE=ens8
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=104.50.201.84
PREFIX=29
GATEWAY=104.50.201.86
DEFROUTE=yes
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=800628FE-6265-4902-80A6-4FC0D018D305
DNS1=8.8.8.8

For more details on how to configure network interfaces, see

Configuring a Network Interface

Reboot the second time, both interfaces should be operational.

Verification

After reboot, the guest must be accessible over ssh from an external network.

The routing table should be similar to:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         104.50.201.86   0.0.0.0         UG        0 0          0 ens8
104.50.201.80   0.0.0.0         255.255.255.248 U         0 0          0 ens8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth1