Attaching a Guest Directly to a Virtualization Host Network Interface with a macvtap Driver

From NovaOrdis Knowledge Base
Jump to: navigation, search

External

Internal

Overview

This procedure assumes the guest wants to have two network interfaces: one that connects the guest to an internal virtual network, used by guests to communicate with each other (eth0), and the second network interface, enabled by a macvtap driver, attached directly to a visualization host network interface (ens8).

The procedure leaves the virtualization host network interface that will be used for direct attachment unconfigured (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.

After assignment, and after the guest boots, the corresponding virtualization host interface shows up as follows:

3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
[...]
11: macvtap1@em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 500
    link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff

The macvtap label index may vary, even between boots. It is probably possible to request specific labels when configuring the interface on the guest.

The same interface shows up as ens9 (or similar) on the guest:

3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff
    inet 104.50.201.84/29 brd 104.50.201.87 scope global noprefixroute ens9
       valid_lft forever preferred_lft forever

The virtualization host macvtap interface (macvtap1@em2) and the guest interface have the same MAC address. In the example above is 52:54:00:5d:9c:33.

Note that multiple macvtap guest interfaces (ensX) can be set at the same time, binding the guest to the hardware of different network interfaces on the virtualization host.

At the same time, multiple independent macvtap interfaces can be set on the same virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the Configure the Virtual Device on Guest section, below.

Procedure

Configure the Physical Network Interface on the Virtualization Host

Leave the network interface unconfigured on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em2 should show the following (only essential configuration is shown):

NAME=em2
DEVICE=em2
TYPE=Ethernet
ONBOOT=no # No network interface should be started at boot, the hardware will be shared by guests
BOOTPROTO=no
DEFROUTE=no # Does not matter, will not be used by the virtualization host
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=85e14c6b-f9d3-49a0-b566-f122d47001c6

After reboot, it should look like this:

3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:dc41:3940:1a66:daff:fe9f:96d7/64 scope global mngtmpaddr dynamic
       valid_lft 3543sec preferred_lft 3543sec
    inet6 fe80::1a66:daff:fe9f:96d7/64 scope link
       valid_lft forever preferred_lft forever


The example above applies to a network interface that is left unconfigured. It is also possible to use an interface that is configured and used by the virtualization host for its own purposes:

2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
       valid_lft forever preferred_lft forever

Configure the Virtual Device on Guest

Follow the guest network reconfiguration procedure and update the network configuration as follows:

...
<interface type='direct'>
   <source dev='em2' mode='private'/>
</interface>

<!-- More than one interface can be configured in one step -->
<interface type='direct'>
   <source dev='em1' mode='private'/>
</interface>
...

where "em2" is the virtualization host network interface to be directly exposed to the guest.


Multiple guests can attach to the same virtualization host physical interface, and in "private" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("VEPA", "bridge", etc.) are available.

Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows:

virsh dumpxml <guest-name>
...
<interface type='direct'>
  <mac address='52:54:00:5d:9c:33'/>
  <source dev='em2' mode='private'/>
  <target dev='macvtap1'/>
  <model type='rtl8139'/>
  <alias name='net2'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
...

After guest reboot, the MAC address reported by the configuration must corresponds to the MAC address of the virtualization host macvtap1@em2 interface on the virtualization host, and it must be different from the MAC address of the virtualization host em2 interface.

Optionally, a specific MAC address could be configured on the libvirt guest interface, by specifying

<interface type='direct'>
   ...
   <mac address='...'/>
   ...
</interface>
...

in the guest configuration.

The new network interface, in this case ens9, shows up on the guest, alongside the default network interface eth0:

<syntaxhighlight lang='bash'>
ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
[...]
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff
    inet 104.50.201.84/29 brd 104.50.201.87 scope global noprefixroute ens9
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:b4:87:0f brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.249/24 brd 192.168.122.255 scope global noprefixroute dynamic eth0

The MAC address reported by the guest mavtap ens9 interface must be the same as the MAC address of the corresponding mavtap1@em2 on the virtualization host.

Primary Interface Name Instability


A certain primary interface name instability has been observed after adding the second virtual network interface in the domain definition file. The name of the primary network interface would be assigned non-deterministically to "eth0" and "eth1", respectively, upon successive reboots. An attempt to configure "device" in the domain name interface definition was ineffective. The issue was worked around - without a good explanation of why it is happening - by adding both /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1. As a side effect of the instability, NetworkManager was not able to properly detect the interfaces at boot, so it was disabled: Disable NetworkManager and Configure the Interfaces Manually.

Configure the Network Interface on Guest

eth0 Manual Configuration

⚠️ This is important, if the eth0 is not configured as shown below, the new network interface will not work correctly.

Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains:

DEFROUTE=no # The default route will be provided by the macvtap interface.
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf

The default route will be provided by the macvtap interface, so DEFROUTE should be set to "no". Also, we don't want DHCP to modify /etc/resolv.conf so PEERDNS is set to false.

ens9 (macvtap interface) Manual Configuration

⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.

Update /etc/sysconfig/network-scripts/ifcfg-ens9 as follows:

Make sure to configure:

DEFROUTE=yes # macvtap interface will provide the default route

Also make sure UUID is unique, you can generate a new UUID with uuidgen.

The configuration should be similar to:

NAME=ens9
DEVICE=ens9
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=104.50.201.84
NETMASK=255.255.255.248
PREFIX=29
GATEWAY=104.50.201.86
DEFROUTE=yes # macvtap interface will provide the default route
PEERROUTES=no
PEERDNS=no
DNS1=8.8.8.8
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=010B8B2C-9809-48E1-B14F-0BF224A8EFB2
For more details on how to configure network interfaces, see
Configuring a Network Interface

Second macvtap interface Manual Configuration

In case a second macvtap interface connects the guest to an internal network, the configuration should be similar to the example provided below. Note the "DEFROUTE=no" configuration.

NAME=ens8
DEVICE=ens8
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
PREFIX=24
GATEWAY=192.168.1.1
DEFROUTE=no
PEERROUTES=no
PEERDNS=no
DNS1=8.8.8.8
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=7af78337-479d-47e2-8b16-a04dc42227c8

Reboot the second time, both interfaces should be operational.

Verification

After reboot, the guest must be accessible over ssh from an external network.

The routing table should be similar to:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         104.50.201.86   0.0.0.0         UG        0 0          0 ens9
104.50.201.80   0.0.0.0         255.255.255.248 U         0 0          0 ens9
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 ens8
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0