Attaching a Guest Directly to a Virtualization Host Network Interface with a macvtap Driver

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

External

Internal

Overview

This procedure assumes the guest wants to have at least two kinds of network interfaces: one (eth0) connecting the guest to an internal virtual network that exists inside the virtualization host and it is used by guests to communicate with each other, and the others (ens8, ens9), enabled by a macvtap driver, attached directly to the visualization host network interfaces. These interfaces can be used by the guest to connect to different networks. Usually, there is a network interface that routes traffic to the LAN (192.168.1.*) traffic., and the other with a publicly routable IP address that services external traffic.

This procedure is written for two such network interfaces. The first one is assumed to connect the guest to the internal LAN, and the second one to a public network.

The procedure leaves the virtualization host network interfaces that will be used for direct attachment unconfigured (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.

After assignment, and after the guest boots, the corresponding virtualization host interfaces show up as follows: 

[...]
# this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
       valid_lft forever preferred_lft forever
# this macvtap interface corresponds to the ens9 (internet facing) interface on guest. The common element is the MAC address: 52:54:00:43:e0:2f
23: macvtap5@em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:bc:e5:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febc:e536/64 scope link
       valid_lft forever preferred_lft forever

The macvtap label index may vary, even between boots. How can we request specific "macvtapX@" labels, deterministically? So far, these were generated automatically.

The corresponding guest interfaces are ens8 and ens9 (or similar): 

[...]
# this guest interface corresponds to macvtap4@em1. The common element is the MAC address: 52:54:00:4d:8e:35
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.15/24 brd 192.168.1.255 scope global ens8
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
       valid_lft forever preferred_lft forever
# this guest interface corresponds to macvtap5@em3. The common element is the MAC address:52:54:00:43:e0:2f
3: ens9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:43:e0:2f brd ff:ff:ff:ff:ff:ff
    [...]

The virtualization host macvtap interface and its corresponding guest interface have the same MAC address.

Multiple independent macvtap interfaces can be set on the same virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the Configure the Virtual Device on Guest section, below.

Procedure

Configure the Physical Network Interface on the Virtualization Host

In case of the virtualization host interface that connects the virtualization host to the local network (em1), it is surely already configured. The content of /etc/sysconfig/network-scripts/ifcfg-em1 should be similar to: 

BOOTPROTO="none"
DEFROUTE="yes"
DEVICE="em1"
DNS1="192.168.1.1"
DNS2="8.8.8.8"
DOMAIN="local"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.13"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes" 
NAME="em1"
ONBOOT="yes"
PREFIX="24"
TYPE="Ethernet"
UUID="75f88553-aa66-4783-aa47-8430c3757fa5"

In case of the virtualization host network interface that gives access to internet (em3), leave it unconfigured on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em3 should show the following (only essential configuration is shown): 

BOOTPROTO="none"
DEFROUTE="no" # Does not matter, will not be used by the virtualization host
DEVICE="em3"
IPV4_FAILURE_FATAL="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="em3"
ONBOOT="no" # No network interface should be started at boot, the hardware will be shared by guests
PEERDNS="no"
PEERROUTES="no"
TYPE="Ethernet"
UUID="85e14c6b-f9d3-49a0-b566-f122d47001c6"

After reboot, the virtualization host interfaces look similar to: 

2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
       valid_lft forever preferred_lft forever
[...]
4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:543c:b010:1a66:daff:fe9f:96d8/64 scope global mngtmpaddr dynamic
       valid_lft 2591942sec preferred_lft 604742sec
    inet6 fe80::1a66:daff:fe9f:96d8/64 scope link
       valid_lft forever preferred_lft forever

Configure the Virtual Device on Guest

Follow the guest network reconfiguration procedure and update the network configuration as follows: 

...
<domain type='kvm' id='9'>
  [...]
  <devices>
    [...]
    <interface type='direct'>
      <source dev='em1' mode='private'/>
    </interface>
    <!-- More than one interface can be configured in one step -->
    <interface type='direct'>
      <source dev='em3' mode='private'/>
    </interface>
  </devices>
</domain>

This is the configuration step where the virtualization host network interfaces are chosen. Pick one that connects to the LAN, and another one that connects to a gateway device. <source dev='emX' ... /> is the configuration line that selects the virtualization host interface to attach to. Multiple guests and multiple interfaces on the same guest may attach to the same virtualization host interface.


Multiple guests can attach to the same virtualization host physical interface, and in "private" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("VEPA", "bridge", etc.) are available.

Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows: 

virsh dumpxml <guest-name>

<domain type='kvm' id='9'>
  [...]
  <devices>
    [...]
    <interface type='direct'>
      <mac address='52:54:00:4d:8e:35'/>
      <source dev='em1' mode='private'/>
      <target dev='macvtap4'/>
      <model type='rtl8139'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:43:e0:2f'/>
      <source dev='em3' mode='private'/>
      <target dev='macvtap5'/>
      <model type='rtl8139'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
  </devices>
</domain>

After guest reboot, the MAC address reported by the configuration, on the guest, must corresponds to the MAC address of the corresponding virtualization host macvtap*@em* interfaces.

Optionally, a specific MAC address could be configured on the libvirt guest interface: 

<interface type='direct'>
   ...
   <mac address='...'/>
   ...
</interface>
...

in the guest configuration.

Primary Interface Name Instability


A certain primary interface name instability has been observed after adding the second virtual network interface in the domain definition file. The name of the primary network interface would be assigned non-deterministically to "eth0" and "eth1", respectively, upon successive reboots. An attempt to configure "device" in the domain name interface definition was ineffective. The issue was worked around - without a good explanation of why it is happening - by adding both /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1. As a side effect of the instability, NetworkManager was not able to properly detect the interfaces at boot, so it was disabled: Disable NetworkManager and Configure the Interfaces Manually.

Configure the Network Interface on Guest

eth0 Manual Configuration

⚠️ This is important, if the eth0 is not configured as shown below, the new network interface will not work correctly.

Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains: 

DEFROUTE=no # The default route will be provided by the macvtap interface.
IPV6_DEFROUTE=no
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf

The default route will be provided by the public macvtap interface, so DEFROUTE should be set to "no". Also, we don't want DHCP to modify /etc/resolv.conf so PEERDNS is set to false.

ens8 (macvtap interface that connects to the LAN) Manual Configuration

⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.

Verify that the MAC address of the ens8 interface on guest matches the MAC address of the macvtap4@em1 interface on the virtualization host.

Update /etc/sysconfig/network-scripts/ifcfg-ens8 as follows:

Make sure to configure DEFROUTE="no". For external access to work, the default route should use the public interface. 

DEFROUTE="no" # the public macvtap interface will provide the default route, not this one

Also make sure UUID is unique, you can generate a new UUID with uuidgen.

The configuration should be similar to: 

BOOTPROTO="none"
DEFROUTE=no # the public macvtap interface will provide the default route, not this one
DEVICE="ens8"
DNS1="8.8.8.8"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.15"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens8"
NETMASK="255.255.255.0"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="24"
TYPE="Ethernet"
UUID=adcf7be8-0dd7-46f5-b572-c6d273e23d87

For more details on how to configure network interfaces, see

Configuring a Network Interface

Second, public macvtap (ens9) interface Manual Configuration

A second macvtap interface connects the guest to public internet. The configuration should be similar to the example provided below, and written in /etc/sysconfig/network-scripts/ifcfg-ens9. "DEFROUTE=yes" is essential.

⚠️ Important: The routable IP address used below must not be used by another guests. 

BOOTPROTO="none"
DEFROUTE="yes" # essential
DEVICE="ens9"
DNS1="8.8.8.8"
GATEWAY="104.50.201.86"
IPADDR="104.50.201.85" # must not conflict with other guests' public address
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens9"
NETMASK="255.255.255.248"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="29"
TYPE="Ethernet"
UUID="30C65EA0-1EC3-46C6-8AC1-202D83102B35"

Reboot the second time, both interfaces should be operational.

Verification

After reboot, the guest must be accessible over ssh from an external network.

The routing table should be similar to: 

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         104.50.201.86   0.0.0.0         UG        0 0          0 ens9
104.50.201.80   0.0.0.0         255.255.255.248 U         0 0          0 ens9
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens9
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 ens8
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0

Retrieved from "https://kb.novaordis.com/index.php?title=Attaching_a_Guest_Directly_to_a_Virtualization_Host_Network_Interface_with_a_macvtap_Driver&oldid=97556"