Attaching a Guest Directly to a Virtualization Host Network Interface with a macvtap Driver: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(90 intermediate revisions by the same user not shown)
Line 11: Line 11:
=Overview=
=Overview=


This procedure assumes the guest wants to have two network interfaces: one that connects the guest to an [[KVM_Virtual_Networking_Concepts#Virtual_Network| internal virtual network]], used by guests to communicate with each other (eth0), and the second network interface, enabled by a [[KVM_Virtual_Networking_Concepts#macvtap_Driver|macvtap driver]], attached directly to a visualization host network interface (ens8).
This procedure assumes the guest wants to have at least two kinds of network interfaces: one (<code>eth0</code>) connecting the guest to an [[KVM_Virtual_Networking_Concepts#Virtual_Network| internal virtual network]] that exists inside the virtualization host and it is used by guests to communicate with each other, and the others (<code>ens8</code>, <code>ens9</code>), enabled by a [[KVM_Virtual_Networking_Concepts#macvtap_Driver|macvtap driver]], attached directly to the visualization host network interfaces. These interfaces can be used by the guest to connect to different networks. Usually, there is a network interface that routes traffic to the LAN (192.168.1.*) traffic., and the other with a publicly routable IP address that services external traffic.


The procedure leaves the virtualization host network interface that will be used for direct attachment '''unconfigured''' (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.  
This procedure is written for two such network interfaces. The first one is assumed to connect the guest to the internal LAN, and the second one to a public network.


After assignment, and after the guest boots, the corresponding virtualization host interface shows up as follows:
The procedure leaves the virtualization host network interfaces that will be used for direct attachment '''unconfigured''' (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.


<syntaxhighlight lang='bash'>
After assignment, and after the guest boots, the corresponding virtualization host interfaces show up as follows:
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
...
11: macvtap0@em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state LOWERLAYERDOWN qlen 500
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
</syntaxhighlight>


The same interface shows up as ens8 (or similar) on the guest:
<font size=-2>
[...]
<font color=SlateGray># this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35</font>
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
        valid_lft forever preferred_lft forever
<font color=SlateGray># this macvtap interface corresponds to the ens9 (internet facing) interface on guest. The common element is the MAC address: 52:54:00:43:e0:2f</font>
23: macvtap5@em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:bc:e5:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febc:e536/64 scope link
        valid_lft forever preferred_lft forever
</font>


<syntaxhighlight lang='bash'>
The macvtap label index may vary, even between boots. <font color=darkkhaki>How can we request specific "macvtapX@" labels, deterministically? So far, these were generated automatically.</font>
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
</syntaxhighlight>


The virtualization host macvtap interface (macvtap0@em2) and the guest interface have the same MAC address.
The corresponding guest interfaces are <code>ens8</code> and <code>ens9</code> (or similar):


Note that multiple macvtap guest interfaces can be set at the same time, binding the guest to the hardware of different network interfaces on the virtualization host.
<font size=-2>
[...]
<font color=SlateGray># this guest interface corresponds to macvtap4@em1. The common element is the MAC address: 52:54:00:4d:8e:35</font>
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.15/24 brd 192.168.1.255 scope global ens8
        valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
        valid_lft forever preferred_lft forever
<font color=SlateGray># this guest interface corresponds to macvtap5@em3. The common element is the MAC address:52:54:00:43:e0:2f</font>
3: ens9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:43:e0:2f brd ff:ff:ff:ff:ff:ff
    [...]
</font>
The virtualization host macvtap interface and its corresponding guest interface have the same MAC address.


At the same time, multiple independent macvtap interfaces can be set on the '''same''' virtualization host network interface. They can be configured independently with different IP addresses.
Multiple independent macvtap interfaces can be set on the '''same''' virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the [[#Configure_the_Virtual_Device_on_Guest|Configure the Virtual Device on Guest]] section, below.


=Procedure=
=Procedure=
Line 42: Line 59:
==Configure the Physical Network Interface on the Virtualization Host==
==Configure the Physical Network Interface on the Virtualization Host==


Leave the network interface '''unconfigured''' on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em3 should show the following:
In case of the virtualization host interface that connects the virtualization host to the local network (<code>em1</code>), it is surely already configured. The content of <code>/etc/sysconfig/network-scripts/ifcfg-em1</code> should be similar to:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
NAME=em3
BOOTPROTO="none"
DEVICE=em3
DEFROUTE="yes"
TYPE=Ethernet
DEVICE="em1"
ONBOOT=yes
DNS1="192.168.1.1"
BOOTPROTO=none
DNS2="8.8.8.8"
DEFROUTE=no # really? Research that, it might be an error
DOMAIN="local"
PEERDNS=no
GATEWAY="192.168.1.1"
PEERROUTES=no
IPADDR="192.168.1.13"
IPV4_FAILURE_FATAL=yes
IPV4_FAILURE_FATAL="yes"
IPV6INIT=no
IPV6_AUTOCONF="yes"
IPV6_AUTOCONF=no
IPV6_DEFROUTE="yes"
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL="no"
IPV6_PEERDNS=no
IPV6INIT="no"
IPV6_PEERROUTES=no
IPV6_PEERDNS="yes"
IPV6_FAILURE_FATAL=no
IPV6_PEERROUTES="yes"
NAME="em1"
ONBOOT="yes"
PREFIX="24"
TYPE="Ethernet"
UUID="75f88553-aa66-4783-aa47-8430c3757fa5"
</syntaxhighlight>
</syntaxhighlight>


After reboot, it should look like this:
In case of the virtualization host network interface that gives access to internet (<code>em3</code>), leave it '''unconfigured''' on the virtualization host. The corresponding configuration file <code>/etc/sysconfig/network-scripts/ifcfg-em3</code> should show the following (only essential configuration is shown):
 
<syntaxhighlight lang='bash'>
BOOTPROTO="none"
DEFROUTE="no" # Does not matter, will not be used by the virtualization host
DEVICE="em3"
IPV4_FAILURE_FATAL="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="em3"
ONBOOT="no" # No network interface should be started at boot, the hardware will be shared by guests
PEERDNS="no"
PEERROUTES="no"
TYPE="Ethernet"
UUID="85e14c6b-f9d3-49a0-b566-f122d47001c6"
</syntaxhighlight>


  4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop '''state DOWN''' qlen 1000
After reboot, the virtualization host interfaces look similar to:
<font size=-2>
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
        valid_lft forever preferred_lft forever
    inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
        valid_lft forever preferred_lft forever
[...]
  4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
     link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
     link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:543c:b010:1a66:daff:fe9f:96d8/64 scope global mngtmpaddr dynamic
        valid_lft 2591942sec preferred_lft 604742sec
    inet6 fe80::1a66:daff:fe9f:96d8/64 scope link
        valid_lft forever preferred_lft forever
</font>


==Configure the Virtual Device on Guest==
==Configure the Virtual Device on Guest==
Line 72: Line 127:
<syntaxhighlight lang='xml'>
<syntaxhighlight lang='xml'>
...
...
<interface type='direct'>
<domain type='kvm' id='9'>
  <source dev='em3' mode='private'/>
  [...]
</interface>
  <devices>
...
    [...]
    <interface type='direct'>
      <source dev='em1' mode='private'/>
    </interface>
    <!-- More than one interface can be configured in one step -->
    <interface type='direct'>
      <source dev='em3' mode='private'/>
    </interface>
  </devices>
</domain>
</syntaxhighlight>
</syntaxhighlight>


where "em3" is the virtualization host network interface to be directly exposed to the guest.
This is the configuration step where the virtualization host network interfaces are chosen. Pick one that connects to the LAN, and another one that connects to a gateway device. <code>&lt;source dev='emX' ... /&gt;</code> is the configuration line that selects the virtualization host interface to attach to. Multiple guests and multiple interfaces on the same guest may attach to the same virtualization host interface.


{{Note|Multiple guests can attach to the same virtualization host physical interface, and in "[[KVM_Virtual_Networking_Concepts#private|private]]" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("[[KVM_Virtual_Networking_Concepts#VEPA|VEPA]]", "[[KVM_Virtual_Networking_Concepts#bridge|bridge]]", etc.) are available.}}
{{Note|Multiple guests can attach to the same virtualization host physical interface, and in "[[KVM_Virtual_Networking_Concepts#private|private]]" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("[[KVM_Virtual_Networking_Concepts#VEPA|VEPA]]", "[[KVM_Virtual_Networking_Concepts#bridge|bridge]]", etc.) are available.}}


Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows:
Note that after creating the domain based on the XML configuration, <code>libvirt</code> will update the stored configuration as follows:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
Line 88: Line 152:
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang='xml'>
<syntaxhighlight lang='xml'>
...
<domain type='kvm' id='9'>
<interface type='direct'>
  [...]
  <mac address='52:54:00:f8:40:ff'/>
  <devices>
  <source dev='em3' mode='private'/>
    [...]
  <target dev='macvtap2'/>
    <interface type='direct'>
  <model type='rtl8139'/>
      <mac address='52:54:00:4d:8e:35'/>
  <alias name='net1'/>
      <source dev='em1' mode='private'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
      <target dev='macvtap4'/>
</interface>
      <model type='rtl8139'/>
...
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:43:e0:2f'/>
      <source dev='em3' mode='private'/>
      <target dev='macvtap5'/>
      <model type='rtl8139'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
  </devices>
</domain>
</syntaxhighlight>
</syntaxhighlight>


After guest reboot, the MAC address reported by the configuration must corresponds to the MAC address of the virtualization host macvtap...@em3 interface on the virtualization host, and it must be different from the MAC address of the virtualization host em3 interface.  
After guest reboot, the MAC address reported by the configuration, on the guest, must corresponds to the MAC address of the corresponding virtualization host <code>macvtap*@em*</code> interfaces.


Optionally, a specific MAC address could be configured on the libvirt guest interface, by specifying
Optionally, a specific MAC address could be configured on the <code>libvirt</code> guest interface:


<syntaxhighlight lang='xml'>
<syntaxhighlight lang='xml'>
Line 114: Line 190:


in the guest configuration.
in the guest configuration.
The new network interface shows up on the guest, alongside the default network interface eth1:
ip addr
...
2: '''ens8''': <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:e4:e6:8a brd ff:ff:ff:ff:ff:ff
The MAC address reported by the guest mavtap ens8 interface must be the same as the MAC address of the corresponding mavtap...@em3 on the virtualization host.


==Primary Interface Name Instability==
==Primary Interface Name Instability==
Line 135: Line 199:
===eth0 Manual Configuration===
===eth0 Manual Configuration===


Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains:
⚠️ This is important, if the <code>eth0</code> is not configured as shown below, the new network interface will not work correctly.


DEFROUTE=no
Make sure that <code>/etc/sysconfig/network-scripts/ifcfg-eth0</code> contains:
<syntaxhighlight lang='bash'>
DEFROUTE=no # The default route will be provided by the macvtap interface.
IPV6_DEFROUTE=no
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf
</syntaxhighlight>


The default route will be provided by the directly attached interface.
The default route will be provided by the '''public''' macvtap interface, so [[Linux_7_Configuring_a_Network_Interface#DEFROUTE|DEFROUTE]] should be set to "no". Also, we don't want DHCP to modify <code>/etc/resolv.conf</code> so [[Linux_7_Configuring_a_Network_Interface#PEERDNS|PEERDNS]] is set to false.


===ens8 Manual Configuration===
===ens8 (macvtap interface that connects to the LAN) Manual Configuration===


After the first boot, figure out the name of the new network interface and add a /etc/sysconfig/network-scripts/ifcfg-<interface-name> configuration file, with a content similar to the one shown below.
⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.


Make sure to configure:
Verify that the MAC address of the <code>ens8</code> interface on guest matches the MAC address of the <code>macvtap4@em1</code> interface on the virtualization host.


DEFROUTE=yes
Update <code>/etc/sysconfig/network-scripts/ifcfg-ens8</code> as follows:


For more than one interface, configure the one connected to the internal network to serve the default route.
Make sure to configure <code>DEFROUTE="no"</code>. For external access to work, the default route should use the public interface.
 
<syntaxhighlight lang='bash'>
DEFROUTE="no" # the public macvtap interface will provide the default route, not this one
</syntaxhighlight>


Also make sure UUID is unique, you can generate a new UUID with <tt>[[uuidgen]]</tt>.
Also make sure UUID is unique, you can generate a new UUID with <tt>[[uuidgen]]</tt>.


Configuration for a network interface that connects to an internal network.
The configuration should be similar to:
<syntaxhighlight lang='bash'>
BOOTPROTO="none"
DEFROUTE=no # the public macvtap interface will provide the default route, not this one
DEVICE="ens8"
DNS1="8.8.8.8"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.15"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens8"
NETMASK="255.255.255.0"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="24"
TYPE="Ethernet"
UUID=adcf7be8-0dd7-46f5-b572-c6d273e23d87
</syntaxhighlight>
 
For more details on how to configure network interfaces, see {{Internal|Linux 7 Configuring a Network Interface|Configuring a Network Interface}}


<syntaxhighlight lang="bash">
===Second, public macvtap (ens9) interface Manual Configuration===
NAME=ens8
DEVICE=ens8
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
PREFIX=24
GATEWAY=192.168.1.1
DEFROUTE=yes
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=7af78337-479d-47e2-8b16-a04dc42227c8
DNS1=8.8.8.8
</syntaxhighlight>


A second macvtap interface connects the guest to public internet. The configuration should be similar to the example provided below, and written in <code>/etc/sysconfig/network-scripts/ifcfg-ens9</code>. <code>"DEFROUTE=yes"</code> is essential.


Configuration for a network interface that connects to an external network.
⚠️ Important: The routable IP address used below must not be used by another guests.


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
NAME=ens8
BOOTPROTO="none"
DEVICE=ens8
DEFROUTE="yes" # essential
TYPE=Ethernet
DEVICE="ens9"
ONBOOT=yes
DNS1="8.8.8.8"
BOOTPROTO=none
GATEWAY="104.50.201.86"
IPADDR=104.50.201.84
IPADDR="104.50.201.85" # must not conflict with other guests' public address
PREFIX=29
IPV4_FAILURE_FATAL="yes"
GATEWAY=104.50.201.86
IPV6_AUTOCONF="no"
DEFROUTE=yes
IPV6_DEFROUTE="no"
PEERDNS=no
IPV6_FAILURE_FATAL="no"
PEERROUTES=no
IPV6INIT="no"
IPV4_FAILURE_FATAL=yes
IPV6_PEERDNS="no"
IPV6INIT=no
IPV6_PEERROUTES="no"
IPV6_AUTOCONF=no
NAME="ens9"
IPV6_DEFROUTE=no
NETMASK="255.255.255.248"
IPV6_PEERDNS=no
ONBOOT="yes"
IPV6_PEERROUTES=no
PEERDNS="no"
IPV6_FAILURE_FATAL=no
PEERROUTES="no"
UUID=800628FE-6265-4902-80A6-4FC0D018D305
PREFIX="29"
DNS1=8.8.8.8
TYPE="Ethernet"
UUID="30C65EA0-1EC3-46C6-8AC1-202D83102B35"
</syntaxhighlight>
</syntaxhighlight>
For more details on how to configure network interfaces, see {{Internal|Linux 7 Configuring a Network Interface|Configuring a Network Interface}}


Reboot the second time, both interfaces should be operational.
Reboot the second time, both interfaces should be operational.
Line 213: Line 290:


The routing table should be similar to:
The routing table should be similar to:
 
<font size=-1>
  Destination    Gateway        Genmask        Flags  MSS Window  irtt Iface
  Destination    Gateway        Genmask        Flags  MSS Window  irtt Iface
  0.0.0.0        104.50.201.86  0.0.0.0        UG        0 0          0 ens8
  0.0.0.0        104.50.201.86  0.0.0.0        UG        0 0          0 ens9
  104.50.201.80  0.0.0.0        255.255.255.248 U        0 0          0 ens8
  104.50.201.80  0.0.0.0        255.255.255.248 U        0 0          0 ens9
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 ens8
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 ens8
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 eth1
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 ens9
  192.168.122.0  0.0.0.0        255.255.255.0  U        0 0          0 eth1
169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 eth0
192.168.1.0    0.0.0.0        255.255.255.0  U        0 0          0 ens8
  192.168.122.0  0.0.0.0        255.255.255.0  U        0 0          0 eth0
</font>

Latest revision as of 03:45, 6 August 2023

External

Internal

Overview

This procedure assumes the guest wants to have at least two kinds of network interfaces: one (eth0) connecting the guest to an internal virtual network that exists inside the virtualization host and it is used by guests to communicate with each other, and the others (ens8, ens9), enabled by a macvtap driver, attached directly to the visualization host network interfaces. These interfaces can be used by the guest to connect to different networks. Usually, there is a network interface that routes traffic to the LAN (192.168.1.*) traffic., and the other with a publicly routable IP address that services external traffic.

This procedure is written for two such network interfaces. The first one is assumed to connect the guest to the internal LAN, and the second one to a public network.

The procedure leaves the virtualization host network interfaces that will be used for direct attachment unconfigured (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.

After assignment, and after the guest boots, the corresponding virtualization host interfaces show up as follows: 

[...]
# this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
       valid_lft forever preferred_lft forever
# this macvtap interface corresponds to the ens9 (internet facing) interface on guest. The common element is the MAC address: 52:54:00:43:e0:2f
23: macvtap5@em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:bc:e5:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febc:e536/64 scope link
       valid_lft forever preferred_lft forever

The macvtap label index may vary, even between boots. How can we request specific "macvtapX@" labels, deterministically? So far, these were generated automatically.

The corresponding guest interfaces are ens8 and ens9 (or similar): 

[...]
# this guest interface corresponds to macvtap4@em1. The common element is the MAC address: 52:54:00:4d:8e:35
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.15/24 brd 192.168.1.255 scope global ens8
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
       valid_lft forever preferred_lft forever
# this guest interface corresponds to macvtap5@em3. The common element is the MAC address:52:54:00:43:e0:2f
3: ens9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:43:e0:2f brd ff:ff:ff:ff:ff:ff
    [...]

The virtualization host macvtap interface and its corresponding guest interface have the same MAC address.

Multiple independent macvtap interfaces can be set on the same virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the Configure the Virtual Device on Guest section, below.

Procedure

Configure the Physical Network Interface on the Virtualization Host

In case of the virtualization host interface that connects the virtualization host to the local network (em1), it is surely already configured. The content of /etc/sysconfig/network-scripts/ifcfg-em1 should be similar to: 

BOOTPROTO="none"
DEFROUTE="yes"
DEVICE="em1"
DNS1="192.168.1.1"
DNS2="8.8.8.8"
DOMAIN="local"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.13"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes" 
NAME="em1"
ONBOOT="yes"
PREFIX="24"
TYPE="Ethernet"
UUID="75f88553-aa66-4783-aa47-8430c3757fa5"

In case of the virtualization host network interface that gives access to internet (em3), leave it unconfigured on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em3 should show the following (only essential configuration is shown): 

BOOTPROTO="none"
DEFROUTE="no" # Does not matter, will not be used by the virtualization host
DEVICE="em3"
IPV4_FAILURE_FATAL="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="em3"
ONBOOT="no" # No network interface should be started at boot, the hardware will be shared by guests
PEERDNS="no"
PEERROUTES="no"
TYPE="Ethernet"
UUID="85e14c6b-f9d3-49a0-b566-f122d47001c6"

After reboot, the virtualization host interfaces look similar to: 

2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
       valid_lft forever preferred_lft forever
[...]
4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:543c:b010:1a66:daff:fe9f:96d8/64 scope global mngtmpaddr dynamic
       valid_lft 2591942sec preferred_lft 604742sec
    inet6 fe80::1a66:daff:fe9f:96d8/64 scope link
       valid_lft forever preferred_lft forever

Configure the Virtual Device on Guest

Follow the guest network reconfiguration procedure and update the network configuration as follows: 

...
<domain type='kvm' id='9'>
  [...]
  <devices>
    [...]
    <interface type='direct'>
      <source dev='em1' mode='private'/>
    </interface>
    <!-- More than one interface can be configured in one step -->
    <interface type='direct'>
      <source dev='em3' mode='private'/>
    </interface>
  </devices>
</domain>

This is the configuration step where the virtualization host network interfaces are chosen. Pick one that connects to the LAN, and another one that connects to a gateway device. <source dev='emX' ... /> is the configuration line that selects the virtualization host interface to attach to. Multiple guests and multiple interfaces on the same guest may attach to the same virtualization host interface.


Multiple guests can attach to the same virtualization host physical interface, and in "private" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("VEPA", "bridge", etc.) are available.

Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows: 

virsh dumpxml <guest-name>

<domain type='kvm' id='9'>
  [...]
  <devices>
    [...]
    <interface type='direct'>
      <mac address='52:54:00:4d:8e:35'/>
      <source dev='em1' mode='private'/>
      <target dev='macvtap4'/>
      <model type='rtl8139'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:43:e0:2f'/>
      <source dev='em3' mode='private'/>
      <target dev='macvtap5'/>
      <model type='rtl8139'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
  </devices>
</domain>

After guest reboot, the MAC address reported by the configuration, on the guest, must corresponds to the MAC address of the corresponding virtualization host macvtap*@em* interfaces.

Optionally, a specific MAC address could be configured on the libvirt guest interface: 

<interface type='direct'>
   ...
   <mac address='...'/>
   ...
</interface>
...

in the guest configuration.

Primary Interface Name Instability


A certain primary interface name instability has been observed after adding the second virtual network interface in the domain definition file. The name of the primary network interface would be assigned non-deterministically to "eth0" and "eth1", respectively, upon successive reboots. An attempt to configure "device" in the domain name interface definition was ineffective. The issue was worked around - without a good explanation of why it is happening - by adding both /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1. As a side effect of the instability, NetworkManager was not able to properly detect the interfaces at boot, so it was disabled: Disable NetworkManager and Configure the Interfaces Manually.

Configure the Network Interface on Guest

eth0 Manual Configuration

⚠️ This is important, if the eth0 is not configured as shown below, the new network interface will not work correctly.

Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains: 

DEFROUTE=no # The default route will be provided by the macvtap interface.
IPV6_DEFROUTE=no
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf

The default route will be provided by the public macvtap interface, so DEFROUTE should be set to "no". Also, we don't want DHCP to modify /etc/resolv.conf so PEERDNS is set to false.

ens8 (macvtap interface that connects to the LAN) Manual Configuration

⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.

Verify that the MAC address of the ens8 interface on guest matches the MAC address of the macvtap4@em1 interface on the virtualization host.

Update /etc/sysconfig/network-scripts/ifcfg-ens8 as follows:

Make sure to configure DEFROUTE="no". For external access to work, the default route should use the public interface. 

DEFROUTE="no" # the public macvtap interface will provide the default route, not this one

Also make sure UUID is unique, you can generate a new UUID with uuidgen.

The configuration should be similar to: 

BOOTPROTO="none"
DEFROUTE=no # the public macvtap interface will provide the default route, not this one
DEVICE="ens8"
DNS1="8.8.8.8"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.15"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens8"
NETMASK="255.255.255.0"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="24"
TYPE="Ethernet"
UUID=adcf7be8-0dd7-46f5-b572-c6d273e23d87

For more details on how to configure network interfaces, see

Configuring a Network Interface

Second, public macvtap (ens9) interface Manual Configuration

A second macvtap interface connects the guest to public internet. The configuration should be similar to the example provided below, and written in /etc/sysconfig/network-scripts/ifcfg-ens9. "DEFROUTE=yes" is essential.

⚠️ Important: The routable IP address used below must not be used by another guests. 

BOOTPROTO="none"
DEFROUTE="yes" # essential
DEVICE="ens9"
DNS1="8.8.8.8"
GATEWAY="104.50.201.86"
IPADDR="104.50.201.85" # must not conflict with other guests' public address
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens9"
NETMASK="255.255.255.248"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="29"
TYPE="Ethernet"
UUID="30C65EA0-1EC3-46C6-8AC1-202D83102B35"

Reboot the second time, both interfaces should be operational.

Verification

After reboot, the guest must be accessible over ssh from an external network.

The routing table should be similar to: 

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         104.50.201.86   0.0.0.0         UG        0 0          0 ens9
104.50.201.80   0.0.0.0         255.255.255.248 U         0 0          0 ens9
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens9
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 ens8
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0

Retrieved from "https://kb.novaordis.com/index.php?title=Attaching_a_Guest_Directly_to_a_Virtualization_Host_Network_Interface_with_a_macvtap_Driver&oldid=97556"