Attaching a Guest Directly to a Virtualization Host Network Interface with a macvtap Driver: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
 
(107 intermediate revisions by the same user not shown)
Line 7: Line 7:
* [[KVM_Virtual_Networking_Concepts#macvtap_Driver_Operations|Linux Virtualization Networking Concepts]]
* [[KVM_Virtual_Networking_Concepts#macvtap_Driver_Operations|Linux Virtualization Networking Concepts]]
* [[Linux_Virtualization_Operations#Virtualization_Host_Network_Operations|Linux Virtualization Operations]]
* [[Linux_Virtualization_Operations#Virtualization_Host_Network_Operations|Linux Virtualization Operations]]
* [[Virt-install#Network_Post-Configuration|virt-install | Network Post-Configuration]]


=Overview=
=Overview=


This procedure assumes the guest wants to have two network interfaces: one that connects the guest to an [[KVM_Virtual_Networking_Concepts#Virtual_Network| internal virtual network]], used by guests to communicate with each other (eth0), and the second network interface, enabled by a [[KVM_Virtual_Networking_Concepts#macvtap_Driver|macvtap driver]], attached directly to a visualization host network interface (ens8).
This procedure assumes the guest wants to have at least two kinds of network interfaces: one (<code>eth0</code>) connecting the guest to an [[KVM_Virtual_Networking_Concepts#Virtual_Network| internal virtual network]] that exists inside the virtualization host and it is used by guests to communicate with each other, and the others (<code>ens8</code>, <code>ens9</code>), enabled by a [[KVM_Virtual_Networking_Concepts#macvtap_Driver|macvtap driver]], attached directly to the visualization host network interfaces. These interfaces can be used by the guest to connect to different networks. Usually, there is a network interface that routes traffic to the LAN (192.168.1.*) traffic., and the other with a publicly routable IP address that services external traffic.


The procedure leaves the network interface that will be used for direct attachment '''unconfigured''' on the virtualization host and declares a network interface of type "direct" in the guest configuration. After assignment, and after the guest boots, the corresponding interface shows up on host as follows:
This procedure is written for two such network interfaces. The first one is assumed to connect the guest to the internal LAN, and the second one to a public network.


<syntaxhighlight lang='bash'>
The procedure leaves the virtualization host network interfaces that will be used for direct attachment '''unconfigured''' (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.
3: em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
 
    link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
After assignment, and after the guest boots, the corresponding virtualization host interfaces show up as follows:
...
 
11: macvtap0@em2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state LOWERLAYERDOWN qlen 500
<font size=-2>
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
[...]
</syntaxhighlight>
<font color=SlateGray># this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35</font>
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
        valid_lft forever preferred_lft forever
<font color=SlateGray># this macvtap interface corresponds to the ens9 (internet facing) interface on guest. The common element is the MAC address: 52:54:00:43:e0:2f</font>
23: macvtap5@em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:bc:e5:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febc:e536/64 scope link
        valid_lft forever preferred_lft forever
</font>
 
The macvtap label index may vary, even between boots. <font color=darkkhaki>How can we request specific "macvtapX@" labels, deterministically? So far, these were generated automatically.</font>


The same interface shows up as ens8 on the guest:
The corresponding guest interfaces are <code>ens8</code> and <code>ens9</code> (or similar):


<syntaxhighlight lang='bash'>
<font size=-2>
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
[...]
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
<font color=SlateGray># this guest interface corresponds to macvtap4@em1. The common element is the MAC address: 52:54:00:4d:8e:35</font>
</syntaxhighlight>
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.15/24 brd 192.168.1.255 scope global ens8
        valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
        valid_lft forever preferred_lft forever
<font color=SlateGray># this guest interface corresponds to macvtap5@em3. The common element is the MAC address:52:54:00:43:e0:2f</font>
3: ens9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:43:e0:2f brd ff:ff:ff:ff:ff:ff
    [...]
</font>
The virtualization host macvtap interface and its corresponding guest interface have the same MAC address.


Note that the virtualization host macvtap interface (macvtap0@em2) and the guest interface have the same MAC address.
Multiple independent macvtap interfaces can be set on the '''same''' virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the [[#Configure_the_Virtual_Device_on_Guest|Configure the Virtual Device on Guest]] section, below.


=Procedure=
=Procedure=
Line 35: Line 59:
==Configure the Physical Network Interface on the Virtualization Host==
==Configure the Physical Network Interface on the Virtualization Host==


Leave the network interface '''unconfigured''' on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em3 should show the following:
In case of the virtualization host interface that connects the virtualization host to the local network (<code>em1</code>), it is surely already configured. The content of <code>/etc/sysconfig/network-scripts/ifcfg-em1</code> should be similar to:


<syntaxhighlight lang='bash'>
<syntaxhighlight lang='bash'>
NAME=em3
BOOTPROTO="none"
DEVICE=em3
DEFROUTE="yes"
TYPE=Ethernet
DEVICE="em1"
ONBOOT=no
DNS1="192.168.1.1"
BOOTPROTO=none
DNS2="8.8.8.8"
DEFROUTE=no
DOMAIN="local"
PEERDNS=no
GATEWAY="192.168.1.1"
PEERROUTES=no
IPADDR="192.168.1.13"
IPV4_FAILURE_FATAL=no
IPV4_FAILURE_FATAL="yes"
IPV6INIT=no
IPV6_AUTOCONF="yes"
IPV6_AUTOCONF=no
IPV6_DEFROUTE="yes"
IPV6_DEFROUTE=no
IPV6_FAILURE_FATAL="no"
IPV6_PEERDNS=no
IPV6INIT="no"
IPV6_PEERROUTES=no
IPV6_PEERDNS="yes"
IPV6_FAILURE_FATAL=no
IPV6_PEERROUTES="yes"
NAME="em1"
ONBOOT="yes"
PREFIX="24"
TYPE="Ethernet"
UUID="75f88553-aa66-4783-aa47-8430c3757fa5"
</syntaxhighlight>
</syntaxhighlight>


After reboot, it should look like this:
In case of the virtualization host network interface that gives access to internet (<code>em3</code>), leave it '''unconfigured''' on the virtualization host. The corresponding configuration file <code>/etc/sysconfig/network-scripts/ifcfg-em3</code> should show the following (only essential configuration is shown):
 
<syntaxhighlight lang='bash'>
BOOTPROTO="none"
DEFROUTE="no" # Does not matter, will not be used by the virtualization host
DEVICE="em3"
IPV4_FAILURE_FATAL="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="em3"
ONBOOT="no" # No network interface should be started at boot, the hardware will be shared by guests
PEERDNS="no"
PEERROUTES="no"
TYPE="Ethernet"
UUID="85e14c6b-f9d3-49a0-b566-f122d47001c6"
</syntaxhighlight>


  4: em3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop '''state DOWN''' qlen 1000
After reboot, the virtualization host interfaces look similar to:
<font size=-2>
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
        valid_lft forever preferred_lft forever
    inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
        valid_lft forever preferred_lft forever
[...]
  4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
     link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
     link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:543c:b010:1a66:daff:fe9f:96d8/64 scope global mngtmpaddr dynamic
        valid_lft 2591942sec preferred_lft 604742sec
    inet6 fe80::1a66:daff:fe9f:96d8/64 scope link
        valid_lft forever preferred_lft forever
</font>


==Configure the Virtual Device on Guest==
==Configure the Virtual Device on Guest==


With the guest down, assign it to the guest in configuration with [[virsh edit]] or in the corresponding XML configuration file as follows:
Follow the [[Linux Virtualization Reconfigure Guest Network|guest network reconfiguration procedure]] and update the network configuration as follows:
 
<syntaxhighlight lang='xml'>
<syntaxhighlight lang='xml'>
...
...
<interface type='direct'>
<domain type='kvm' id='9'>
  <source dev='em3' mode='private'/>
  [...]
</interface>
  <devices>
...
    [...]
    <interface type='direct'>
      <source dev='em1' mode='private'/>
    </interface>
    <!-- More than one interface can be configured in one step -->
    <interface type='direct'>
      <source dev='em3' mode='private'/>
    </interface>
  </devices>
</domain>
</syntaxhighlight>
</syntaxhighlight>


where "em3" is the virtualization host network interface to be directly exposed to the guest.
This is the configuration step where the virtualization host network interfaces are chosen. Pick one that connects to the LAN, and another one that connects to a gateway device. <code>&lt;source dev='emX' ... /&gt;</code> is the configuration line that selects the virtualization host interface to attach to. Multiple guests and multiple interfaces on the same guest may attach to the same virtualization host interface.


{{Note|Multiple guests can attach to the same virtualization host physical interface, and in "[[KVM_Virtual_Networking_Concepts#private|private]]" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("[[KVM_Virtual_Networking_Concepts#VEPA|VEPA]]", "[[KVM_Virtual_Networking_Concepts#bridge|bridge]]", etc.) are available.}}
{{Note|Multiple guests can attach to the same virtualization host physical interface, and in "[[KVM_Virtual_Networking_Concepts#private|private]]" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("[[KVM_Virtual_Networking_Concepts#VEPA|VEPA]]", "[[KVM_Virtual_Networking_Concepts#bridge|bridge]]", etc.) are available.}}


Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows:
Note that after creating the domain based on the XML configuration, <code>libvirt</code> will update the stored configuration as follows:


<syntaxhighlight lang='bash'>
virsh dumpxml <guest-name>
</syntaxhighlight>
<syntaxhighlight lang='xml'>
<syntaxhighlight lang='xml'>
...
<domain type='kvm' id='9'>
<interface type='direct'>
  [...]
  <mac address='...'/>
  <devices>
  <source dev='em3' mode='private'/>
    [...]
  <model type='rtl8139'/>
    <interface type='direct'>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
      <mac address='52:54:00:4d:8e:35'/>
</interface>
      <source dev='em1' mode='private'/>
...
      <target dev='macvtap4'/>
      <model type='rtl8139'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:43:e0:2f'/>
      <source dev='em3' mode='private'/>
      <target dev='macvtap5'/>
      <model type='rtl8139'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
  </devices>
</domain>
</syntaxhighlight>
</syntaxhighlight>


After guest reboot, the MAC address reported by the configuration must corresponds to the MAC address of the virtualization host macvtap...@em3 interface on the virtualization host, and it must be different from the MAC address of the virtualization host em3 interface. Optionally, a specific MAC address could be configured on the libvirt guest interface, by specifying
After guest reboot, the MAC address reported by the configuration, on the guest, must corresponds to the MAC address of the corresponding virtualization host <code>macvtap*@em*</code> interfaces.
 
Optionally, a specific MAC address could be configured on the <code>libvirt</code> guest interface:


<syntaxhighlight lang='xml'>
<syntaxhighlight lang='xml'>
Line 101: Line 190:


in the guest configuration.
in the guest configuration.
The new network interface shows up on the guest, alongside the default network interface eth1:
ip addr
...
2: '''ens8''': <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:02:72:ed brd ff:ff:ff:ff:ff:ff
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:e4:e6:8a brd ff:ff:ff:ff:ff:ff
The MAC address reported by the guest mavtap ens8 interface must be the same as the MAC address of the corresponding mavtap...@em3 on the virtualization host.


==Primary Interface Name Instability==
==Primary Interface Name Instability==
Line 122: Line 199:
===eth0 Manual Configuration===
===eth0 Manual Configuration===


Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains:
⚠️ This is important, if the <code>eth0</code> is not configured as shown below, the new network interface will not work correctly.


DEFROUTE=no
Make sure that <code>/etc/sysconfig/network-scripts/ifcfg-eth0</code> contains:
<syntaxhighlight lang='bash'>
DEFROUTE=no # The default route will be provided by the macvtap interface.
IPV6_DEFROUTE=no
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf
</syntaxhighlight>


The default route will be provided by the directly attached interface.
The default route will be provided by the '''public''' macvtap interface, so [[Linux_7_Configuring_a_Network_Interface#DEFROUTE|DEFROUTE]] should be set to "no". Also, we don't want DHCP to modify <code>/etc/resolv.conf</code> so [[Linux_7_Configuring_a_Network_Interface#PEERDNS|PEERDNS]] is set to false.


===ens8 Manual Configuration===
===ens8 (macvtap interface that connects to the LAN) Manual Configuration===


After the first boot, figure out the name of the new network interface and add a /etc/sysconfig/network-scripts/ifcfg-<interface-name> configuration file, with a content similar to the one shown below.
⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.


Make sure to configure:
Verify that the MAC address of the <code>ens8</code> interface on guest matches the MAC address of the <code>macvtap4@em1</code> interface on the virtualization host.


DEFROUTE=yes
Update <code>/etc/sysconfig/network-scripts/ifcfg-ens8</code> as follows:
 
Make sure to configure <code>DEFROUTE="no"</code>. For external access to work, the default route should use the public interface.
 
<syntaxhighlight lang='bash'>
DEFROUTE="no" # the public macvtap interface will provide the default route, not this one
</syntaxhighlight>


Also make sure UUID is unique, you can generate a new UUID with <tt>[[uuidgen]]</tt>.
Also make sure UUID is unique, you can generate a new UUID with <tt>[[uuidgen]]</tt>.


<syntaxhighlight lang="bash">
The configuration should be similar to:
NAME=ens8
<syntaxhighlight lang='bash'>
DEVICE=ens8
BOOTPROTO="none"
TYPE=Ethernet
DEFROUTE=no # the public macvtap interface will provide the default route, not this one
ONBOOT=yes
DEVICE="ens8"
BOOTPROTO=none
DNS1="8.8.8.8"
IPADDR=104.50.201.84
GATEWAY="192.168.1.1"
PREFIX=29
IPADDR="192.168.1.15"
GATEWAY=104.50.201.86
IPV4_FAILURE_FATAL="yes"
DEFROUTE=yes
IPV6_AUTOCONF="no"
PEERDNS=no
IPV6_DEFROUTE="no"
PEERROUTES=no
IPV6_FAILURE_FATAL="no"
IPV4_FAILURE_FATAL=yes
IPV6INIT="no"
IPV6INIT=no
IPV6_PEERDNS="no"
IPV6_AUTOCONF=no
IPV6_PEERROUTES="no"
IPV6_DEFROUTE=no
NAME="ens8"
IPV6_PEERDNS=no
NETMASK="255.255.255.0"
IPV6_PEERROUTES=no
ONBOOT="yes"
IPV6_FAILURE_FATAL=no
PEERDNS="no"
UUID=800628FE-6265-4902-80A6-4FC0D018D305
PEERROUTES="no"
DNS1=8.8.8.8
PREFIX="24"
TYPE="Ethernet"
UUID=adcf7be8-0dd7-46f5-b572-c6d273e23d87
</syntaxhighlight>
</syntaxhighlight>


For more details on how to configure network interfaces, see {{Internal|Linux 7 Configuring a Network Interface|Configuring a Network Interface}}
For more details on how to configure network interfaces, see {{Internal|Linux 7 Configuring a Network Interface|Configuring a Network Interface}}
===Second, public macvtap (ens9) interface Manual Configuration===
A second macvtap interface connects the guest to public internet. The configuration should be similar to the example provided below, and written in <code>/etc/sysconfig/network-scripts/ifcfg-ens9</code>. <code>"DEFROUTE=yes"</code> is essential.
⚠️ Important: The routable IP address used below must not be used by another guests.
<syntaxhighlight lang="bash">
BOOTPROTO="none"
DEFROUTE="yes" # essential
DEVICE="ens9"
DNS1="8.8.8.8"
GATEWAY="104.50.201.86"
IPADDR="104.50.201.85" # must not conflict with other guests' public address
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens9"
NETMASK="255.255.255.248"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="29"
TYPE="Ethernet"
UUID="30C65EA0-1EC3-46C6-8AC1-202D83102B35"
</syntaxhighlight>


Reboot the second time, both interfaces should be operational.
Reboot the second time, both interfaces should be operational.
Line 170: Line 290:


The routing table should be similar to:
The routing table should be similar to:
 
<font size=-1>
  Destination    Gateway        Genmask        Flags  MSS Window  irtt Iface
  Destination    Gateway        Genmask        Flags  MSS Window  irtt Iface
  0.0.0.0        104.50.201.86  0.0.0.0        UG        0 0          0 ens8
  0.0.0.0        104.50.201.86  0.0.0.0        UG        0 0          0 ens9
  104.50.201.80  0.0.0.0        255.255.255.248 U        0 0          0 ens8
  104.50.201.80  0.0.0.0        255.255.255.248 U        0 0          0 ens9
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 ens8
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 ens8
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 eth1
  169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 ens9
  192.168.122.0  0.0.0.0        255.255.255.0  U        0 0          0 eth1
169.254.0.0    0.0.0.0        255.255.0.0    U        0 0          0 eth0
192.168.1.0    0.0.0.0        255.255.255.0  U        0 0          0 ens8
  192.168.122.0  0.0.0.0        255.255.255.0  U        0 0          0 eth0
</font>

Latest revision as of 03:45, 6 August 2023

External

Internal

Overview

This procedure assumes the guest wants to have at least two kinds of network interfaces: one (eth0) connecting the guest to an internal virtual network that exists inside the virtualization host and it is used by guests to communicate with each other, and the others (ens8, ens9), enabled by a macvtap driver, attached directly to the visualization host network interfaces. These interfaces can be used by the guest to connect to different networks. Usually, there is a network interface that routes traffic to the LAN (192.168.1.*) traffic., and the other with a publicly routable IP address that services external traffic.

This procedure is written for two such network interfaces. The first one is assumed to connect the guest to the internal LAN, and the second one to a public network.

The procedure leaves the virtualization host network interfaces that will be used for direct attachment unconfigured (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.

After assignment, and after the guest boots, the corresponding virtualization host interfaces show up as follows: 

[...]
# this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
       valid_lft forever preferred_lft forever
# this macvtap interface corresponds to the ens9 (internet facing) interface on guest. The common element is the MAC address: 52:54:00:43:e0:2f
23: macvtap5@em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether 52:54:00:bc:e5:36 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febc:e536/64 scope link
       valid_lft forever preferred_lft forever

The macvtap label index may vary, even between boots. How can we request specific "macvtapX@" labels, deterministically? So far, these were generated automatically.

The corresponding guest interfaces are ens8 and ens9 (or similar): 

[...]
# this guest interface corresponds to macvtap4@em1. The common element is the MAC address: 52:54:00:4d:8e:35
2: ens8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.15/24 brd 192.168.1.255 scope global ens8
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe4d:8e35/64 scope link
       valid_lft forever preferred_lft forever
# this guest interface corresponds to macvtap5@em3. The common element is the MAC address:52:54:00:43:e0:2f
3: ens9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:43:e0:2f brd ff:ff:ff:ff:ff:ff
    [...]

The virtualization host macvtap interface and its corresponding guest interface have the same MAC address.

Multiple independent macvtap interfaces can be set on the same virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the Configure the Virtual Device on Guest section, below.

Procedure

Configure the Physical Network Interface on the Virtualization Host

In case of the virtualization host interface that connects the virtualization host to the local network (em1), it is surely already configured. The content of /etc/sysconfig/network-scripts/ifcfg-em1 should be similar to: 

BOOTPROTO="none"
DEFROUTE="yes"
DEVICE="em1"
DNS1="192.168.1.1"
DNS2="8.8.8.8"
DOMAIN="local"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.13"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes" 
NAME="em1"
ONBOOT="yes"
PREFIX="24"
TYPE="Ethernet"
UUID="75f88553-aa66-4783-aa47-8430c3757fa5"

In case of the virtualization host network interface that gives access to internet (em3), leave it unconfigured on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em3 should show the following (only essential configuration is shown): 

BOOTPROTO="none"
DEFROUTE="no" # Does not matter, will not be used by the virtualization host
DEVICE="em3"
IPV4_FAILURE_FATAL="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="em3"
ONBOOT="no" # No network interface should be started at boot, the hardware will be shared by guests
PEERDNS="no"
PEERROUTES="no"
TYPE="Ethernet"
UUID="85e14c6b-f9d3-49a0-b566-f122d47001c6"

After reboot, the virtualization host interfaces look similar to: 

2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
       valid_lft forever preferred_lft forever
[...]
4: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 18:66:da:9f:96:d8 brd ff:ff:ff:ff:ff:ff
    inet6 2600:1700:543c:b010:1a66:daff:fe9f:96d8/64 scope global mngtmpaddr dynamic
       valid_lft 2591942sec preferred_lft 604742sec
    inet6 fe80::1a66:daff:fe9f:96d8/64 scope link
       valid_lft forever preferred_lft forever

Configure the Virtual Device on Guest

Follow the guest network reconfiguration procedure and update the network configuration as follows: 

...
<domain type='kvm' id='9'>
  [...]
  <devices>
    [...]
    <interface type='direct'>
      <source dev='em1' mode='private'/>
    </interface>
    <!-- More than one interface can be configured in one step -->
    <interface type='direct'>
      <source dev='em3' mode='private'/>
    </interface>
  </devices>
</domain>

This is the configuration step where the virtualization host network interfaces are chosen. Pick one that connects to the LAN, and another one that connects to a gateway device. <source dev='emX' ... /> is the configuration line that selects the virtualization host interface to attach to. Multiple guests and multiple interfaces on the same guest may attach to the same virtualization host interface.


Multiple guests can attach to the same virtualization host physical interface, and in "private" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("VEPA", "bridge", etc.) are available.

Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows: 

virsh dumpxml <guest-name>

<domain type='kvm' id='9'>
  [...]
  <devices>
    [...]
    <interface type='direct'>
      <mac address='52:54:00:4d:8e:35'/>
      <source dev='em1' mode='private'/>
      <target dev='macvtap4'/>
      <model type='rtl8139'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:43:e0:2f'/>
      <source dev='em3' mode='private'/>
      <target dev='macvtap5'/>
      <model type='rtl8139'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
  </devices>
</domain>

After guest reboot, the MAC address reported by the configuration, on the guest, must corresponds to the MAC address of the corresponding virtualization host macvtap*@em* interfaces.

Optionally, a specific MAC address could be configured on the libvirt guest interface: 

<interface type='direct'>
   ...
   <mac address='...'/>
   ...
</interface>
...

in the guest configuration.

Primary Interface Name Instability


A certain primary interface name instability has been observed after adding the second virtual network interface in the domain definition file. The name of the primary network interface would be assigned non-deterministically to "eth0" and "eth1", respectively, upon successive reboots. An attempt to configure "device" in the domain name interface definition was ineffective. The issue was worked around - without a good explanation of why it is happening - by adding both /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1. As a side effect of the instability, NetworkManager was not able to properly detect the interfaces at boot, so it was disabled: Disable NetworkManager and Configure the Interfaces Manually.

Configure the Network Interface on Guest

eth0 Manual Configuration

⚠️ This is important, if the eth0 is not configured as shown below, the new network interface will not work correctly.

Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains: 

DEFROUTE=no # The default route will be provided by the macvtap interface.
IPV6_DEFROUTE=no
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf

The default route will be provided by the public macvtap interface, so DEFROUTE should be set to "no". Also, we don't want DHCP to modify /etc/resolv.conf so PEERDNS is set to false.

ens8 (macvtap interface that connects to the LAN) Manual Configuration

⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.

Verify that the MAC address of the ens8 interface on guest matches the MAC address of the macvtap4@em1 interface on the virtualization host.

Update /etc/sysconfig/network-scripts/ifcfg-ens8 as follows:

Make sure to configure DEFROUTE="no". For external access to work, the default route should use the public interface. 

DEFROUTE="no" # the public macvtap interface will provide the default route, not this one

Also make sure UUID is unique, you can generate a new UUID with uuidgen.

The configuration should be similar to: 

BOOTPROTO="none"
DEFROUTE=no # the public macvtap interface will provide the default route, not this one
DEVICE="ens8"
DNS1="8.8.8.8"
GATEWAY="192.168.1.1"
IPADDR="192.168.1.15"
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens8"
NETMASK="255.255.255.0"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="24"
TYPE="Ethernet"
UUID=adcf7be8-0dd7-46f5-b572-c6d273e23d87

For more details on how to configure network interfaces, see

Configuring a Network Interface

Second, public macvtap (ens9) interface Manual Configuration

A second macvtap interface connects the guest to public internet. The configuration should be similar to the example provided below, and written in /etc/sysconfig/network-scripts/ifcfg-ens9. "DEFROUTE=yes" is essential.

⚠️ Important: The routable IP address used below must not be used by another guests. 

BOOTPROTO="none"
DEFROUTE="yes" # essential
DEVICE="ens9"
DNS1="8.8.8.8"
GATEWAY="104.50.201.86"
IPADDR="104.50.201.85" # must not conflict with other guests' public address
IPV4_FAILURE_FATAL="yes"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="no"
IPV6_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_PEERDNS="no"
IPV6_PEERROUTES="no"
NAME="ens9"
NETMASK="255.255.255.248"
ONBOOT="yes"
PEERDNS="no"
PEERROUTES="no"
PREFIX="29"
TYPE="Ethernet"
UUID="30C65EA0-1EC3-46C6-8AC1-202D83102B35"

Reboot the second time, both interfaces should be operational.

Verification

After reboot, the guest must be accessible over ssh from an external network.

The routing table should be similar to: 

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         104.50.201.86   0.0.0.0         UG        0 0          0 ens9
104.50.201.80   0.0.0.0         255.255.255.248 U         0 0          0 ens9
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens8
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 ens9
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 ens8
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0

Retrieved from "https://kb.novaordis.com/index.php?title=Attaching_a_Guest_Directly_to_a_Virtualization_Host_Network_Interface_with_a_macvtap_Driver&oldid=97556"