Attaching a Guest Directly to a Virtualization Host Network Interface with a macvtap Driver: Difference between revisions
| Line 21: | Line 21: | ||
<font size=-2> | <font size=-2> | ||
[...] | [...] | ||
<font color= | <font color=SlateGray># this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35</font> | ||
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 | 22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 | ||
link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff | link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff | ||
Revision as of 00:33, 6 August 2023
External
Internal
- Linux Virtualization Networking Concepts
- Linux Virtualization Operations
- virt-install | Network Post-Configuration
Overview
This procedure assumes the guest wants to have at least two kinds of network interfaces: one that connects the guest to an internal virtual network, used by guests to communicate with each other (eth0), and the others, enabled by a macvtap driver, attached directly to the visualization host network interfaces (ens8, ens9). These virtualization host interfaces can be connected to different networks. Note that at least two of these virtual network interfaces are necessary when the host services public addresses: one of the interfaces will be used to service external traffic, and the other, internal LAN (192.168.1.*) traffic.
The procedure is written for two of such network interfaces. The first one is assumed to connect to the internal LAN, and the second one to the public network.
The procedure leaves the virtualization host network interfaces that will be used for direct attachment unconfigured (the hardware will be used directly by the guest network stack) and declares a network interface of type "direct" in the guest configuration.
After assignment, and after the guest boots, the corresponding virtualization host interfaces show up as follows:
[...]
# this macvtap interface corresponds to the ens8 (local network) interface on guest. The common element is the MAC address: 52:54:00:4d:8e:35
22: macvtap4@em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
link/ether 52:54:00:4d:8e:35 brd ff:ff:ff:ff:ff:ff
inet6 fe80::5054:ff:fe4d:8e35/64 scope link
valid_lft forever preferred_lft forever
The macvtap label index may vary, even between boots. It is probably possible to request specific labels when configuring the interface on the guest.
The same interface shows up as ens9 (or similar) on the guest:
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff
inet 104.50.201.84/29 brd 104.50.201.87 scope global noprefixroute ens9
valid_lft forever preferred_lft forever
The virtualization host macvtap interface (macvtap1@em2) and the guest interface have the same MAC address. In the example above is 52:54:00:5d:9c:33.
Note that multiple macvtap guest interfaces (ensX) can be set at the same time, binding the guest to the hardware of different network interfaces on the virtualization host.
At the same time, multiple independent macvtap interfaces can be set on the same virtualization host physical network interface, sharing the hardware. They can be configured independently with different IP addresses. More details are available in the Configure the Virtual Device on Guest section, below.
Procedure
Configure the Physical Network Interface on the Virtualization Host
Leave the network interface unconfigured on the virtualization host. The corresponding configuration file /etc/sysconfig/network-scripts/ifcfg-em2 should show the following (only essential configuration is shown):
NAME=em2
DEVICE=em2
TYPE=Ethernet
ONBOOT=no # No network interface should be started at boot, the hardware will be shared by guests
BOOTPROTO=no
DEFROUTE=no # Does not matter, will not be used by the virtualization host
PEERDNS=no
PEERROUTES=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=85e14c6b-f9d3-49a0-b566-f122d47001c6
After reboot, it should look like this:
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 18:66:da:9f:96:d7 brd ff:ff:ff:ff:ff:ff
inet6 2600:1700:dc41:3940:1a66:daff:fe9f:96d7/64 scope global mngtmpaddr dynamic
valid_lft 3543sec preferred_lft 3543sec
inet6 fe80::1a66:daff:fe9f:96d7/64 scope link
valid_lft forever preferred_lft forever
The example above applies to a network interface that is left unconfigured. It is also possible to use an interface that is configured and used by the virtualization host for its own purposes:
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 18:66:da:9f:96:d6 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.13/24 brd 192.168.1.255 scope global em1
valid_lft forever preferred_lft forever
inet6 fe80::1a66:daff:fe9f:96d6/64 scope link
valid_lft forever preferred_lft forever
Configure the Virtual Device on Guest
Follow the guest network reconfiguration procedure and update the network configuration as follows:
...
<interface type='direct'>
<source dev='em2' mode='private'/>
</interface>
<!-- More than one interface can be configured in one step -->
<interface type='direct'>
<source dev='em1' mode='private'/>
</interface>
...
where "em2" is the virtualization host network interface to be directly exposed to the guest.
Multiple guests can attach to the same virtualization host physical interface, and in "private" mode, they will be isolated from each other. Each guest can be individually configured with IP addresses that make sense on the network the physical address is connected to, while the corresponding network interface on the virtualization host will stay unconfigured. Other modes ("VEPA", "bridge", etc.) are available.
Note that after creating the domain based on the XML configuration, libvirt will update the stored configuration as follows:
virsh dumpxml <guest-name>
...
<interface type='direct'>
<mac address='52:54:00:5d:9c:33'/>
<source dev='em2' mode='private'/>
<target dev='macvtap1'/>
<model type='rtl8139'/>
<alias name='net2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
...
After guest reboot, the MAC address reported by the configuration must corresponds to the MAC address of the virtualization host macvtap1@em2 interface on the virtualization host, and it must be different from the MAC address of the virtualization host em2 interface.
Optionally, a specific MAC address could be configured on the libvirt guest interface:
<interface type='direct'>
...
<mac address='...'/>
...
</interface>
...
in the guest configuration.
The new network interface, in this case ens9, shows up on the guest, alongside the default network interface eth0:
<syntaxhighlight lang='bash'>
ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
[...]
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:5d:9c:33 brd ff:ff:ff:ff:ff:ff
inet 104.50.201.84/29 brd 104.50.201.87 scope global noprefixroute ens9
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:b4:87:0f brd ff:ff:ff:ff:ff:ff
inet 192.168.122.249/24 brd 192.168.122.255 scope global noprefixroute dynamic eth0
The MAC address reported by the guest mavtap ens9 interface must be the same as the MAC address of the corresponding mavtap1@em2 on the virtualization host.
Primary Interface Name Instability
A certain primary interface name instability has been observed after adding the second virtual network interface in the domain definition file. The name of the primary network interface would be assigned non-deterministically to "eth0" and "eth1", respectively, upon successive reboots. An attempt to configure "device" in the domain name interface definition was ineffective. The issue was worked around - without a good explanation of why it is happening - by adding both /etc/sysconfig/network-scripts/ifcfg-eth0 and /etc/sysconfig/network-scripts/ifcfg-eth1. As a side effect of the instability, NetworkManager was not able to properly detect the interfaces at boot, so it was disabled: Disable NetworkManager and Configure the Interfaces Manually.
Configure the Network Interface on Guest
eth0 Manual Configuration
⚠️ This is important, if the eth0 is not configured as shown below, the new network interface will not work correctly.
Make sure that /etc/sysconfig/network-scripts/ifcfg-eth0 contains:
DEFROUTE=no # The default route will be provided by the macvtap interface.
IPV6_DEFROUTE=no
PEERDNS=no # We do not want DHCP to modify /etc/resolv.conf
The default route will be provided by the macvtap interface, so DEFROUTE should be set to "no". Also, we don't want DHCP to modify /etc/resolv.conf so PEERDNS is set to false.
ens9 (macvtap interface) Manual Configuration
⚠️ This is important, if the macvtap interface is not configured as shown below, it will not work correctly.
Verify that the MAC address of the ens8 interface on guest matches the MAC address of the macvtap4@em1 interface on the virtualization host.
Update /etc/sysconfig/network-scripts/ifcfg-ens8 as follows:
Make sure to configure:
DEFROUTE=yes # macvtap interface will provide the default route
Also make sure UUID is unique, you can generate a new UUID with uuidgen.
The configuration should be similar to:
NAME=ens9
DEVICE=ens9
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=104.50.201.84
NETMASK=255.255.255.248
PREFIX=29
GATEWAY=104.50.201.86
DEFROUTE=yes # macvtap interface will provide the default route
PEERROUTES=no
PEERDNS=no
DNS1=8.8.8.8
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=010B8B2C-9809-48E1-B14F-0BF224A8EFB2
For more details on how to configure network interfaces, see
Second macvtap interface Manual Configuration
In case a second macvtap interface connects the guest to an internal network, the configuration should be similar to the example provided below. Note the "DEFROUTE=no" configuration.
NAME=ens8
DEVICE=ens8
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
PREFIX=24
GATEWAY=192.168.1.1
DEFROUTE=no
PEERROUTES=no
PEERDNS=no
DNS1=8.8.8.8
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
IPV6_AUTOCONF=no
IPV6_DEFROUTE=no
IPV6_PEERDNS=no
IPV6_PEERROUTES=no
IPV6_FAILURE_FATAL=no
UUID=7af78337-479d-47e2-8b16-a04dc42227c8
Reboot the second time, both interfaces should be operational.
Verification
After reboot, the guest must be accessible over ssh from an external network.
The routing table should be similar to:
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 104.50.201.86 0.0.0.0 UG 0 0 0 ens9
104.50.201.80 0.0.0.0 255.255.255.248 U 0 0 0 ens9
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ens8
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0