ease cloning of virtual images by disabling mac address rules

What are these devices?

Are they the MAC address of the emulated vmware interface, or the MAC address of the host-side interface?

Hi,
this is for the emulated NICs inside the image - in this case vmware, but AFAIK it works in the same way for xen, virtualbox and maybe kvm.

Please supply the requested information

Attaching "ubuntu-bug udev" from a vmware virtual machine.

I'm not sure if the virtual nics appear to the OS as any different from the actual hardware they are emulating, but I do know all virtual NICs inside vmware vms have MAC addresses starting with 00:0c:29 or 00:50:56.

I think everyone who uses Debian or Ubuntu inside a vmware vm experiences this issue when cloning vms or when the MAC address changes for another reason. udev remembers the old mac address and assigns a new interface name, breaking the network configuration of the machine.

http://coffer.com/mac_find/?string=vmware confirms these two vendor prefixes from VMWare (and two more).

However, I still don't quite understand why merely blacklisting that vendor will help/won't break anything. What about VMs with more than one NIC? Such as a virtualized firewall? They certainly need persistent ethN enumeration just as much as "real" machines with multiple NICs?

Likewise, if a cloned instance gets a new MAC, then the generated rule in /etc/udev/rules.d/70-persistent-net.rules won't match, and thus should be equivalent to not existing at all.

I'm trying to understand what breaks here, and why the blacklisting would help with that.

I hadn't considered vms with multiple nics and having them enumerated correctly might be worth more than fixing this issue.

On an existing vm with one interface, /etc/udev/rules.d/70-persistent-net.rules might be something like:

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM ... ATTR{address}=="00:50:56:80:68:23" ... NAME="eth0"

After a clone the file still has the previous line but also an additional line:

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM ... ATTR{address}=="00:50:56:80:13:f7" ... NAME="eth1"

So the result is there is still only one interface and it's eth1 but the rest of the networking configuration is looking for eth0.

Thanks for the explanation, Matthew.

Seems we are between a rock and a hard place here -- we should not break setups with multiple interfaces, but this is indeed an inconvenience, too. The current udev rules already ignores "xen" devices, that would speak for ignoring VMWare as well for consistency.

Unfortunately we can't cleanly do something like "If the machine only has one NIC, do not generate a persistency file", since they might be added/removed/changed any time.

Server team, do you have some input here? (Also wrt. KVM)

Is there any way that udev could see that although the mac address has changed, the device (PCI device 0x8086:0x100f (e1000)) is the same, so modify the existing rule instead of creating a new one?

The bug here is really no different than if you have a "real" server, and need to change the network card. You'd stop the server, replace the card, start the server and no longer have eth0.

There are a number of things that really *should* be done when "cleaning" an image, this is just one of them. Other things might be removing /root/.bash_history (its an example, possibly not a good one).

I think at the moment this is best left as a "cleaning" step that must occur. Things like this are why I recommend using the pristine tarball rather than a live image when rebundling an ec2 image.

I just re-read the above and see that xen devices are currently ignored. Given that, I do think it makes sense to also handle vmware (and possibly even virtio devices the same way).

That said, we don't want xen, vmware, or any systems with multiple network interfaces to give non-deterministic ordering. If its not already common, I would expect virtual machines with multiple interfaces to be common in the future. It would be nice to something to handle that correctly.

Would it be possible to do add persisitent rules based not on MAC addr, but on bus location? The idea would be that in a new VM, the devices would be attached the same, but different MACs. if the rule said:
  "device on pci bus at 0000:00:19.0 is eth0"
  "device on pci bus at 0000:00:19.1 is eth1"
That might be persistent across VMs (or at very least could be made to be), and would be deterministic for multiple network devices.

Just confirming that this also occurs on a KVM virtual machine, in my case I was using 'virt-clone' to clone a system, which changes the MAC addr... on boot of the clone you have eth0 moved to eth1 due to the persistent udev rule entries.

PCI based addressing ought to work, although this would break the (admittedly rather theoretical) possibility to move them to a different slot, etc. For hotpluggable network cards (USB etc, as you might find them on e. g. the Beagle board or other embedded controllers) we must stay at MAC-based identification.

Just my two cents here:

   I ran into this issue last week. I was attempting to clone a virtual machine on a remote server to create a new VM. The lag between my location and the server is big enough to make doing installations remotely a pain, but the commandline clone functionality is superb. When I cloned the template (which is so named because it is the template for new virtualmachines on this server), and booted the new vm, there was no network connectivity. I logged into the machine through the Virtual Machine Manager utility and removed the udev rules for the network card, but it was still a good half hour searching around to find out why it didn't work by itself.

    Thanks for making such good tools, though.

On Thu, 11 Feb 2010, Martin Pitt wrote:

> PCI based addressing ought to work, although this would break the
> (admittedly rather theoretical) possibility to move them to a different
> slot, etc. For hotpluggable network cards (USB etc, as you might find
> them on e. g. the Beagle board or other embedded controllers) we must
> stay at MAC-based identification.

Ok, so assuming "addressed based persistent naming" is desirable, I have
2 thoughts on how it could enabled:
a.) for all cards on the pci bus, use pci based persistent naming.
  This fails in the moving a card from slot 1 to slot 2 case as you
  mentioned above.
b.) for all devices deemed "virtual" use address based persistent naming.
  This fails if we can't determine something to be "virtual". If we
selectively enable for known MAC addrs OR known drivers, then we'd miss
anywhere where the hypervisor implements a emulated device (pcnet32 or
e1000 in kvm) and selects a MAC address that we don't whitelist.

b, seems safer as we can have more strict control on the white list.

Thoughts?

http://hg.fedorahosted.org/hg/python-virtinst/file/ae6ca033e8c5/virtinst/util.py#l181 indicates that:
- 00:16:3E allocated to xensource
- 52:54:00 used by qemu/kvm

http://libvirt.org/drvesx.html indicates:
- 00:0c:29 and 00:50:56 are owned by vmware

Further, I know that
- d0-0d used by Eucalyptus

maybe it is sufficient to simply apply to these MAC addrs what we already do to xen. It just seems that then we are vulnerable to device/driver ordering issues.

One other thing to consider is that kvm implements pci hotplug for network devices (http://www.linux-kvm.org/page/Hotadd_pci_devices). So whitelisting of kvm addresses would result in inconsistent naming with 2 network devices being added and removed in somewhat arbitrary order.

I moved this out of incomplete and into triaged. I think everyone here understands the nature of the problem.

FWIW Debian has ignored VMware virtual interfaces by mac:

If you grab http://ftp.us.debian.org/debian/pool/main/u/udev/udev_157-1_i386.deb and look at lib/udev/rules.d/70-persistent-net-generator.rules it has:

# ignore interfaces with locally administered or null MAC addresses
# and VMWare virtual interfaces
ENV{MATCHADDR}=="?[2367abef]:*", ENV{MATCHADDR}=""
ENV{MATCHADDR}=="00:00:00:00:00:00", ENV{MATCHADDR}=""
ENV{MATCHADDR}=="00:0c:29:*|00:50:56:*", ENV{MATCHADDR}=""

This has recently been discussed in #udev, and we found that it does not hurt that much to blacklist them. The kernel reliably sorts eth* enumeration by bus number, so as long as you only have cards from one vendor (or more precisely, drivers), the enumeration will be stable. Having cards from different vendors is very unlikely in VMs.

Marco submitted the blacklisting for kvm:

http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;h=af29bf0966994bc233282148f519e12ea29e1c5a

Fixed KVM MAC address range:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;h=c03180194cb62cda286efcc1563391a1408394ff

Added VMWare MACs:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;h=d4de0a0321b471e9dd8c32f1f3266b0b466457c7

udev (162-1) maverick; urgency=low

  * New upstream release. Changes since our previous git snapshot:
    - cdrom_id: Fix DVD-RW media and blank DVD detection.
    - Do not create persistent name rules for kvm/qemu/vmware interfaces.
      (LP: #341006)
  * Add debian/watch.

 -- Martin Pitt <email address hidden> Sun, 12 Sep 2010 20:12:37 +0200

While it helps in a pragmatic kind of way, carving out exemptions is really a kludge.

The issue is that both aspects of the issue are valid. Persistence is important in many situations but in others, it's more important to not have things changing when they don't need to be changing. Arguably, there are many more installations that will only be needing one network card, called eth0 than not but...

At first I thought that the obvious answer is to make this configurable. Obviously, it is in a "You can edit the files" kind of way but perhaps an install option "Use persistent nic labelling y/n" would be the way to go.

Then I thought some more. The real issue is expecting nic labeling to be more than a convenience. There is zero reason to expect eth0 to mean anything. It's not the persistent coding that's at issue, it's the scripts further down the process that are broken. Assigning 192.168.0.5 to eth0 or setting it DHCP? Wrong! The default action should be the scripts to determine the interface based on rules. Probably by "first available interface" for most set-ups and probably by mac address where more complex set-ups are involved.

That's my take on it anyway. Obviously I think more discussion is warranted. If anyone knows of one ongoing, I'd appreciate a link.

Actually, it occurs to me that if the down-process scripts were written how I suggested above, the whole persistent-nic thing would be moot except as a convenience to us silly humans. And that's as it should be.

I guess the meat of the matter boils down to "What to you mean when you say 'eth0'?"

Interface aliases might be another way to do things but I expect that might break a lot of stuff in a lot of nasty ways.

I know it's too old, but today i have big problem without persistent net rules.
(sorry for my English)

I have VMWare 5.5.0 host and Ubuntu 12.04 guest with many virtual NICs (with MACs 00:0c:29*)
Today I updated kernel from 3.13.0-32 to 3.13.0-45
Rebooted VM
Lost connection to my VM over one of the NIC (which is connected to internet).

I connected to VMWare host (by VMware vSphere Client) and to Ubuntu guest (over LAN virtual NIC who worked)

And I saw, that NICs in Ubuntu guest have correct order and matches IP-ETHnumber like before kernel update and reboot, but IP-ETH does not match MAC-VLAN bindings in VMWare, except eth0 (LAN NIC).

I don't understend how it happened, as I remember, I updated kernel before many times (to 3.13.0-32) without problems.

So today I am commenting line after "ignore VMWare virtual interfaces" to make IP-ETH-MAC match persistent.

This is from dmesg:
[ 0.932308] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps
[ 0.965537] vmxnet3 0000:0c:00.0 eth1: NIC Link is Up 10000 Mbps
[ 0.973137] vmxnet3 0000:13:00.0 eth2: NIC Link is Up 10000 Mbps
[ 0.979222] vmxnet3 0000:14:00.0 eth3: NIC Link is Up 10000 Mbps
[ 0.985227] vmxnet3 0000:1b:00.0 eth4: NIC Link is Up 10000 Mbps
[ 4.733558] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4.733563] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 4.733565] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[ 4.733568] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[ 4.733570] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready
[ 6.543452] vmxnet3 0000:0c:00.0 eth1: intr type 3, mode 0, 2 vectors allocated
[ 6.543850] vmxnet3 0000:0c:00.0 eth1: NIC Link is Up 10000 Mbps
[ 6.545858] vmxnet3 0000:1b:00.0 eth4: intr type 3, mode 0, 2 vectors allocated
[ 6.546065] vmxnet3 0000:1b:00.0 eth4: NIC Link is Up 10000 Mbps
[ 6.547294] vmxnet3 0000:13:00.0 eth2: intr type 3, mode 0, 2 vectors allocated
[ 6.547496] vmxnet3 0000:13:00.0 eth2: NIC Link is Up 10000 Mbps
[ 6.548714] vmxnet3 0000:0b:00.0 eth0: intr type 3, mode 0, 2 vectors allocated
[ 6.549006] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps
[ 6.550449] vmxnet3 0000:14:00.0 eth3: intr type 3, mode 0, 2 vectors allocated
[ 6.550716] vmxnet3 0000:14:00.0 eth3: NIC Link is Up 10000 Mbps

Realy wrong order?

Just my two cents here regarding #21:

Sorting is fine, as long as the network adapters are stable. I today had a re-ordering issue with a VM, which previously had two interfaces and was configured as such. Then I added another one, and it became eth0, changing the numbering of all other VMs.

I understand the inconvenience that occurs when cloning a VM, but in my opinion the current state is not a solution either.