dynamic block device attach/detach not functional with karmic KVM

Dan-

What would you say the priority of this is?

Is Eucalyptus dependent on this functionality?

:-Dustin

Not Dan, but here's my take anyway.

I would say it is fairly critical. The EBS functionality (on which the block device attach/detach functionality depends) is a pretty important feature. Without, data persistence is not obvious. For example, you would run MySQL in the cloud by keeping table data and indices on an EBS volume.

I agree with Etienne on this issue; another thing to consider is that many virtual appliances that are developed for EC2/UEC will be depending on EBS functionality, due to the persistence observation made above.

Okay, any chance you can help narrow down where this functionality regressed?

:-Dustin

Dustin, here is my progress thus far. On a install of Karmic alpha6 CD,
with qemu-kvm 0.11.0~rc2-0ubuntu4

FIRST:
-----------------------------------
boot VM
   ubuntu jaunty
   kernel 2.6.28-11-generic

load 'acpiphp' module in VM
   modprobe acpiphp

try 'virsh attach-device
   dd if=/dev/zero of=/tmp/foobar bs=1M count=64
   cat /tmp/meh
      <disk type='block'><driver name='phy'/><source
dev='/tmp/foobar'/><target dev='sdb'/></disk>
   virsh attach-device 1 /tmp/meh
      Connecting to uri: qemu:///system
      Device attached successfully

no dice, just see some SCSI init stuff in dmesg (see bug report), but no
actual disk devices are showing up

NEXT:
------------------------------------------

roll back qemu-kvm to version 0.11.0~rc2-0ubuntu2, same result

NEXT:
-------------------------------------------

grab the 'kvm' binary from a jaunty box (1:84+dfsg-0ubuntu12.3), throw it in
/usr/bin on the karmic box, softlink /usr/share/qemu to /usr/share/kvm -
success! Somewhere between this version and 0ubuntu2, the problem cropped
up. I'm not sure if this is helpful or not. Here is the nice dmesg I get
with the older KVM, in the guest:

[ 258.062895] pci 0000:00:07.0: reg 10 io port: [0x00-0xff]
[ 258.062964] pci 0000:00:07.0: reg 14 32bit mmio: [0x000000-0x0003ff]
[ 258.063026] pci 0000:00:07.0: reg 18 32bit mmio: [0x000000-0x001fff]
[ 258.063692] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
[ 258.063868] decode_hpp: Could not get hotplug parameters. Use defaults
[ 258.092439] sym53c8xx 0000:00:07.0: enabling device (0000 -> 0003)
[ 258.093953] sym53c8xx 0000:00:07.0: PCI INT A -> Link[LNKC] -> GSI 10
(level, high) -> IRQ 10
[ 258.097161] sym1: <895a> rev 0x0 at pci 0000:00:07.0 irq 10
[ 258.100701] sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking
[ 258.104043] sym1: SCSI BUS has been reset.
[ 258.113949] scsi3 : sym-2.2.3
[ 261.120993] scsi 3:0:1:0: Direct-Access QEMU QEMU HARDDISK
0.9. PQ: 0 ANSI: 3
[ 261.121008] scsi target3:0:1: tagged command queuing enabled, command
queue depth 16.
[ 261.121038] scsi target3:0:1: Beginning Domain Validation
[ 261.122125] scsi target3:0:1: Domain Validation skipping write tests
[ 261.122129] scsi target3:0:1: Ending Domain Validation
[ 261.133097] sd 3:0:1:0: [sdb] 131072 512-byte hardware sectors: (67.1
MB/64.0 MiB)
[ 261.133285] sd 3:0:1:0: [sdb] Write Protect is off
[ 261.133290] sd 3:0:1:0: [sdb] Mode Sense: 14 00 00 00
[ 261.133610] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[ 261.134228] sd 3:0:1:0: [sdb] 131072 512-byte hardware sectors: (67.1
MB/64.0 MiB)
[ 261.134426] sd 3:0:1:0: [sdb] Write Protect is off
[ 261.134431] sd 3:0:1:0: [sdb] Mode Sense: 14 00 00 00
[ 261.134738] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled,
doesn't support DPO or FUA
[ 261.134752] sdb: unknown partition table
[ 261.137920] sd 3:0:1:0: [sdb] Attached SCSI disk
[ 261.138465] sd 3:0:1:0: Attached scsi generic sg2 type 0

Let me know how you would like me to proceed, if I can help in any way!

-d

On Sat, Sep 19, 2009 at 10:15 PM, Dustin Kirkland <<email address hidden>
> wrote:

> Okay, any cha...

Apparently a regression in kvm-qemu, not something to fix in the Eucalyptus package... please reopen if I got it wrong.

Patch attached.

:-Dustin

Sorry, no patch. Disregard that.

This appears possibly related:
 * https://bugzilla.redhat.com/show_bug.cgi?id=510425
 * http://www.redhat.com/archives/libvir-list/2009-July/msg00115.html

However, we already have that code in our libvirt. Mentioning here for completeness.

:-Dustin

I don't think this is a libvirt problem. It also appears intermittent here. In doing other testing I was looping attach and detach and found that kvm (presumably) died, with the guest disappearing (and needed to be started again). Attached is a (very crude) test script that can be called with:

$ ./432154.sh <vm name> device # for attach-device
$ ./432154.sh <vm name> disk # for attach-disk

Don't use this on a VM you care about. Also, I found that kvm died late in the guest boot process (my testing was with a 32 bit Jaunty install on x86_64 host). Also, while the test script tries to be careful about restoring the guest xml, it doesn't always work, and you may need to remove the attach device by editing the XML via dumpxml/edit xml/define.

Let me be clear: my testing is on a 9.10 x86_64 host with qemu-kvm 0.11.0~rc2-0ubuntu4 and a Jaunty 32 bit guest.

Okay, I just took libvirt out of the equation.

Dropping to a qemu console (ctrl-alt-2), and running pci_add, according to:
 * http://www.linux-kvm.org/page/Hotadd_pci_devices
crashes qemu-kvm:

kvm: /build/buildd/qemu-kvm-0.11.0~rc2/hw/pci.c:1029: pci_qdev_init: Assertion `pci_dev' failed.

:-Dustin

Okay, my core dump is actually Bug #419590, which results from a syntax error.

From the qemu console, the proper invocation is:
pci_add auto storage file=image.img,if=virtio

Note "auto" rather than 1 or 0.

So with that, the command succeeds and doesn't crash the VM. However, I'm not yet seeing the block device dynamically pop up in the guest.

:-Dustin

Okay, and now I am able to successfully attach a device to a running VM in both the qemu console, and from virt-manager (which uses libvirt). In both cases, I'm seeing the device dynamically pop up in the guest. Note that I had to modprobe acpiphp. Perhaps we should consider loading this module in init or in modules.conf.

:-Dustin

And I can also confirm Jamie's report. This is intermittent. :-/

:-Dustin

My crash after several iterations turns out to be not intermittent at all. I decided to add a counter and see how many times I could attach/detach before the crash. Interestingly, it is 28 times, every time. Attached is an updated reproducer with the counter and an added 'sleep 30' after starting the VM to give it a chance to fully boot before attach/detach. This crash feels like a different bug than Daniel's.

all, here is as much information as I could think of in an attempt to allow for reproducability (see attached file). Here is what each file in the attached tarball contains:

setup - quick description of machine set up

orig.libvirt.xml - the XML vm description that is passed to libvirt
kvmcmdline - upon VM run, this is the kvm process (with cmdline arguments) that is actually running
dumpxml.libvirt.xml - virsh dumpxml output from running VM

i-42CC08B6.log.b4-attach - /var/log/libvirt/qemu/... logfile of the running VM before attach
vm_dmesg.b4-attach - dmesg output from inside VM before attach
vm_dmesg.after-attach - dmesg output from inside VM after attach
i-42CC08B6.log.after-attach - /var/log/libvirt/qemu/... logfile of the running VM after attach

attach-device.cmds - script of commands ran to attach disk
attach-device.xml - input to virsh attach-device

-------------------------

with dustin's instuctions above (sending pci_add directly to KVM through the monitor), I was able to reproduce the problem when if=scsi. with if=virtio, the device shows up just fine.

all, here is as much information as I could think of in an attempt to allow for reproducability (see attached file). Here is what each file in the attached tarball contains:

setup - quick description of machine set up

orig.libvirt.xml - the XML vm description that is passed to libvirt
kvmcmdline - upon VM run, this is the kvm process (with cmdline arguments) that is actually running
dumpxml.libvirt.xml - virsh dumpxml output from running VM

i-42CC08B6.log.b4-attach - /var/log/libvirt/qemu/... logfile of the running VM before attach
vm_dmesg.b4-attach - dmesg output from inside VM before attach
vm_dmesg.after-attach - dmesg output from inside VM after attach
i-42CC08B6.log.after-attach - /var/log/libvirt/qemu/... logfile of the running VM after attach

attach-device.cmds - script of commands ran to attach disk
attach-device.xml - input to virsh attach-device

-------------------------

with dustin's instuctions above (sending pci_add directly to KVM through the monitor), I was able to reproduce the problem when if=scsi. with if=virtio, the device shows up just fine.

Dustin Kirkland wrote:
> Note that I had to modprobe acpiphp. Perhaps we should consider loading
> this module in init or in modules.conf.

I have reported this issue in bug #364916.

On Tue, 29 Sep 2009, Etienne Goyer wrote:

> Dustin Kirkland wrote:
> > Note that I had to modprobe acpiphp. Perhaps we should consider loading
> > this module in init or in modules.conf.
>
> I have reported this issue in bug #364916.

Well, specifically in that bug you reported that there was no acpiphp to
load. Now, the -virtual kernel has that module (per bug 364916) and
-virtual package is installed in the image (bug 429169).

If we think acpiphp.ko needs to be loaded (insmod/modprobe) by default in
uec images, then we should bring that up in a different bug.

Another option that would work is CONFIG_HOTPLUG_PCI_ACPI=y (as Etienne
originally suggested)

> Another option that would work is CONFIG_HOTPLUG_PCI_ACPI=y (as Etienne
> originally suggested)

Per Tim Gardner in irc:
<rtg> smoser, I think you're gonna have to go the initramfs route.
<smoser> i accept your answer, but just curious, why not build in?
<rtg> smoser, regression potential for existing platforms. its kind of a
      major hunk of code to impose at this late date.

On Tue, Sep 29, 2009 at 1:24 PM, Scott Moser <email address hidden> wrote:
> Another option that would work is CONFIG_HOTPLUG_PCI_ACPI=y (as Etienne
> originally suggested)

I think we need to do this. Are there any circumstances where we
would *not* want that?

:-Dustin

Ah, I just saw Scott's quote of Tim's response.

:-Dustin

Soren,

Any chance you can lend a hand with this bug?

:-Dustin

Here's a status update...

I can reproduce this problem as follows, without libvirt or eucalyptus in the picture...

Create some small auxiliary storage to dynamically add to a running VM.
  (host) $ dd if=/dev/zero of=/tmp/foo bs=1M count=64

Boot a vm in Karmic under qemu-kvm-0.11.0 like this:
  (host) $ kvm -drive file=karmic-server.img,if=scsi,boot=on -boot c

To reproduce the problem as Eucalyptus is experiencing it, you *must* use if=scsi. This problem manifests itself when *subsequent* scsi storage devices are added. If the device is the *first* scsi device on the system, then it would succeed. It's the 2nd scsi device that causes the problem.

Once the system is booted, login and in the vm, load this module:
  (vm) $ sudo modprobe acpiphp

Check that the acpiphp slots are loaded in dmesg. And note that there is only one /dev/sd[a-z] device.

Now, drop to the qemu console with ctrl-alt-2, and add the storage:
  (qemu) pci_add auto storage file=/tmp/foo,if=scsi
  OK domain 0, bus 0, slot 6, function 0

Switch back to the vm linux shell with ctrl-alt-1, and look at the dmesg output.
  (vm) $ dmesg | tail -n 12
[ 44.033397] pci 0000:00:06.0: reg 10 io port: [0x00-0xff]
[ 44.033443] pci 0000:00:06.0: reg 14 32bit mmio: [0x000000-0x0003ff]
[ 44.033486] pci 0000:00:06.0: reg 18 32bit mmio: [0x000000-0x001fff]
[ 44.033899] pci 0000:00:02.0: BAR 6: bogus alignment [0x0-0x0] flags 0x2
[ 44.033975] decode_hpp: Could not get hotplug parameters. Use defaults
[ 44.042277] sym53c8xx 0000:00:06.0: enabling device (0000 -> 0003)
[ 44.043230] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[ 44.043247] sym53c8xx 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, high) -> IRQ 11
[ 44.045237] sym1: <895a> rev 0x0 at pci 0000:00:06.0 irq 11
[ 44.047586] sym1: No NVRAM, ID 7, Fast-40, LVD, parity checking
[ 44.055399] sym1: SCSI BUS has been reset.
[ 44.063329] scsi3 : sym-2.2.3

More importantly, note that no /dev/sd[b-z] device shows up.

This is Eucalyptus' use case (though they use libvirt to do the above).

The official, supported way of doing this according to upstream qemu and kvm is to use virtio instead of scsi. They have acknowledged that scsi is buggy, and can in fact lose data when the buffer is saturated.

Truly, if you now drop to the qemu console with ctrl-alt-2, and do this:
  (qemu) pci_add auto storage file=/tmp/foo,if=virtio
  OK domain 0, bus 0, slot 7, function 0

Going back to the vm with ctrl-alt-1, you can now see a new /dev/vda device registered. Eucalyptus notes that this is not ideal because the device is called /dev/vda instead of /dev/sda or /dev/sdb. They are concerned that this breaks compatibility with EC2 images which expect disks in the /dev/sd[a-z] namespace, particularly because some of these images hardcode such device names in /etc/fstab. Fortunately, in Ubuntu we use UUIDs, so we don't suffer from this, really, in our guests.

Now, all of that said, it is actually possible to hot-add a second scsi device (though upstream recommends doing so through virtio instead). However, this method is not yet supported by libvirt. The key is that with modern qemu, you have to add a driv...

As discussed in Friday's release meeting, using virtio seems to be better than having no attach/detach at all for 9.10

My experiments with hot-attaching virtio disks to eucalyptus VMs have been partially successful. I've been able to attach/detach a virtio disk to/from a VM once, but the second attach causes a kvm segfault followed by a libvirtd segfault. Removing eucalyptus from the scenario, this problem can be reproduced by passing the attach/detach/attach commands directly using virsh as follows. Note that, although i'm using an AOE device as the 'source' in the following example, the same problem exists if you replace the source dev with a local file.

root@sg:~# ps ax | grep kvm
25947 ? Sl 0:10 /usr/bin/kvm -S -M pc-0.11 -m 128 -smp 1 -name i-3D7E06DF -uuid 8f09357d-c295-523b-97fa-b72e0a46497b -nographic -monitor unix:/var/run/libvirt/qemu/i-3D7E06DF.monitor,server,nowait -boot c -kernel /var/lib/eucalyptus/instances/admin/i-3D7E06DF/kernel -initrd /var/lib/eucalyptus/instances/admin/i-3D7E06DF/ramdisk -append root=/dev/sda1 console=ttyS0 -drive file=/var/lib/eucalyptus/instances/admin/i-3D7E06D /disk,if=scsi,index=0,boot=on -net nic,macaddr=d0:0d:3d:7e:06:df,vlan=0,model=e1000,name=e1000.0 -net tap,fd=17,vlan=0,name=tap.0 -serial file:/var/lib/eucalyptus/instances/admin/i-3D7E06DF/console.log -parallel none -usb

root@sg:~# cat meh
<disk type='block'><driver name='virtio'/><source dev='/dev/etherd/e0.7'/><target dev='vda' bus='virtio'/></disk>

root@sg:~# virsh list
Connecting to uri: qemu:///system
 Id Name State
----------------------------------
  1 i-3D7E06DF running

root@sg:~# virsh attach-device 1 meh
Connecting to uri: qemu:///system
Device attached successfully

root@sg:~# virsh detach-device 1 meh
Connecting to uri: qemu:///system
Device detached successfully

root@sg:~# virsh attach-device 1 meh
Connecting to uri: qemu:///system
error: Failed to attach device from meh
error: server closed connection

[44227.486821] kvm[25947]: segfault at 8d0 ip 000000000041ccd7 sp 00007fff99644d70 error 6 in qemu-system-x86_64[400000+226000]
[44227.730408] libvirtd[25467]: segfault at 70 ip 00000000004345a2 sp 00007f5b3f5e3b70 error 4 in libvirtd[400000+77000]

Am I perhaps passing an incorrectly formatted disk XML for virtio? Versions:

Host:
root@sg:~# apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.7.0-1ubuntu10
  Candidate: 0.7.0-1ubuntu10
  Version table:
 *** 0.7.0-1ubuntu10 0
        500 http://us.archive.ubuntu.com karmic/main Packages
        100 /var/lib/dpkg/status
root@sg:~# apt-cache policy qemu-kvm
qemu-kvm:
  Installed: 0.11.0-0ubuntu1
  Candidate: 0.11.0-0ubuntu1
  Version table:
 *** 0.11.0-0ubuntu1 0
        500 http://us.archive.ubuntu.com karmic/main Packages
        100 /var/lib/dpkg/status
root@sg:~# uname -a
Linux sg 2.6.31-12-server #41-Ubuntu SMP Wed Oct 7 20:33:27 UTC 2009 x86_64 GNU/Linux

Guest:
root@ubuntu:~# uname -a
Linux ubuntu 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:58:03 UTC 2009 x86_64 GNU/Linux

I found this bit of documentation, which I thought I'd link here since it's pertinent:
 * http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Virtualization_Guide/chap-Virtualization-KVM_Para_virtualized_Drivers.html

:-Dustin

Dan-

I have reproduced your crash on the second invocation of adding a virtio disk.

I'm looking for a fix now.

:-Dustin

I think this is related to Bug #419590.

:-Dustin

This bug was fixed in the package qemu-kvm - 0.11.0-0ubuntu5

---------------
qemu-kvm (0.11.0-0ubuntu5) karmic; urgency=low

  [ Dustin Kirkland ]
  * debian/patches/10_fix_scsi_hotplug.patch: cherry-pick patch from
    upstream to fix scsi hotplug regression, LP: #432154
  * debian/patches/11_fix_virtio-blk_hot_add_after_remove.patch: cherry-pick
    patch from upstream to fix virtio hotplug add/remove, LP: #419590

  [ James Westby ]
  * Add transitional kvm and qemu packages, LP: #451508
    - Force the kvm package version to be higher so that it supercedes that
      from the kvm source. Thanks to Steve Langasek and Michael Vogt

 -- Dustin Kirkland <email address hidden> Wed, 14 Oct 2009 11:35:27 -0500

I believe this bug is still existing in the latest 10.10 server release. When attaching a volume the nc.log shows that it was properly mounted via iSCSI and one can access it on the node but it will not appear in the guest. The guest is the latest UEC release of Lucid. Can anybody confirm that this problem exists in 10.10 UEC?

@Wolfgang I believe this is still an issue in maverick/10.10 aswell. I can attach a volume with UEC (10.10 metal and 10.10 instance).

If you issue a `shutdown -r now` within the instance, the volume will be attached after the reboot... just spent about 8 hours getting that far!

@Wolfgang and @Kiall: are you sure that the acpiphp module is loaded
in your guest?

Yes - 100% sure.

I've since discovered that using /dev/vdX works without a reboot, while /dev/sdX does not work until after a reboot!

vdX triggers the use of virtio, while sdX uses the "standard" method .. whatever that is! I stopped digging once i got it working with vdX :)

@Serge: Yes acpiphp is definitely loaded. I used the UEC images.

@Kiall: Thanks for the hint about vdX. Does that mean you just specify (for example) vda when attaching the device to the instance and that will work? I have no cluster installed with which i could test atm.

@Wolfgang: Yes - Specifying, for example, /dev/vda triggers the use of virtio over the "standard" - and works exactly as you would expect EBS volumes to :)

There *is* still a bug preventing /dev/sdX from working .. but I've sank too much time into finding it at the moment, I'll probably dig in again later.

I spent about 3 hr before I can find correct command to attach iSCSI /dev/vda to my instance
euca-attach-volume vol-59A00634 -d /dev/vdb -i i-511908A5
I hope this tip save time for somebody.