qemu-img convert blocks other tasks

Thanks for taking the time to report this bug and helping to make Ubuntu better.

For my own test on a natty server, I started with a 30G qcow2 disk with a lucid install on it. It had about 1G allocated. Conversion from qcow2 to raw took probably less than 10 seconds. Conversion back to qcow took a lot longer (perhaps a minute). So for a full 40G allocated drive I certainly would expect it to be slow.

However you certainly do seem to have a real problem there. In your CurrentDmesg, I see

[257894.409748] Buffer I/O error on device dm-9, logical block 0
[257894.409786] Buffer I/O error on device dm-9, logical block 0

This could indicate a real problem, or just a device which you've since removed (i.e. usb thumb drive). Could you look under /sys/block/dm-9 for more information?

Can you look for relevant info in /var/log/syslog from this event and paste them here?

Finally, could you re-test with the qemu-kvm package from the server-edgers's archive (see https://launchpad.net/~ubuntu-server-edgers/+archive/server-edgers-qemu-kvm)?

You're right about the errors. Apparently I had a full snapshot, but it doesn't make a difference.

Meanwhile, I've had the time to test this. It is easy to reproduce:

- Create a snapshot (1 is enough)
- Cause some I/O, like qemu-img convert (this does /not/ have to be the snapshotted volume)
- Watch your virtual machines crash and/or your system burn

Doing the same qemu-img without any snapshots present is no problem.

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]

It's not good to let this bug expire. LVM snapshots are a severe risk and an important feature on file servers.

Does this:

kvm D 0000000000000000 0 9632 1 0x00000000
 ffff8801a4269ca8 0000000000000086 0000000000015bc0 0000000000015bc0
 ffff8802004fdf38 ffff8801a4269fd8 0000000000015bc0 ffff8802004fdb80
 0000000000015bc0 ffff8801a4269fd8 0000000000015bc0 ffff8802004fdf38
Call Trace:
 [<ffffffff815596b7>] __mutex_lock_slowpath+0x107/0x190
 [<ffffffff815590b3>] mutex_lock+0x23/0x50
 [<ffffffff810f5899>] generic_file_aio_write+0x59/0xe0
 [<ffffffff811d7879>] ext4_file_write+0x39/0xb0
 [<ffffffff81143a8a>] do_sync_write+0xfa/0x140
 [<ffffffff81084380>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81252316>] ? security_file_permission+0x16/0x20
 [<ffffffff81143d88>] vfs_write+0xb8/0x1a0
 [<ffffffff81144722>] sys_pwrite64+0x82/0xa0
 [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
kdmflush D 0000000000000002 0 396 2 0x00000000
 ffff88022eeb3d10 0000000000000046 0000000000015bc0 0000000000015bc0
 ffff88022f489a98 ffff88022eeb3fd8 0000000000015bc0 ffff88022f4896e0
 0000000000015bc0 ffff88022eeb3fd8 0000000000015bc0 ffff88022f489a98

show up in the guest, or the host? (I'd assume and hope the guest).

Does this happen only with LVM volumes?

For LVM, does it help if you create a temporary readonly snapshot of the volume you want to copy, use qemu-img on the snapshot, and then delete the snapshot? Does that allow the VM to continue running with the original volume with no errors?

Unfortunately, this shows up in both host and guests.

I haven't tried to use qemu-img on a snapshot, but I will. I fear it will not make much difference. Just having snapshots (without accessing them) is enough to bring the system down on moments with heavy I/O.

Sometime this week, I will warn the users of their impending doom and try out whether it makes a difference.

Quoting Alvin (<email address hidden>):
> Unfortunately, this shows up in both host and guests.
>
> I haven't tried to use qemu-img on a snapshot, but I will. I fear it
> will not make much difference. Just having snapshots (without accessing
> them) is enough to bring the system down on moments with heavy I/O.
>
> Sometime this week, I will warn the users of their impending doom and
> try out whether it makes a difference.

I assume you have very fast underlying storage and that's the trigger.
Rather than running qemu-img under 'nice', I wonder whether you would
be better off running qemu-img in a cgroup with tighter controls over
its blkio. I've not experimented with this myself, but
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/cgroups/blkio-controller.txt;h=465351d4cf853e8a308c9c84abef789b3dcfa42c;hb=HEAD
should give some good info.

I also must point out that the only storage failures I've seen myself
(roughly a year ago) were due to ext4 itself. I since switched to
xfs for all root and data partitions and have not seen the like since.
Unfortunately that's likely not a viable option for you as you have
users already depending on the system (plus 2.6.38 is rumored to bring
its own xfs instabilities, though I've not seen those here). However
if it were possible, I'd highly recommending testing with xfs.

I've tried again to reproduce this to no avail.

Do you get different behavior if you do

   qemu-img convert -f raw -O raw /dev/yourvg/yourlv lv1.raw

and if you do

   dd if=/dev/yourvg/yourlv of=lv2.raw

?

If not, then the compression if you're using it, or simply memory pressure appears to be the cause, so beside ionice you may want to try to simply use 'nice'

Also, please let us know if using a cgroup works for you (and, if not, please provide all the settings you used).

I still haven't had the chance to test this properly. (server is in production), but the problem manifested itself by accident. So, here's a little bit of information.

I started downloading (rsync) a qcow image from the file server and noticed it was bit slow. 14MB/sec max - 790kB/s after a while. This is writing to a simple single sata disk. Then the virtual machines started to become unresponsive and I had to interrupt the download, or the server would have gone down. It's not memory pressure. Memory was at +/- 900MB (of 8GB). Only 2 small virtual machines were running, BUT there was a snapshot present.

It's not qemu-img that causes the panics. Any I/O will do it. Then the famous messages start appearing in kern.log:
[149164.740056] INFO: task kvm:21354 blocked for more than 120 seconds.
[149164.740294] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[...]
[149164.740595] [<ffffffff8155d557>] __mutex_lock_slowpath+0x107/0x190

Restarting the download without lvm snapshot present yields:
- 11MB/s, but more or less constant.
- Running virtual machines are responsive as ever
- No errors in kern.log

Thanks for the info, Alvin. So I wonder if we could reproduce this without
qemu, simply creating two LVM partitions, one with a snapshot, with both
originals mounted, rsyncing to the one while trying to get something done
(maybe a slower rate-limited rsync) in the other.

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]