[regression] dpkg's fsync causes massive regression in Ubuntu Server and Alternate installation times

This would need to be workarounded in dpkg, to make it support a without-fdisk mode that would be triggered on new installs only.

I talked to the release team about it, their decision is to releasenote this issue and work to get it fixed for 10.04.1.

Performance hit also affects alternate/netboot installs, as well as any upgrades (including Desktop)

Thierry-

Regarding upgrades, I think I agree with the current design. Data integrity on a running system is of paramount importance.

New installs, though, are a different beast, in my opinion, since a failure would necessitate a reinstall anyway.

This appears to be a duplicate of bug #537241.

Phillip-

Thanks for the link.

The cause, is in fact, the same (fsync's in dpkg).

The solution, though, might be subtlety different. As I've said above, I can understand and agree with the change in behavior for updates/upgrades on a running system.

This current bug, though, is about what's going on in the installer on fresh installs, when you have no critical data yet on the system. I'm hoping this bug will be fixed separately from bug #537241.

I've release-noted this as follows (feel free to tweak from here):

== Default file system; package manager performance ==

The default file system for installations of Ubuntu 10.04 LTS is `ext4`, the latest version in the popular series of Linux extended file systems. `ext4` includes a number of performance tuning changes relative to previous versions such as `ext3`, the file system used by default up to Ubuntu 9.04. These generally produce improvements, but some particular workloads are known to be significantly slower when using `ext4` than when using `ext3`. If you have performance-sensitive applications, we recommend that you run benchmarks using multiple file systems in your environment and select the most appropriate.

In particular, the `dpkg` package manager is known to run significantly slower on `ext4` (causing installations using the server or alternate install CD to take on the order of twice as long as before). `ext4` does not guarantee atomic renames of new files over existing files in the event of a power failure shortly after the rename, and so `dpkg` needs to force the contents of the new file out to disk before renaming it in order to avoid leaving corrupt zero-length files after power failures. This operation involves waiting for the disk significantly more than it strictly needs to, and so degrades performance. If fast package management operations are most important to you, then you should use `ext3` instead. (Bug:570805)

The simplest way to select a different file system such as `ext3` at installation time is to add the `partman/default_filesystem=ext3` boot parameter when starting the installer. If you are deploying Ubuntu automatically using Kickstart or preseeding, then you can set a different file system in the partitioning recipe instead.

Looks great. Very informative, Colin.

I second that, i love to see some background info!

But it looks like either the release notes or the ext4 documenation needs updating:

===
Release notes
===
`ext4` does not guarantee atomic renames of
new files over existing files in the event of a
power failure shortly after the rename

===
ext4 docs ( http://www.mjmwired.net/kernel/Documentation/filesystems/ext4.txt#338 )
===
the data blocks of the new file are forced
to disk before the rename() operation is
committed. This provides roughly the same level
of guarantees as ext3, and avoids the
"zero-length" problem that can happen when a
system crashes before the delayed allocation
blocks are forced to disk.

The explanation in the release notes is confusing. First of all, as Jakob wrote above, ext4 starting with 2.6.30 provides similar behavior to ext3 with data=ordered. So, it is not clear why fsync() is necessary on ext4, but not on ext3.

More importantly, the default mode for ext3 was changed to data=writeback in 2.6.30, which does not provide the same guarantee as "ordered". Even if the default was changed to be "ordered" in Ubuntu kernel, users still may use an upstream kernel or change the mode to "writeback" on ext3, because the "ordered" mode has horrible latency (20 times or more than "writeback"). Does it mean that anyone running ext3 with data=writeback can face corruption of their package repository?

Jean-Baptists Lallement tested this behaviour and found that ext4's guarantee seems to be ... not so much of a guarantee in reality. I asked him to file it on bugzilla.kernel.org.

Sorry, "Jean-Baptiste Lallement"

Am 05/05/10 15:53, schrieb Jean-Baptiste Lallement:
> ** Bug watch added: Linux Kernel Bug Tracker #15910
> http://bugzilla.kernel.org/show_bug.cgi?id=15910
>
> ** Also affects: linux via
> http://bugzilla.kernel.org/show_bug.cgi?id=15910
> Importance: Unknown
> Status: Unknown

I wonder if this says what you meant to say ( from the kernel.org bug ):
   ---
   auto_da_alloc doesn't detect the replace-via-rename (at least in the
   case of dpkg.)
   ---
Is dpkg really ***replacing***-via-rename when installing new packages?
Where do the old files that are overwritten come from?

Thanks!

Note that for the install case where corruption does not matter, a
export LD_PRELOAD=./libeatmydata.so
should bring performance back to normal (or even above) in a very simple way. See http://www.flamingspork.com/projects/libeatmydata/

I would appreciate it if people affected by this bug could test the dpkg package from:

  https://launchpad.net/~cjwatson/+archive/ppa

This is probably easier to test if you're seeing problems with dpkg performance after installation, although I do plan to hack it into an installation environment after I get back from UDS and do some comparisons. Please report before-and-after timings if you can, preferably with a cold cache.

If this makes a significant improvement, I'll get it into Ubuntu 10.04.1.

Colin-

Poke me when you have an install media ready to test. That's my
primary use case, and I'll gladly help test that.

I ran some tests on my Dell Latitude D430, which has ext4 and a ridiculously slo
w hard disk. I tested with:

$ echo 3 | sudo tee /proc/sys/vm/drop_caches; sudo dpkg -i /var/cache/apt/archives/openoffice.org-common_1%3a3.2.0-7ubuntu4_all.deb | ts

old dpkg:
openoffice.org-common: 56s unpack, 25s config/triggers
ubuntu-docs: 25s unpack, 4s config

Colin's PPA dpkg:
openoffice.org-common: 23s unpack, 23s config/triggers
ubuntu-docs: 23s unpack, 4s config

So it makes quite a difference for the unpacking of openoffice.org-common (2860 files, huge), and not so much for ubuntu-docs (7454 files, smaller)

Test on ext3 (Karmic upgraded to Lucid), 5400rpm notebook hd. Note that i think in addition to dropping the caches, we have to sync before running the test to get somewhat stable results.

Commands used (as root):
  sync; echo 3 > /proc/sys/vm/drop_caches; time dpkg -i /var/cache/apt/archives/openoffice.org-common_1%3a3.2.0-7ubuntu4_all.deb | ts
  sync; echo 3 > /proc/sys/vm/drop_caches; time dpkg -i /var/cache/apt/archives/ubuntu-docs_10.04.3_all.deb | ts

Results, old dpkg (1.15.5.6ubuntu4):
  openoffice.org-common: 26s unpack, 93s real
  ubuntu-docs: 24s unpack, 51s real

Results, Colin's PPA dpkg (1.15.5.6ubuntu5~ppa1):
  openoffice.org-common: 31s unpack, 94s real
  ubuntu-docs: 23s unpack, 54s real

So ext3 doesn't care at all - it's not regressing at least.

I wonder if my problem is related to this bug - I am observing excessive harddisk activity in the unpack phase of package installations on all three Lucid systems:
https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/580537

dpkg from Colin Watsons PPA solved the problem: I installed "Blender" and the unpacking phase of the 10Mb package took less than a second. And I experience no more rattling disks when installing packages.

What benefit is provided by all the fsync action?

Without it, a power-failed install may have zero-length or wrong-content files.

With it, a power-failed install still has a broken package -- each individual file may be fully there or fully not-there, but there will be missing files.

It doesn't help achieve a successful package install. In fact, it _increases_ vulnerability to power failures by making the vulnerable time window more than twice as long (all of the added time is vulnerable time, while some of the original time must be safe prep time).

Either way, system powers back on with a broken package. Either way, the user or the dpkg system must deal with it.

dpkg _should_ do a regular sync() after each package; I imagine (without checking source) that it already does. It already has notes on which packages were in transition. Make sure _those_ are fully sync'd, fsync'd if that's the right way to do it -- those tell dpkg where to pick up, which package to fix, after the power cycle.

I think this code should be retracted, even for normal post-install package installs. Cleanup is going to be needed after a mid-install power failure either way; don't make users suffer through slow, noisy, HD-punishing package installs for no [or negative] benefit.

Ok, looked at Colin's backport:

It's the right way to do it; and I suspect this is true for all Unixish OSes with all filesystems, whether or not they have synchronous sync().

Using the testing method from #17 for the package oxygen-icon-theme (5890 mostly small files), tested on a 5400 RPM laptop harddisk formatted with ext4.

  dpkg 1.15.5.6ubuntu4:
    unpack: 37s
    real: 53s

  dpkg 1.15.5.6ubuntu5~ppa1:
    unpack: 5s
    real: 22s

Also tested on a RAID 0 array consisting of two 7200 RPM desktop drives, formatted with XFS:

  dpkg 1.15.5.6ubuntu4:
    unpack: 56s
    real: 53s

  dpkg 1.15.5.6ubuntu5~ppa1:
    unpack: 13s
    real: 21s

So quite a noticable difference in both cases :-)

Guillem's patch is in Maverick now (1.15.7.2 and newer). I've uploaded 1.15.5.6ubuntu4.1 to lucid-proposed, awaiting approval.

Accepted dpkg into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

In Xubuntu 10.04 LTS the software update tool complains that the proposed dpkg package could not be authenticated. Is this expected at this stage? Thank you.

Regarding the authentication issue. I switched to the man update server from the mirror I usually use and now there are no package authentication problems. So far version 1.15.5.6ubuntu2 works fine.

Correction: I meant *main* update server.

10.04.1 candidate images are not yet available.

I did, however, measure the performance difference on a running system.

 1) Install Ubuntu 10.04 Server
 2) sudo apt-get update
 3) sudo apt-get dist-upgrade --download-only
     (32 updates available now)
 4) time sudo apt-get dist-upgrade
      takes 3m33s

Then:
 1) Install Ubuntu 10.04 Server
 2) Upgrade dpkg to -proposed
 3) sudo apt-get update
 4) sudo apt-get dist-upgrade --download-only
     (32 updates available now)
 5) time sudo apt-get dist-upgrade
      takes 1m52s

There's clearly a huge performance improvement. I would like to see this promoted from -proposed to -updates, and would very much like to test this on a real ISO whenever that's available.

I also have used the proposed package extensively in the past weeks, and did not notice any problems. Together with Dustin's testing I consider this verified.

This bug was fixed in the package dpkg - 1.15.5.6ubuntu4.1

---------------
dpkg (1.15.5.6ubuntu4.1) lucid-proposed; urgency=low

  * Backport proposed patch from Guillem Jover:
    - On Linux, call sync() (which is synchronous) before rename() rather
      than calling fsync() once per file (LP: #570805).
 -- Colin Watson <email address hidden> Mon, 28 Jun 2010 14:32:02 +0100

when use ext4, unpack package is still unbelievable slow in netboot installer,
I use netboot.tar.gz from here:
http://archive.ubuntu.com/ubuntu/dists/lucid-proposed/main/installer-amd64/current/images/netboot/

using maverick and btrfs, dpkg is as slow as a dead horse and sometimes completely freezes, manually running the sync command seems to pull it out of its dead lock...