LUKS is extremely slow on amd64 builds but not on i386

TL;DR: Disk I/O with LUKS is extremely slow on amd64 builds, however, it is much faster on i386 builds. This has been verified by testing on multiple hardware platforms and with multiple versions of Ubuntu. The possible cause of the slowness has been shown to somehow be related to the architecture. --- seems to actually be 10.10, since it's slow in i386, too, just not quite -as- slow; see comment #5.

Longer version:
My friend and I have the same Lenovo T61p laptops. We have the same hardware specs:
 - 2.5 GHz Core 2 Duo
 - 4 GB RAM
 - 80 GB Intel SSD

We even use the same OS, file systems, etc.:
 - Kubuntu
 - ext4
 - Whole disk encryption with LUKS
 - Disk set up with LVM

We have one difference though: for a long time, I have been using the 32 bit version of Kubuntu, my friend has always been using the 64 bit version.

Since we bought our laptops and still used hard discs instead of SSDs, my friends laptop has always been noticable slower with I/O.
In the beginning we just thought that he had gotten a bad disk. We both soon bought a new 80 GB Intel SSD in order to obtain better I/O performance.
When we installed on the SSD we both saw a huge increase in I/O performance. However, my friends system was still much slower than mine.
We thought that it might be due to alignment issues, so my friend researched this subject a lot and made sure that it was perfectly aligned.
Still, his performance with much poorer than mine.

We have really tried debugging this in countless ways. I will not even try to describe all of it here.

We both work on some of the same software projects. Thus, we have the same SVN checkouts on our file systems.
We took one of these checkouts (one with lots of files in it, about 13k files) and used it as reference.
We then measured how long it took to do a find |wc -l inside the folder. Here are the results:
On my friends system it took about 90 seconds. On my system it took less than 15 seconds.
We have repeated this test on several different Ubuntu installations from circa 8.04 up until the most recent 10.04 and we always got the same results.

Then a few months ago I upgraded to Ubuntu 10.10 and my friend did too. Now, suddenly, I began experiencing the same slowness.
Now, my computer also takes about 90 seconds to do the find in the same folder.
This was strange. We feared that maybe our SSDs were failing. We have heard stories about SSDs becoming slow before failing.

During a discussion about possible causes for this behavior it struck me: the only difference between our systems was the architechture. My friend always use amd64, and I use i386.
Recently, when I installed Kubuntu 10.10, it chose the 64 bit version, because I no longer had any legacy apps requiring 32 bit.
It was after this install that my system suddenly began performing badly.

So we have results like this:
Time C: Time taken by the find command on my friend's system
Time T: Time taken by the find command on my system
Arch C: My friend's architecture
Arch T: My architecture

Ubuntu version | Time C | Time T | Arch C | Arch T
---------------+----------+--------+---------+---------
8.04 | 90 s | <15 s | amd64 | i386
8.10 | 90 s | <15 s | amd64 | i386
9.04 | 90 s | <15 s | amd64 | i386
9.10 | 90 s | <15 s | amd64 | i386
10.04 | 90 s | <15 s | amd64 | i386
10.10 | 90 s | 90 s | amd64 | amd64

(I hope this gets formatted correctly. If not, there is a correctly formatted version here: http://paste.adora.dk/P1977.html)

Of course, we made sure to drop cache before testing with the command:
sync ; echo 3 > /proc/sys/vm/drop_caches

So the full command we ran was:
sync ; echo 3 > /proc/sys/vm/drop_caches ; time find $PATH_TO_WC |wc -l

We also did each test multiple times to make sure we got consistent results.

(I am trying to provide as much data as possible so as to make it easier to find and fix the bug)

In order to try to disprove our hypothesis about the architecture making the difference, I tried booting up on a 32 bit Knoppix Live CD on my laptop. I then mounted my LUKS encrypted rootfs from the Knoppix CD and did a find inside the same folder. Now it took about between 10 and 15 seconds to do the find.

OK, so we have found some correlation between the architechture of Ubuntu and the slow I/O.
There were still several possible places that could be responsible for the problem:
 * LVM
 * LUKS
 * Ext4

I then wanted to see which one of these that made a difference.
My /boot partition is not encrypted with LUKS, and I had enough free space available to copy the reference folder to /boot and then did the test there.
On the /boot partition, it took less than 2 seconds. I tried this both in the installed Kubuntu 10.10 (amd64) and from the 32 bit Knoppix live cd. Both systems took less than 2 seconds.

Here is the data from the Knoppix tests from encrypted / and non-encrypted /boot:
http://p.adora.dk/P1978.txt

It is clear that non-encrypted /boot is much quicker than encrypted /. However, all of the tests from the encrypted / partition takes less than 15 sec. This same test takes about 90 sec in amd64 version of Ubuntu.

The results from 64 bit Ubuntu 10.10 are available here:
http://paste.adora.dk/P1979.txt

/boot is ext4, so we can eliminate ext4 as the cause of the problem.

In order to test if LVM is the cause, I created a 100mb file on /boot. I made this file into a loop back device and set up an LVM volume on it.
I then created a file system inside it and mounted. Then I copied the reference folder to this file system and did the tests again.
All tests from this LVM volume took less than 2 seconds. So LVM is eliminated as the cause.

I see the combination of LUKS/amd64 as the only possible cause left.

I investigated further to check if it was a problem related to LUKS only or if it was also present on ecryptfs. For this test, I copied the reference folder to my ecryptfs encrypted homedir on my netbook (Lenovo S10 with Atom CPU, 2 GB RAM, and a 160 GB hard drive -- no SSD). The tests on the netbook took about 6-7 seconds.
This was actually pretty fast for a hard drive. So fast that I guess that it is safe to conclude that the problem does not occur on ecryptfs.

I have now layed out all the data and methology of the investigation. I really hope someone can take the next step and isolate the problem so it can be fixed.
As it is, I/O performance on 64 bit Ubuntu with LUKS is almost unusably slow.

My guess is that either the 64 bit or the 32 bit kernel has some kind of compile flag or other setting that makes the difference.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: linux-image-2.6.35-25-generic 2.6.35-25.44
Regression: No
Reproducible: Yes
ProcVersionSignature: Ubuntu 2.6.35-25.44-generic 2.6.35.10
Uname: Linux 2.6.35-25-generic x86_64
NonfreeKernelModules: nvidia
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
   Subdevices: 2/2
   Subdevice #0: subdevice #0
   Subdevice #1: subdevice #1
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: tdn 2187 F.... pulseaudio
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfe220000 irq 49'
   Mixer name : 'Analog Devices AD1984'
   Components : 'HDA:11d41984,17aa20bb,00100400'
   Controls : 31
   Simple ctrls : 19
Card29.Amixer.info:
 Card hw:29 'ThinkPadEC'/'ThinkPad Console Audio Control at EC reg 0x30, fw 7KHT24WW-1.08'
   Mixer name : 'ThinkPad EC 7KHT24WW-1.08'
   Components : ''
   Controls : 1
   Simple ctrls : 1
Card29.Amixer.values:
 Simple mixer control 'Console',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Date: Tue Mar 8 14:10:10 2011
HibernationDevice: RESUME=UUID=bd03b060-e813-403f-b6c0-dbd58049fbfa
InstallationMedia: Kubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
MachineType: LENOVO 6460D8G
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.35-25-generic root=/dev/mapper/hostname-root ro quiet splash
ProcEnviron:
 LANGUAGE=
 PATH=(custom, user)
 LANG=en_DK.UTF-8
 SHELL=/bin/zsh
RelatedPackageVersions: linux-firmware 1.38.4
SourcePackage: linux
WifiSyslog:

dmi.bios.date: 11/14/2008
dmi.bios.vendor: LENOVO
dmi.bios.version: 7LETC5WW (2.25 )
dmi.board.name: 6460D8G
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr7LETC5WW(2.25):bd11/14/2008:svnLENOVO:pn6460D8G:pvrThinkPadT61p:rvnLENOVO:rn6460D8G:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 6460D8G
dmi.product.version: ThinkPad T61p
dmi.sys.vendor: LENOVO