mounting Luks encrypted USB-HDD does not work reliably

I have the same problems. The last time automount of luks encrypted disks worked for me was with edgy.

I have the same problems. First mount after reboot is OK. Subsequent attempts either fail to popup the passphrase dialog or take the passphrase but do not mount the drive.

In the case where the passphrase dialog did not appear, I attempted to mount the volume by hand using

/usr/bin/gnome-mount --hal-udi=/org/freedesktop/Hal/devices/volume_uuid_c52cc2c6_8fbd_41b3_a99f_f4a7d3132c9c

but this command did not pop up the passphrase dialog and exited with status 1.

I then recompiled the gnome-mount package with DEB_BUILD_OPTIONS=noopt,nostrip and installed it. I repeated the same gnome-mount command with the addition of the "-b" flag (to stop the mount command from daemonizing). Under gdb, I traced the problem this far:

3077 if (clear_udi != NULL) {
3078 g_warning (_("Crypto volume '%s' is already setup with clear volume '%s'"),
3079 udi, clear_udi);

(gdb) print clear_udi
$3 = 0x659f10 "/org/freedesktop/Hal/devices/volume_uuid_488bc1b0_3344_41aa_88b7_e6e1a36ee062"

So, in this case it thinks it already has constructed the clear text volume? Perhaps HAL has some stale information in its cache from the previous mount?

I'm away from my Ubuntu system at the moment so cannot give any more information for a week or too.

Instead of rebooting, you may try

sudo /etc/init.d/dbus restart

This restarts dbus and a load of daemons which are dependent on it, including HAL. This had the same effect as a reboot for me.

<humour>
$ /usr/sbin/hald --version
HAL package version: 9000
I'm sorry Dave, I'm afraid I can't do that...
</humour>

I also have the same problem.
Mounting encryped external media don't work correctly for me since Feisty. In Feisty it was horrible. Sometimes it worked correctly, bust most of the time it didn't. See also this report: https://bugs.launchpad.net/ubuntu/+source/hal/+bug/88213
Now I'm glad that things have changed in Gutsy. Until now, every first insert of an encrypted USB-harddisk/stick works fine. In my case that is a huge improvement in comparison with the situation in Feisty. But if I eject the encryped media and insert it again, nothing happens. No promt for the password, nothing you can notice at the desktop.

Peter Wainwright wrote: "So, in this case it thinks it already has constructed the clear text volume? Perhaps HAL has some stale information in its cache from the previous mount?"

I'm the same opinion. If I start gnome-mount manually, it tells me something similar:

xxx@yyy:~$ gnome-mount -vbd /dev/sdb1
gnome-mount 0.6

** (gnome-mount:6526): WARNING **: Crypto volume '/org/freedesktop/Hal/devices/volume_uuid_e6dff7d4_8180_44e4_a245_313a7c629f87' is already setup with clear volume '/org/freedesktop/Hal/devices/volume_uuid_ed993252_5606_4872_98a3_43bea8e20a7a'
xxx@yyy:~$

There are two strange things I have noticed:
1. In 3-5% everything is ok! So sometimes (but realy only sometimes) it isn't a problem to eject and insert again. Until now I don't know the reason for.
2. I think it is strange that the output of gnome-mount (like I wrote above) contains two different uuids. (maybe that is ok, I don't know.)

Stefan,
I think having two UUIDs is OK; as far as I know, the system should regard the encrypted volume and the plaintext volume as separate objects (they correspond to different devices, linked by the device mapper).
My experience with Feisty was better than yours - this stuff worked OK for me until I upgraded to Gutsy.

I can confirm this (won't work the first time etiher, for me) but I think this is a hal rather than a gnome-mount issue?

I have to clarify this, I get asked for password and device mapper device is setup correctly, but not mounted.

I strongly suspect this is a HAL issue, though I cannot confirm it.

This experience makes me suspect HAL:

I recompiled HAL with debugging enabled and optimization off. Then I used gdb to step through lshal after the problem had been triggered.

Running lshal gave "Dumping N devices from the Global device list"; however I saw only N-1 devices in the output. This should be impossible if the database is intact. The only way this can happen is if there is a device in the list whose parent is not in the list. Therefore it seems that the HAL GDL has become corrupt.

Looking more closely using gdb, it seems that the "orphaned" device is the cleartext volume which should have been torn down when the USB stick was removed the first time.

I would suppose that HAL is responsible for removing the child devices when the parent is removed and thus ensuring the integrity of the GDL.

I suspect this is also related to bug #117011...

It is possible that the problem lies with cryptsetup. Even when manually performing the steps

cryptsetup luksOpen /dev/sdb1 luks
cryptsetup luksClose luks

I still end up with some "leaked" devices. These can cause the "already setup" error. For example I have

/org/freedesktop/Hal/devices/volume_part_1_size_64000

[prw@pax cryptsetup-1.0.5]$ lshal -l -u /org/freedesktop/Hal/devices/volume_part_1_size_64000
udi = '/org/freedesktop/Hal/devices/volume_part_1_size_64000'
  block.device = '/dev/mapper/temporary-cryptsetup-24059' (string)
  block.is_volume = true (bool)
  block.major = 254 (0xfe) (int)
  block.minor = 0 (0x0) (int)
  block.storage_device = '/org/freedesktop/Hal/devices/storage_serial_disgo_disgo_021D93010900B1E0_0_0' (string)
  info.capabilities = {'volume', 'block'} (string list)
  info.category = 'volume' (string)
  info.parent = '/org/freedesktop/Hal/devices/storage_serial_disgo_disgo_021D93010900B1E0_0_0' (string)
  info.product = 'Volume' (string)
  info.udi = '/org/freedesktop/Hal/devices/volume_part_1_size_64000' (string)
  linux.hotplug_type = 3 (0x3) (int)
  linux.sysfs_path = '/sys/block/dm-0' (string)
  storage.model = '' (string)
  volume.block_size = 512 (0x200) (int)
  volume.crypto_luks.clear.backing_volume = '/org/freedesktop/Hal/devices/volume_uuid_c52cc2c6_8fbd_41b3_a99f_f4a7d3132c9c' (string)
  volume.fstype = '' (string)
  volume.fsusage = '' (string)
  volume.fsversion = '' (string)
  volume.is_disc = false (bool)
  volume.is_mounted = false (bool)
  volume.is_mounted_read_only = false (bool)
  volume.is_partition = false (bool)
  volume.label = '' (string)
  volume.linux.is_device_mapper = true (bool)
  volume.mount_point = '' (string)
  volume.num_blocks = 125 (0x7d) (int)
  volume.size = 64000 (0xfa00) (uint64)
  volume.uuid = '' (string)

This temporary-cryptsetup-XXXXX device is created by cryptsetup luksOpen, but it was supposed to be freed by clear_mapping().

By the way, my Fedora 7 box does this perfectly. You might want to see what is different with that.

Horrid, Horrid! Handling of removable devices in Linux SUCKS!

It's all just too complicated, with a mess of kernel, udev, hal, gnome-volume-manager, gnome-mount, cryptsetup... Lets have a single monolithic kernel-based routine for the whole lot!

[Sorry, I just had to let off steam there, I have spent so many fruitless hours chasing this bug].

The following shows what happens when udev attempts to remove the device corresponding to /dev/mapper/temporary-cryptsetup-XXXX. I have added some extra info() calls to udevd to see the details. I list the entries in "/dev/mapper" at the beginning of udev_device_event. As you can see, apparently it has been asked to remove a device which does not exist (yet?) Therefore udev_device_event returns with an error code (-1), and as a result the remove event is not passed up to HAL and the stale device stays in the HAL database.

Is it possible there is some kind of race condition here? The file /dev/mapper/temporary-cryptsetup-XXXX is created and then almost immediately destroyed, but I would have thought there should be some mechanism to ensure that the device node actually exists before the udev event is run? Perhaps this is in fact a kernel or a udev bug.

Dec 3 14:57:18 pax udevd[2753]: udev_event_run: seq 2670 forked, pid [6052], 'remove' 'block', 0 seconds old
Dec 3 14:57:18 pax udevd-event[6052]: udev_device_event: XXXX device node remove '/block/dm-0'
Dec 3 14:57:18 pax udevd-event[6052]: udev_device_event: XXXX name=''
Dec 3 14:57:18 pax udevd-event[6052]: udev_device_event: XXXX mapper entry 'control' OK
Dec 3 14:57:18 pax udevd-event[6052]: udev_db_delete_device: XXXX udev_db_delete_device running for '/dev/.udev/db/\x2fblock\x2fdm-0'
Dec 3 14:57:18 pax udevd-event[6052]: name_index: removing index: '/dev/.udev/names/mapper\x2ftemporary-cryptsetup-6028/\x2fblock\x2fdm-0'
Dec 3 14:57:18 pax udevd-event[6052]: name_index: removing index: '/dev/.udev/names/disk\x2fby-id\x2fdm-name-temporary-cryptsetup-6028/\x2fblock\x2fdm-0'
Dec 3 14:57:18 pax udevd-event[6052]: udev_db_delete_device: XXXX udev_db_delete_device finished for '/dev/.udev/db/\x2fblock\x2fdm-0'
Dec 3 14:57:18 pax udevd-event[6052]: udev_node_remove: XXXX remove '/dev/mapper/temporary-cryptsetup-6028'
Dec 3 14:57:18 pax udevd-event[6052]: udev_node_remove: XXXX device node '/dev/mapper/temporary-cryptsetup-6028' not found
Dec 3 14:57:18 pax udevd-event[6052]: udev_device_event: XXXX udev_node_remove returned -1
Dec 3 14:57:18 pax udevd-event[6052]: udev_node_update_symlinks: update symlink 'disk/by-id/dm-name-temporary-cryptsetup-6028' of '/block/dm-0'
Dec 3 14:57:18 pax udevd-event[6052]: udev_db_get_devices_by_name: no index directory '/dev/.udev/names/disk\x2fby-id\x2fdm-name-temporary-cryptsetup-6028': No such file or directory
Dec 3 14:57:18 pax udevd-event[6052]: update_link: found -1 devices with name 'disk/by-id/dm-name-temporary-cryptsetup-6028'
Dec 3 14:57:18 pax udevd-event[6052]: update_link: no reference left, remove 'disk/by-id/dm-name-temporary-cryptsetup-6028'
Dec 3 14:57:18 pax udevd-event[6052]: udev_event_process: XXXX udev_device_event returned -1, ignore=0, run=1
Dec 3 14:57:18 pax udevd-event[6...

I have fixed this problem by REMOVING /etc/udev/rules.d/65-dmsetup.rules. Now there are no stale devices, the password dialog appears every time I insert the stick, and the volume is auto-mounted reliably.

Fedora (7) does not have any analogous file and does not have any problems with LUKS-encrypted USB devices.

cryptsetup (libdevmapper) creates the nodes in /dev/mapper itself. If you let udev do it there is a potential for nasty race conditions (see e.g. https://wiki.ubuntu.com/UdevDeviceMapper).

You may want to check this file (it appears as debian/dmsetup.udev in the source) to see if the symlinks are still needed, but I'm pretty sure that you should not be using NAME="mapper/$env{DM_NAME}" because it conflicts with libdevmapper.

Just to confirm removing /etc/udev/rules.d/65-dmsetup.rules also solves this problem on my system. No visible side effects yet.

Unfortunately I still occasionally get the temporary-cryptsetup mapping left around (as shown by sudo dmsetup ls), though this is much less frequent now. I'm afraid that race conditions are probably endemic due to the design of udev; the way it forks processes into the background it seems like there is no guarantee that create/remove events will be handled in the right order. I wish I knew why this all worked (for me) under Gutsy.

... More coffee needed - I meant Feisty. Gutsy has the regression.

Just wanted to confirm the exact same behavior. That is, it worked 100% in Feisty then abruptly stopped in Gutsy. I just removed /etc/udev/rules.d/65-dmsetup.rules, rebooted and encrypted removable drives works again. Does anyone know if this is a factor of it being an upgrade from Feisty to Gutsy?

Added devmapper to the affects list, since that's the package which contains the problematic file.

Attaching the 65-dmsetup.rules file for reference.

Removing 65-dmsetup.rules does not help here. The LUKS volume is correctly initialized and an entry in /dev/mapper is created but the volume is not mounted.

This is a race condition in hal, it grabs the temporary device before the final name is decided.

The problem is originally caused by a race condition between cryptsetup (actually libdevicemapper) and UDEV. In a UDEV system cryptsetup should not add or remove the device node by itself but this is not so easy to fix I think.

Anyway, it should be no problem if HAL correctly recognizes all additions/removals of nodes under /dev/mapper but this is not the case. If UDEV gets a 'device added' notification and the node is already in place then it leaves the node as it is and continues. On the other hand if UDEV gets a 'device removed' kernel notification and the device node is not present then no 'RUN' rules are executed! One can see this in the debug logs below. In the failed case the 'udev_node_remove' call is missing and, most important, also the 'pass_env_to_socket' calls. Due to this HAL will not know that the device node was removed!

== good notification ==
Jan 10 20:26:30 axel udevd[2679]: udev_event_run: seq 2918 forked, pid [8792], 'remove' 'block', 0 seconds old
Jan 10 20:26:30 axel udevd-event[8792]: name_index: removing index: '/dev/.udev/names/mapper\x2fluks_crypto_10926a57-5d12-4800-9422-e16288e999db/\x2fblock\x2fdm-2'
Jan 10 20:26:30 axel udevd-event[8792]: name_index: removing index: '/dev/.udev/names/disk\x2fby-id\x2fdm-name-luks_crypto_10926a57-5d12-4800-9422-e16288e999db/\x2fblock\x2fdm-2'
Jan 10 20:26:30 axel udevd-event[8792]: name_index: removing index: '/dev/.udev/names/disk\x2fby-uuid\x2fd66d2764-0ebf-4dd0-8b38-19b96d58a1cf/\x2fblock\x2fdm-2'
Jan 10 20:26:30 axel udevd-event[8792]: name_index: removing index: '/dev/.udev/names/disk\x2fby-label\x2fBackup/\x2fblock\x2fdm-2'
Jan 10 20:26:30 axel udevd-event[8792]: udev_node_remove: removing device node '/dev/mapper/luks_crypto_10926a57-5d12-4800-9422-e16288e999db'
Jan 10 20:26:30 axel udevd-event[8792]: udev_node_update_symlinks: update symlink 'disk/by-id/dm-name-luks_crypto_10926a57-5d12-4800-9422-e16288e999db' of '/block/dm-2'
Jan 10 20:26:30 axel udevd-event[8792]: udev_db_get_devices_by_name: no index directory '/dev/.udev/names/disk\x2fby-id\x2fdm-name-luks_crypto_10926a57-5d12-4800-9422-e16288e999db': No such file or directory
Jan 10 20:26:30 axel udevd-event[8792]: update_link: found -1 devices with name 'disk/by-id/dm-name-luks_crypto_10926a57-5d12-4800-9422-e16288e999db'
Jan 10 20:26:30 axel udevd-event[8792]: update_link: no reference left, remove 'disk/by-id/dm-name-luks_crypto_10926a57-5d12-4800-9422-e16288e999db'
Jan 10 20:26:30 axel udevd-event[8792]: udev_node_update_symlinks: update symlink 'disk/by-uuid/d66d2764-0ebf-4dd0-8b38-19b96d58a1cf'of '/block/dm-2'
Jan 10 20:26:30 axel udevd-event[8792]: udev_db_get_devices_by_name: no index directory '/dev/.udev/names/disk\x2fby-uuid\x2fd66d2764-0ebf-4dd0-8b38-19b96d58a1cf': No such file or directory
Jan 10 20:26:30 axel udevd-event[8792]: update_link: found -1 devices with name 'disk/by-uuid/d66d2764-0ebf-4dd0-8b38-19b96d58a1cf'
Jan 10 20:26:30 axel udevd-event[8792]: update_link: no reference left, remove 'disk/by-uuid/d66d2764-0ebf-4dd0-8b38-19b96d58a1cf'
Jan 10 20:26:30 axel udevd-event[8792]: udev_node_update_symlinks: update symlink 'disk/by-label/Backup' of '/block/dm-2'
Jan 10 20:26:30 axel udevd-event[87...

There is another problem with Gnome-mount.

As we know, LUKS encrypted partitions can be mounted with password (passphrase) or with key file. When I connect LUKS formatted device, Gnome-mount only asks me for a passphrase. There is no option for mounting device with keyfile.

Usually majority of devices is protected with passphrases (or with keyfile AND passphrase), but it could be protected with keyfile only.

It would be nice to add a possibility of mounting LUKS partition with keyfile from Gnome mounter.

Thanks Jan, your patch completely fixes the problem for me with no observed side effects (I didn't look too thorough, though). Oh, and I'm a Gentoo user, so this fix should probably be propagated upstream.

WARNING!!!!

A word of warning to those who have removed the file 65-dmsetup.rules. I had found that this temporary fix worked quite well on all my machines, but got burned when an automatic update of cryptsetup got installed without me thoroughly considering the possible ramifications. The net result was that my machines were rendered unbootable. All I got at startup was a message that my cryptroot could not be found and then routed to a busy box.

I'm not entirely sure what happened, but think that 65-dmsetup.rules needs to be there if the update is going to update the initframfs. Without the file the update of initramfs doesn't complete. Since there is no warning of this from update manager, you're stuck with an unbootable system on restart. This was my first major panic with Linux and I spent a lot of time googling and futzing with a LiveCD before I figured out the problem and repaired the mess.

Bottom Line: Avoid this situation by putting 65-dmsetup.rules back where it belongs before you run any automatic updates related to the cryptsetup.

Hope this saves someone some time!

Seems something like my proposed patch was already in the upstream udev git development tree...

http://git.kernel.org/?p=linux/hotplug/udev.git;a=commitdiff;h=a0092d28dbb2c1c75c2fac17303b703343f03a35

See also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=457616 where the original upstream patch came from. It looks like as it doesn't only affect LUKS mounting in particular...

Thanks a million, Jan!

This bug was fixed in the package udev - 117-4

---------------
udev (117-4) hardy; urgency=low

  * Add debian/patches/00upstream-RUN-for-remove.patch:
    - Execute RUN rules for device removals even if the device is not present
      any more (like for USB devices).
    - This restores the behaviour of earlier versions and fixes race
      conditions with hal for e. g. cryptsetup, and device naming.
    - Patch taken from upstream git (see patch header).
    - LP: #148003

 -- Martin Pitt <email address hidden> Thu, 07 Feb 2008 09:45:24 +0100

Attached is the debdiff of the straightforward backport to gutsy that seems to work for me.

Michael Hofmann wrote:
> Attached is the debdiff of the straightforward backport to gutsy that
> seems to work for me.
>
> ** Attachment added: "Debdiff of the backport to gutsy"
> http://launchpadlibrarian.net/11889991/udev-cryptsetup-backport-hardy.diff
>

Thanks a lot. How should I use this ? Do I need to use 'patch' on this file,
or is the backport going to turn into an update soon ?

Cyril

As long as Martin is not uploading a backport to gutsy-proposed, you can
get the necessary packages from my PPA with the following lines in your
sources.list:
 deb http://ppa.launchpad.net/mh21/ubuntu gutsy main
 deb-src http://ppa.launchpad.net/mh21/ubuntu gutsy main
You only need the volumeid and udev packages.

Michael

Sorry to insist on a matter that may be simple to you, but:
- I installed packages udev and volumeid 113-0ubuntu16mh1 from these repositories (checked with dpkg -l)
- I rebooted

But the behavior is still the same: when I insert my luks usb key, I get a label in /dev/mapper, but nothing in /media.
Did I miss anything ?

Cyril

I didn't have any problems anymore after I installed the new packages,
but I will test it further.

Michael Hofmann wrote:
> I didn't have any problems anymore after I installed the new packages,
> but I will test it further.
>
>
I installed the update with no problem here. USB drive now works.
Haven't had time to check firewire. Thanks to all who contributed to
this fix!

--
John Ross, Ph.D., P.E.
John Ross & Associates, LLC Voice: 801-359-5957
350 West 800 North, Suite 317 e-mail: <email address hidden>
Salt Lake City, UT 84103 web: www.johnross.com

I sponsored Michael's debdiff (thanks!) and accepted it into gutsy-proposed. Please test the updated packages and give feedback here. Thank you!

I can't find any updated packages even though I have

deb http://de.archive.ubuntu.com/ubuntu gutsy-proposed main restricted universe multiverse

in my sources.list. Do I need to do something else?

It seems to have fixed the problem for me (I tested with the version from http://ppa.launchpad.net/mh21/ubuntu though)

I'm not seeing new volumeid or udev packages in gutsy-proposed either.

Seems that the German mirror (de.archive) is out of date (last update from Feb 19). It's definitively on http://archive.ubuntu.com. It's also on other mirrors (like nl.archive). Sorry, we can't do anything about the German mirror ATM, that has to be fixed by its local admins.

With the new package I had success for a couple of days, but this morning my drive did not mount, I had to manually remove (dmsetup remove) a temp_cryptsetup_xxxxx mapping and then reattach the drive. I still had to run gnome-mount manually to get the device to mount after that (gnome-mount -d /dev/sdxx).

It may have actually been a bad unmount as I noticed it came up as /dev/sdd(2) rather than /dev/sdc2 which was available by the time I looked. After the dmsetup remove and the replug it was at sdc.

Quick question: should I install updated package by myself or is it pushed through updates? Because I didn't see any update and still have problems.

However, I am solving the problem by entering the following command in console:

sudo /etc/init.d/hal restart

I installed voluemid, libvolume-id0, and udev from gutsy-proposed. My
initial testing shows it's working well for me.

For others getting this to work, add the following lines to your
/etc/apt/sources.list:
deb http://archive.ubuntu.com/ubuntu gutsy-proposed main restricted universe
multiverse
deb-src http://archive.ubuntu.com/ubuntu gutsy-proposed main restricted
universe multiverse

Then run:
sudo apt-get update
sudo apt-get install volumeid libvolume-id0 udev

In my case, I removed the extra lines from sources.list and reran "sudo
apt-get update" so that update-manager would stop asking me to install other
packages from proposed.

The new packages solved this problem for me.

Although the volume can now be properly mounted as I said above, it seems that the patch introduces a new bug: Upon unmounting I now occasionally get the warning about "Unsafe Device Removal" - even though I didn't touch the device. This sounds very much like another race condition to me.

Nikolaus, thank you for testing! This is a long-standing bug in gnome-volume-manager and not a regression introduced by this udev update.

Successfully tested by at least three people (Nikolaus, Michael, Robert, and also myself). Considering verified.

Copied to gutsy-updates.

It seems this bug is still not fixed yet.

Today I connected external LUKS encrypted USB drive and got password prompt.

Then I unmounted device, diconnected and connected again.

Nothing happened. However, when I sais sudo /etc/init.d/hal restart, I got password prompt and then drive mounted normaly.

The same applies for me:

I installed a new gusty from scratch, updated to get

libvolume-id0-113-0ubuntu17
volumeid-113-0ubuntu17
udev-113-0ubuntu17

The behavior is still the same (I rebooted since then, of course):

when I insert my luks-encrypted usb key, the passwd dialog pops up, I enter my passwd, then I only get a device labeled luks-<some stuff> in /dev/mapper, but nothing is mounted in /media/

I really don't think the fix actually work (at least for me!).

Cyril

I just wanted to add that I do get a passwd promt every time. The device only never gets mounted in /media although it actually gets mapped in /dev/mapper.

csoler, since you probably have a different problem then, can you please open a new bug against gnome-mount and do the steps on http://wiki.ubuntu.com/DebuggingRemovableDevices?

Do you have the chance to try this on a current Hardy live CD?

It worked for me in Hardy - until recently. Now I got the same problem as csoler.

Created https://bugs.launchpad.net/ubuntu/+source/gnome-mount/+bug/217749 about it.