Ubuntu
snapd package

Can't purge snapd: rm: cannot remove '/snap': Device or resource busy under LXD

Bug #1903967 reported by Paride Legovini
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
lxd (Ubuntu)
Invalid
Undecided
Unassigned
snapd (Ubuntu)
Fix Released
High
Maciej Borzecki

Bug Description

snapd can't be purged when there are snaps installed, it fails with with error:

  rm: cannot remove '/snap': Device or resource busy

Steps to reproduce on Hirsute (but also affects Focal, Groovy):

  lxc launch ubuntu-daily:hirsute purgefail ; \
    sleep 5 ; \
    lxc exec purgefail -- apt-get -y remove --purge snapd

This doesn't immediate affect Bionic as it has no snaps installed by default, but it does fail if a snap is manually installed. To reproduce on Bionic just run:

   snap install hello-world ; apt-get -y remove --purge snapd

The snapd package should perhaps remove all the snaps on purge? This problem affects tools like autopkgtest-build-lxd and autopkgtest-buildvm-ubuntu-cloud that setup an Ubuntu minimal image to be used to run autopkgtests (via /usr/share/autopkgtest/setup-commands/setup-testbed), but fail to remove snapd. In general a package that fails to remove from its default state has in my opinion a high severity bug.

Full Hirsute reproducer output:

$ lxc launch ubuntu-daily:hirsute purgefail ; sleep 5 ; lxc exec purgefail -- apt-get -y remove --purge snapd
Creating purgefail
Starting purgefail
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 squashfs-tools
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  snapd*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 117 MB disk space will be freed.
(Reading database ... 31976 files and directories currently installed.)
Removing snapd (2.47.1+20.10.1) ...
Stopping snap.lxd.activate.service
Stopping unit snap.lxd.activate.service
Waiting until unit snap.lxd.activate.service is stopped [attempt 1]
snap.lxd.activate.service is stopped.
Stopping snap.lxd.daemon.service
Stopping unit snap.lxd.daemon.service
Waiting until unit snap.lxd.daemon.service is stopped [attempt 1]
snap.lxd.daemon.service is stopped.
Stopping snap.lxd.daemon.unix.socket
Stopping unit snap.lxd.daemon.unix.socket
Waiting until unit snap.lxd.daemon.unix.socket is stopped [attempt 1]
snap.lxd.daemon.unix.socket is stopped.
dpkg: warning: while removing snapd, unable to remove directory '/snap': Device or resource busy - directory may be a mount point?
Processing triggers for man-db (2.9.3-2) ...
Processing triggers for dbus (1.12.20-1ubuntu1) ...
Processing triggers for mime-support (3.64ubuntu1) ...
(Reading database ... 31894 files and directories currently installed.)
Purging configuration files for snapd (2.47.1+20.10.1) ...
Stopping snap.lxd.activate.service
Stopping unit snap.lxd.activate.service
Waiting until unit snap.lxd.activate.service is stopped [attempt 1]
snap.lxd.activate.service is stopped.
Removing snap.lxd.activate.service
Stopping snap.lxd.daemon.service
Stopping unit snap.lxd.daemon.service
Waiting until unit snap.lxd.daemon.service is stopped [attempt 1]
snap.lxd.daemon.service is stopped.
Removing snap.lxd.daemon.service
Stopping snap-core18-1932.mount
Stopping unit snap-core18-1932.mount
Waiting until unit snap-core18-1932.mount is stopped [attempt 1]
snap-core18-1932.mount is stopped.
Removing snap core18 and revision 1932
Removing snap-core18-1932.mount
Stopping snap-lxd-18137.mount
Stopping unit snap-lxd-18137.mount
Waiting until unit snap-lxd-18137.mount is stopped [attempt 1]
snap-lxd-18137.mount is stopped.
Removing snap lxd and revision 18137
Stopping unit snap.lxd.daemon.unix.socket
Waiting until unit snap.lxd.daemon.unix.socket is stopped [attempt 1]
snap.lxd.daemon.unix.socket is stopped.
Removing snap-lxd-18137.mount
Stopping snap-snapd-9721.mount
Stopping unit snap-snapd-9721.mount
Waiting until unit snap-snapd-9721.mount is stopped [attempt 1]
snap-snapd-9721.mount is stopped.
Removing snap snapd and revision 9721
Removing snap-snapd-9721.mount
Final directory cleanup
rm: cannot remove '/snap': Device or resource busy
dpkg: error processing package snapd (--purge):
 installed snapd package post-removal script subprocess returned error exit status 1
dmesg: read kernel buffer failed: Operation not permitted
                                                         Errors were encountered while processing:
 snapd
E: Sub-process /usr/bin/dpkg returned an error code (1)

Paride Legovini (paride)
Changed in snapd (Ubuntu):
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in snapd (Ubuntu):
status: New → Confirmed
Revision history for this message
Maciej Borzecki (maciek-borzecki) wrote :

AFAIK, /snap being busy in an lxd container is expected, as /snap appears to be a bind mount:

root@purgefail:~# findmnt /snap
TARGET SOURCE FSTYPE OPTIONS
/snap /dev/sda1[/var/snap/lxd/common/lxd/storage-pools/default/containers/purgefail/rootfs/snap] ext4 rw,relatime

Perhaps the lxd can comment on whether this can be addressed. In any case it's a separate issue.

As for the problem with Bionic, I cannot reproduce that locally, in GCP or as part the snapd CI. Please confirm that there are no snap applications running (eg. snap-store), and that your shell is not somewhere under /snap. Try running `lsof |grep ' /snap'` to check for any processes.

Changed in snapd (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Stéphane Graber (stgraber) wrote :

This is not a LXD issue.

/snap is a mount because snapd makes it so to avoid issues with mount propagation.
On systems where / isn't MS_SHARED, snapd does the equivalent of:
 - mount -o bind /snap /snap
 - mount --make-rshared /snap

If snapd removal then fails, that's because snapd or the packaging scripts are missing logic to undo that bind-mount when it is present.

Booting a regular system or VM without / being rshared should yield the exact same behavior.

Changed in lxd (Ubuntu):
status: New → Invalid
Revision history for this message
Maciej Borzecki (maciek-borzecki) wrote :

@stgraber thanks for the pointers. I was pretty sure we have the right code in postrm to handle that and we do. However it looks like for the container case, the first attempt to remove /snap is tried before umount is executed. I'll look into addressing this shortly.

Revision history for this message
Maciej Borzecki (maciek-borzecki) wrote :

I've proposed a fix for snapd inside containers right here: https://github.com/snapcore/snapd/pull/9812

Changed in snapd (Ubuntu):
status: Incomplete → In Progress
assignee: nobody → Maciej Borzecki (maciek-borzecki)
Maciej Borzecki (maciek-borzecki)
Changed in snapd (Ubuntu):
status: In Progress → Fix Committed
summary: Can't purge snapd: rm: cannot remove '/snap': Device or resource busy
+ under LXD
Maciej Borzecki (maciek-borzecki)
Changed in snapd (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.