2.41 fails to mount root partition

laney@raleigh> sudo fdisk -l ~
[sudo] password for laney:

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002a298

   Device Boot Start End Blocks Id System
/dev/sda1 * 63 1953520064 976760001 83 Linux

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002a298

   Device Boot Start End Blocks Id System
/dev/sdb1 * 63 1953520064 976760001 83 Linux

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa9f073f2

   Device Boot Start End Blocks Id System
/dev/sdc1 * 2048 486172671 243085312 83 Linux
/dev/sdc2 957030400 976771071 9870336 82 Linux swap / Solaris
/dev/sdc3 486174718 957030399 235427841 5 Extended
/dev/sdc5 486174720 957030399 235427840 83 Linux

Partition table entries are not in disk order

Disk /dev/md127: 1000.2 GB, 1000204795904 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953524992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002a298

      Device Boot Start End Blocks Id System
/dev/md127p1 * 63 1953520064 976760001 83 Linux

laney@raleigh> cat /etc/fstab ~
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sdc5 during installation
UUID=d83646e3-8a49-4a5c-9db2-5f8239ca1473 / ext4 errors=remount-ro 0 1
# swap was on /dev/sdc2 during installation
UUID=6e307e8d-4088-49a1-a63a-65d754f36d38 none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
UUID=a9c7370f-f9eb-4d31-bc7b-7fb3def0a4f8 /srv ext4 defaults 0 2
UUID=c69e3939-afc5-4d96-9378-8993ac8a6e6b /media/debian ext4 defaults 0 2
overlay /var/lib/schroot/union/overlay tmpfs defaults
cgroup /cgroup cgroup defaults 0 0

Please attach the output of mountall --verbose. You will probably need to edit /etc/init/mountall.conf and redirect the output to somewhere on /run.

Sure, here it is:

For fun, here's one from this successful boot after I downgraded to 2.40 again.

More investigation (paste from #ubuntu-devel)

02/10 17:58:05 <Laney> slangasek: so commenting out those entries [/var/lib/schroot/.. and /cgroup] indeed made it boot with 2.41
02/10 17:59:04 <Laney> turns out /var/lib/schroot is a symlink to /media/debian/var/lib/schroot/
02/10 18:01:49 <Laney> and it works if I change the mount point to the real path rather than going through the symlink

This bug was fixed in the package mountall - 2.43

---------------
mountall (2.43) unstable; urgency=low

  * Fix a regression introduced in 2.41: parse_mountinfo() doesn't
    generate a mounted event for a filesystem whose canonicalized
    mountpoint doesn't match the mountpoint specified in /etc/fstab. So
    we still need to call mounted() explicitly for this case. LP: #1059471.

 -- Steve Langasek <email address hidden> Tue, 06 Nov 2012 01:03:12 -0800

Hello Iain, or anyone else affected,

Accepted mountall into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mountall/2.42ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hey,

On Thu, Nov 15, 2012 at 12:57:47AM -0000, Clint Byrum wrote:
> Hello Iain, or anyone else affected,
>
> Accepted mountall into quantal-proposed. The package will build now and
> be available at
> http://launchpad.net/ubuntu/+source/mountall/2.42ubuntu0.1 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Your feedback will aid us getting this update
> out to other Ubuntu users.
>
> If this package fixes the bug for you, please change the bug tag from
> verification-needed to verification-done. If it does not, change the
> tag to verification-failed. In either case, details of your testing
> will help us make a better decision.

I don't have the environment in which I triggered this bug any more, so
I can't confirm whether the original case is fixed.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

Marking as verified per the feedback in duplicate bug #1081559.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package mountall - 2.42ubuntu0.1

---------------
mountall (2.42ubuntu0.1) quantal-proposed; urgency=low

  * Fix a regression introduced in 2.41: parse_mountinfo() doesn't
    generate a mounted event for a filesystem whose canonicalized
    mountpoint doesn't match the mountpoint specified in /etc/fstab. So
    we still need to call mounted() explicitly for this case. LP: #1059471.
 -- Steve Langasek <email address hidden> Wed, 07 Nov 2012 20:58:39 -0800

it would seem that mountall is likely the difference between pretty green dots at [1] (ubuntu-quantal-daily-i386-server-20121125) and the evil red dots at [2] (ubuntu-quantal-daily-i386-server-20121128) . The difference in manifests of those 2 builds are http://paste.ubuntu.com/1408166/ .

The likely suspect there is mountall. Note, this is a race condition, and seems to now be the same issue that we were seeing on raring for quite some time as bug 1078926.

I can provide access to reproduce system or console --verbose output. One possibly related issue is that it seems that loss of output to /dev/console is now increased in likelyhood.

--
[1] https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/153/
[2] https://jenkins.qa.ubuntu.com/view/ec2%20AMI%20Testing/view/Overview/job/quantal-server-ec2-daily/154/