Boot failure after upgrading kernel to 2.6.32-25-generic

Why exactly was it marked 'incomplete' without any additional comments? What information is missing?

I can confirm this happens on VMware ESX 4.1. I know for certain it does not happen on older vmware-server 1.0.X series. And when it dumps me into the initramfs console, I can run udevadm and about 50% of the time it will segfault.

I have gone back through the process and can confirm that on ESX 4.1 the 2.6.32-25-generic kernel segfaults during boot, the -pae kernel does not segfault. The udevadm command segfaults:

udevadm[94]: segfault at 1 ip 00000001 sp bfb55514 error 4 in libdl.so.2

After the segfault, the user is placed in the initramfs console. Running udevadm a number of times on this console it segfaults about 50% of the time.

This does not happen on ESX 3.5i or VMware Workstation 7.

Confirming that upgrading from 2.6.32-24 to 2.6.32-25 breaks the system. Segmentation fault occurs while booting.

This problem persists in the recent 2.6.32-26-generic kernel.

Same problem here (after upgrade from ESX 3.5i -> ESXi 4.1i) and 32 bits Virtual Machine.
Ubuntu 10.04.1 LTS & Kernel 2.6.32-25-generic-pae won't boot (udev segault): same errors as below.
Upgrade to 2.6.32-26-generic: same problem.
Older installed kernel (2.6.24-23-server) won't boot too, udev got an error ("libudev: udev_monitor_new_from_netlink: error getting socket: Invalid argument) and segault (wait-for-root[879]: segfault at 00000030 eip b7fc0f2b esp bfee2a70 error 4).
Other 64bits VM's works fine (Ubuntu10.04.1).

Tried with 2.6.32-24-generic: boot withou problems !

Hello,

we had the same problem in our company and I found a solution for this.
In ESXi 4.1 vSphere Client choose the VM Ware-Machine which starts with the "segmentation fault"
and go to the machine settings.
There you have to change the "host operating system" from "Ubuntu 32 bit" to "Another"
"Version 32 Bit".

After that run your Machine and start with the dist-upgrade.
Following the finished update reboot your system and it should work now.

Hello,

we had the same problem in our company and I found a workaround for this.
In ESXi 4.1 vSphere Client choose the VM Ware-Machine which starts with the "segmentation fault"
and go to the machine settings.
There you have to change the "host operating system" from "Ubuntu 32 bit" to "Another"
"Version 32 Bit".
VMI must also be deactivated.

After that run your Machine and start with the dist-upgrade.
Following the finished update reboot your system and it should work now.

I bisected the problem down to this change:

commit 4f797709f4c0f61368c2f0f989cefc805fed0c9e
Author: Brad Figg <email address hidden>
Date: Fri Aug 27 09:10:55 2010 -0700

    mm: make the vma list be doubly linked

    BugLink: http://bugs.launchpad.net/bugs/625392

    commit 297c5eee372478fc32fec5fe8eed711eedb13f3d upstream.

    It's a really simple list, and several of the users want to go backwards
    in it to find the previous vma. So rather than have to look up the
    previous entry with 'find_vma_prev()' or something similar, just make it
    doubly linked instead.

    Tested-by: Ian Campbell <email address hidden>
    Signed-off-by: Linus Torvalds <email address hidden>
    Signed-off-by: Greg Kroah-Hartman <email address hidden>
    Acked-by: Steve Conklin <email address hidden>
    Acked-by: Stefan Bader <email address hidden>
    Acked-by: Brad Figg <email address hidden>
    Signed-off-by: Brad Figg <email address hidden>

 include/linux/mm_types.h | 2 +-
 kernel/fork.c | 7 +++++--
 mm/mmap.c | 39 +++++++++++++++++++++++++--------------
 mm/nommu.c | 7 +++++--
 4 files changed, 36 insertions(+), 19 deletions(-)

Reverting allows me to boot the VM. However if I compile mainline or stable 2.6.32.21 that has this commit as well then I am able to boot just fine.

I expect we are witnessing incompatibility of the patch with Ubuntu "sauce". I suspect it might be this one:

commit 59b35f3ba2cba40bd260ad19dbf4cf445714044f
Author: Kees Cook <email address hidden>
Date: Thu Apr 30 12:47:47 2009 -0700

    UBUNTU: SAUCE: [x86] implement cs-limit nx-emulation for ia32

but it is too tangled up with other changes to be reverted cleanly to test this theory.

I can confirm that this happens with kernel 2.6.32-27-generic-pae on a VMware ESX 4.1 (build 320092). udevadm segfaults right after the kernel has finished loading, and it inevitable leads to failure to mount the root filesystem.

Eagerly waiting for a fix!

Forgot to say that both kernels -27 and -26 work just fine on ESX 4.0.

It may be worthwhile to note that the same errors above look like the same errors when running any netboot image newer than the linux-20081029ubuntu100 eg:
32bit:
http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-i386/20081029ubuntu100/images/netboot/ubuntu-installer/i386/linux
http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-i386/20081029ubuntu100/images/netboot/ubuntu-installer/i386/initrd.gz

64bit:
http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-amd64/20081029ubuntu100/images/netboot/ubuntu-installer/amd64/linux
http://archive.ubuntu.com/ubuntu/dists/lucid/main/installer-amd64/20081029ubuntu100/images/netboot/ubuntu-installer/amd64/initrd.gz

Anything _newer_ (ubuntu101, ubuntu102) seems to fail with the already mentioned segfault.

It looks like the 2.6.35-* versions of the kernel do not have the same problem.

This worked for me:
- If you don't have an ealier kernel version to boot into; in VMware, disable acceleration for the VM. (Right click: Edit Settings... -> Options -> General -> Disable acceleration)
- Start the VM. It'll run very slowly with acceleration disabled.
- Install one of the 2.6.35 kernel versions. Example: linux-image-2.6.35-25-generic
- Shut down the VM.
- Re-enable Acceleration if you disabled it above.
- Start the VM. Right at the start, as grub boots, hold down a shift key. Select your newly installe 2.6.35-* kernel.

I am using a few of different JumpBoxes, and after migrating them from ESXi4.0 to ESXi4.1 I went to boot them back up, & in 2 of them have received the error message saying:

Segmentation fault
Gave up waiting for root device

I have already tried switching the OS Type in vSphere device manager from "Ubuntu Linux 32-bit" to "Other Linux 32-bit"

Still segfaults all the way up to kernel .32 (from .25), on fully updated virtual machines.

I have the same problem after upgrading Vmware ESXi 4.0 to ESXi4.1.
I've found another work around that works for me:

In the vSphere manager, shutdown the virtual machine and edit it settings: go to the Options tab, and select "CPU/MMU Virtualization". There, change the selected option from "Automatic" (the first) to "Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization" (the third).

After that, I can boot with any 2.6.32-3x kernel.

Changing any of the VM settings did not work for me (VMware 4.1), 2.6.32-38-generic failed with udevd segfault.
Finally I installed the virtual kernel (apt-get install linux-image-virtual), which is suitable for VMware according to the Ubuntu Server FAQ:
https://help.ubuntu.com/community/ServerFaq#What_are_the_differences_between_the_server_and_virtual_kernels.3F

This installs a 2.6.32-38-generic-pae kernel and solves the problem.

Failed to start up immediately after a virgin install of 32-bit Server (Minimal Virtual Install option) 10.04 LTS on VMWare ESXi 4.1. Get the segfault after the udevd failure line, then a revert to busybox.

Changing VMWare's identified OS from Ubuntu 32-bit to any other value does not resolve issue.
Changing VMWare's CPU/MMU Virtualization value from Automatic to "Use Intel VT-x/AMD-V" does not resolve issue.
The only way to boot the VM is to turn Acceleration off.

Once booted, I reverted back to the earlier known good kernel, 2.6.34.24:
$ apt-get install linux-image-2.6.32-24-virtual
$ echo "linux-image-2.6.32-33-virtual hold" | sudo dpkg --set-selections # stock kernel on distro
$ echo "linux-image-2.6.32-41-virtual hold" | sudo dpkg --set-selections # current dist-upgrade target for this distro

Then I turned Acceleration back on and rebooted. Works. W00t!

$ uname
Linux myboxname 2.6.32-24-generic-pae #43-Ubuntu SMP Thu Sep 16 15:30:27 UTC 2010 i686 GNU/Linux