karmic boot hung after /scripts/init-bottom

Bug #430272 reported by Vinson Lee
104
This bug affects 19 people
Affects Status Importance Assigned to Milestone
upstart (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

After applying updates to karmic on September 15, 2009, the machine no longer boots. It hangs after running /scripts/init-bottom. This is the case for regular boot and recovery mode as well. A screen shot of the console is attached.

The machine successfully booted with updates on September 14, 2009.

Revision history for this message
Vinson Lee (vlee) wrote :
Revision history for this message
starslights (starslights) wrote :

hello,

same case for me, i have updated today my kubuntu karmic x86 64 and can't start anymore, sometime can login but no peripheric work anymore and if log say : kstartupconfig4 does not exist or fails. the error code is 3

like i have reported in : https://bugs.launchpad.net/ubuntu/+bug/428403

Revision history for this message
Hilario J. Montoliu (hjmf) (hmontoliu) wrote :

I can confirm the problem

I'm not sure which package is causing the issue. The updates I applied before the problem are:

branch nick: ulises-devel /etc repository
timestamp: Tue 2009-09-15 22:10:04 +0200
message:
  committing changes in /etc after apt run

  Package changes:
  -acpid 1.0.6-9ubuntu5
  +acpid 1.0.6-9ubuntu6
  -anacron 2.3-13.1ubuntu8
  +anacron 2.3-13.1ubuntu9
  -at 3.1.11-1ubuntu3
  +at 3.1.11-1ubuntu4
  -avahi-autoipd 0.6.25-1ubuntu2
  -avahi-daemon 0.6.25-1ubuntu2
  +avahi-autoipd 0.6.25-1ubuntu3
  +avahi-daemon 0.6.25-1ubuntu3
  -cron 3.0pl1-106ubuntu2
  +cron 3.0pl1-106ubuntu3
  -dbus 1.2.16-0ubuntu2
  -dbus-x11 1.2.16-0ubuntu2
  +dbus 1.2.16-0ubuntu3
  +dbus-x11 1.2.16-0ubuntu3
  -gdm 2.27.90-0ubuntu4
  +gdm 2.27.90-0ubuntu5
  -hal 0.5.13-1ubuntu2
  +hal 0.5.13-1ubuntu3
  -hostname 2.95
  +hostname 2.95ubuntu1
  -libavahi-client3 0.6.25-1ubuntu2
  -libavahi-common-data 0.6.25-1ubuntu2
  -libavahi-common3 0.6.25-1ubuntu2
  -libavahi-core6 0.6.25-1ubuntu2
  -libavahi-glib1 0.6.25-1ubuntu2
  -libavahi-gobject0 0.6.25-1ubuntu2
  -libavahi-ui0 0.6.25-1ubuntu2
  +libavahi-client3 0.6.25-1ubuntu3
  +libavahi-common-data 0.6.25-1ubuntu3
  +libavahi-common3 0.6.25-1ubuntu3
  +libavahi-core6 0.6.25-1ubuntu3
  +libavahi-glib1 0.6.25-1ubuntu3
  +libavahi-gobject0 0.6.25-1ubuntu3
  +libavahi-ui0 0.6.25-1ubuntu3
  -libdbus-1-3 1.2.16-0ubuntu2
  +libdbus-1-3 1.2.16-0ubuntu3
  -libglib2.0-0 2.21.6-0ubuntu6
  +libglib2.0-0 2.21.6-0ubuntu7
  -libglib2.0-data 2.21.6-0ubuntu6
  +libglib2.0-data 2.21.6-0ubuntu7
  -libhal-storage1 0.5.13-1ubuntu2
  -libhal1 0.5.13-1ubuntu2
  +libhal-storage1 0.5.13-1ubuntu3
  +libhal1 0.5.13-1ubuntu3
  -libnm-glib2 0.8~a~git.20090911t130220.4c77fa0-0ubuntu2
  -libnm-util1 0.8~a~git.20090911t130220.4c77fa0-0ubuntu2
  +libnm-glib2 0.8~a~git.20090911t130220.4c77fa0-0ubuntu3
  +libnm-util1 0.8~a~git.20090911t130220.4c77fa0-0ubuntu3
  +mountall 0.1.3
  -netbase 4.35ubuntu1
  +netbase 4.35ubuntu2
  -network-manager 0.8~a~git.20090911t130220.4c77fa0-0ubuntu2
  +network-manager 0.8~a~git.20090911t130220.4c77fa0-0ubuntu3
  -python-avahi 0.6.25-1ubuntu2
  +python-avahi 0.6.25-1ubuntu3
  -sysv-rc 2.87dsf-4ubuntu2
  -sysvinit-utils 2.87dsf-4ubuntu2
  +sysv-rc 2.87dsf-4ubuntu3
  +sysvinit-utils 2.87dsf-4ubuntu3
  -ubuntu-artwork 46.3
  +ubuntu-artwork 47
  -ubuntu-minimal 1.168
  -upstart 0.6.3-1
  +upstart 0.6.3-2
  -usplash-theme-ubuntu 0.24
  +usplash-theme-ubuntu 0.25
  -xulrunner-1.9.1 1.9.1.3+build1+nobinonly-0ubuntu1
  -xulrunner-1.9.1-dbg 1.9.1.3+build1+nobinonly-0ubuntu1
  -xulrunner-1.9.1-gnome-support 1.9.1.3+build1+nobinonly-0ubuntu1
  +xulrunner-1.9.1 1.9.1.3+build1+nobinonly-0ubuntu2
  +xulrunner-1.9.1-dbg 1.9.1.3+build1+nobinonly-0ubuntu2
  +xulrunner-1.9.1-gnome-support 1.9.1.3+build1+nobinonly-0ubuntu2

I've been trying to boot w/o no succeed

Changed in ubuntu:
status: New → Confirmed
Revision history for this message
robinsk (robinsk) wrote :

I think I might a similar issue. Did a partial upgrade, and booting hangs with a blank screen after the progress bar reaches 100%. I pressed Esc and logged into the console. It seemed two packages were broken: initscripts and another package I can't remember the name of. The other package depended on mountall, which couldn't be found as a deb in any repos. Installed mountall using apt-get source mountall --compile, then dpkg -i ./mountall_x.y.z.deb, followed by apt-get install initscripts and the other package.

When I now reboot, Memtest86 is launched immediately. Nothing happens if I press Esc or F9 during boot. In fact, I'm not even sure it says "GRUB loading".

Revision history for this message
MT (micdhack) wrote :

I can confirm that the same thing happened to me. After before 1-2 hours i did a dist-upgrade and system wont boot after reboot.
The system hangs after /scripts/init-bottom.

In my case i manage to login to the system by editing through grub and putting rw instead of ro and init=/bin/bash
When i tried an apt-get upgrade the package initscripts has been kept back. if i force install it the system ask me to remove almost all packages (gdm, ubuntu desktop, etc)

The worst is that i can get to connect to the net so i have no idea how am i going to fix that system failure.

Revision history for this message
Paul Dorman (paul-dorman) wrote :

I had the same problem today. The good news is that after five minutes the system comes back up. I started on the previous kernel, but I don't think that is a factor.

Hopefully others will have the same experience until a fix is uploaded to the repositories.

Revision history for this message
Paul Dorman (paul-dorman) wrote :

Ah, the joy of speaking too soon. Comes up yes, and hangs...

Revision history for this message
MT (micdhack) wrote :

:D lol
Ok since the initscripts is the one which refuses to be installed im thinking installing it from the source
http://packages.ubuntu.com/karmic/initscripts

Revision history for this message
Paul Dorman (paul-dorman) wrote :

Rebooted again to the newly installed kernel this time, and things look better. The only change was to install mountall.

Michael, you should be able to bring up your network connection with dhclient or ifconfig. dhclient worked for me.

Revision history for this message
Hilario J. Montoliu (hjmf) (hmontoliu) wrote :

Assigned to initramfs-tools as the system hangs at init-bottom (just a clue)

affects: ubuntu → initramfs-tools (Ubuntu)
Revision history for this message
Hilario J. Montoliu (hjmf) (hmontoliu) wrote :

hmm, forget my last comment as initramfs-tools is not an updated package, my fault

affects: initramfs-tools (Ubuntu) → ubuntu
Revision history for this message
Paul Dorman (paul-dorman) wrote :

Just upgraded again. upstart was the only package. Rebooted to one system beep and a blank screen. Left for ten minutes, then rebooted successfully.

Revision history for this message
MT (micdhack) wrote :

I just upgrade too and 4 packages along with initscripts were installed. After restart though im getting a weird message

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)..

Revision history for this message
Paul Dorman (paul-dorman) wrote :

Latest initscripts package installed. Rebooted successfully. Looks like this bug is fixed. Thanks Ubuntu team!

Revision history for this message
MT (micdhack) wrote :

A superblock for that kernel was f***ed so i booted with an older on did an fsck and then rebooted.

I can verify that the system boots normally with the old kernel and the latest one.

Revision history for this message
Hilario J. Montoliu (hjmf) (hmontoliu) wrote :

After applying the last upgrades the system still freeze at boot, but now with message:

udevd[959]: unknown key 'SYMLINK{unique}' in /lib/udev/rules.d/50-udev-default.rules:3
udevd[959]: unknown key 'SYMLINK{unique}' in /lib/udev/rules.d/50-udev-default.rules:4

Revision history for this message
neferty (roman-semko) wrote :

getting the same SYMLINK error as Hilario..

Revision history for this message
Lou Ruppert (louferd) wrote :

I reported the udevd/symlink issue as 430667.

Revision history for this message
David Gibson (dwg) wrote :

Still totally broken here with updates as of Sep 16 (Australian time).

Revision history for this message
David Gibson (dwg) wrote :

More specifically it seems that mountall never mounts /home amongst some other things, never emits the 'filesystem' event and so nothing starts.

Revision history for this message
Lou Ruppert (louferd) wrote :

Since people are still posting here instead of 430667, I yanked the duplicate status (I think).

Revision history for this message
Hilario J. Montoliu (hjmf) (hmontoliu) wrote :

Since seems that some people got this issue fixed by some late night update and I was still stuck w/o booting (no matter if I applied those updates too) I've reinstalled my testing partition and upgrade all pending updates.

Now the system boots fine (I still see those udev warnings from bug #430654 but that's certainly another issue)

So the package (upstart?) that broke the boot process seems to have been fixed in some update last night or today but that fix seems that wasn't able to repair the mess that the buggy update caused at first.

Now I'm happy again :-)

Revision history for this message
Carfield Yim (carfield) wrote :

I still have this problem.... first of all.... How can I boot again and preform fix?? BTW, I just update today, just now, 17th Septh, 0:00 GMT+8

Revision history for this message
Hilario J. Montoliu (hjmf) (hmontoliu) wrote :

Carfield Yim,

I wasn't able to fix the issue directly from updates. However if you want to boot into something you can edit the boot parameters in grub:

"e" to edit
replace the boot parameters from ro xxxx xxxx to "rw init=/bin/bash"

and you'll be able to access a limited shell (no job control)

hence you can enable your network (dhclient didn't work for me) ie:

ifconfig eth0 <yourip>/<your netmask> up; route add default gw <your gw> eth0; echo "nameserver <your favorite dns>" > /etc/resolv.conf

Hence you'll be able to run apt-get or aptitude

As said before, despite all the updates applied in the last 15 hours, I had to reinstall to get the issue fixed

Revision history for this message
luigi.mantellini@gmail.com (comio) wrote :

Hi All,

Can sombody tell me what is wrong in the upstart package and how i can fix? I cannot reinstall again the machine and would like to continue to use the karmic.

best regards,

luigi

Revision history for this message
Paulo Assis (pj-assis) wrote :

Hi,
For those that can't boot to your system no matter what, here is my solution (chroot from live cd):

1. boot from the live cd
2. open console
3. mount your root partition:
    # sudo mkdir /media/SYS_ROOT
    # sudo mount /dev/sda1 /media/SYS_ROOT (change sda1 to whatever device contains your filesystem)
4. chroot to /media/SYS_ROOT:
    # sudo chroot /media/SYS_ROOT
5. update your system
    # sudo apt-get update
    # sudo apt-get upgrade
6. reboot.

The above steps worked for me, and my system is up and running (chroot will be missing /proc this will cause some warnings about apt log but in my case it upgrade correctly)

Regards,
Paulo

Revision history for this message
luigi.mantellini@gmail.com (comio) wrote :

I already upgrade a lot of times my system without success and the problem still remain. I will try to upgrade tomorrow hoping to solve the problem.

best regards,

luigi

Revision history for this message
Craig Kelley (ink) wrote :

I booted from a live cd and ran apt-get update; apt-get upgrade, but the system reboots after doing this:

Setting up upstart (0.6.3-3)

I've done it three times now, and any use of dpkg or apt results in the message:

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct this problem

Which, if I do, causes upstart to run its post-install scripts and reboot the system.

Any ideas?

Revision history for this message
Lou Ruppert (louferd) wrote :

I suspect upstart is the root of the issue, but it's hard to tell.

affects: ubuntu → upstart (Ubuntu)
Revision history for this message
Lou Ruppert (louferd) wrote :

I created a fresh debootstrap install for my system and encountered the same problem, so it's not an issue of timing the updates wrong, as others have suggested.

Revision history for this message
Michael Casadevall (mcasadevall) wrote :

I managed to reproduce this on SPARC/karmic, same issue with the machine spontaneously restarting. However, on my machine, I get the following panic after it hangs for ~20 seconds. upstart on SPARC is very unhappy

[ 111.681670] Kernel panic - not syncing: Attempted to kill init!
[ 111.759719] Call Trace:
[ 111.791832] [000000000046298c] find_new_reaper+0x8c/0xa0
[ 111.862993] [0000000000463ea0] forget_original_parent+0x20/0x220
[ 111.943154] [00000000004640a4] exit_notify+0x4/0x180
[ 112.009512] [0000000000464380] do_exit+0x160/0x320
[ 112.073583] [0000000000464564] do_group_exit+0x24/0xc0
[ 112.142247] [000000000046fb88] get_signal_to_deliver+0x268/0x360
[ 112.222434] [0000000000442818] do_signal32+0x18/0x240
[ 112.289950] [000000000042c768] do_signal+0x348/0x5c0
[ 112.356393] [000000000042ca04] do_notify_resume+0x24/0x60
[ 112.428475] [0000000000404aa8] __handle_signal+0xc/0x2c
[ 112.498265] Press Stop-A (L1-A) to return to the boot prom

Revision history for this message
Steve Langasek (vorlon) wrote :

Thank you for taking the time to report this problem and help to improve Ubuntu.

The original problem in this bug report, hangs at boot time, has been reported fixed by most of the users following up here. Since there have been extensive fixes to the boot scripts in the two days since this was reported, I believe this bug is fixed and as such am marking it as closed.

If you are the original submitter of this bug and are still seeing problems after applying all updates to your system, please feel free to reopen.

If you are *not* the original submitter of this bug and you are still having problems booting after applying all updates to your system, please open a separate bug report and describe precisely what you see on your own system when booting.

If you have questions about how to apply updates to a non-booting system, please see <https://wiki.ubuntu.com/RecoveryMode>.

Changed in upstart (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Václav Šmilauer (eudoxos) wrote :

The link to RecoveryMode page is not evry helpful, as this bug is present in the recovery mode as well (and with init=/bin/sh as well). Someone has a better suggestion?

Revision history for this message
Ernst Zlo (ernst-zlo) wrote :

After upgrading some error messages while booting remain (while beeing able to boot)

Revision history for this message
Václav Šmilauer (eudoxos) wrote :

Ernst: true, but that woule be a separate bug, I think.

For how to recover when recovery mode doesn't work:

1. in grub, use recovery mode, but edit the kernel command line and replace "ro single" with "rw break=init"
2. You will be dropped to the (initramfs) prompts, regular root is mounter at /root.
3. Do chroot /root /usr/bin/*sh (bash and /bin/sh hung for me, pehaps that was the bug? I used zsh instead, try dash as well)
4. Set network connection (ifconfig eth0 10.0.0.18; route add default gw 10.0.0.1; echo nameserver 10.0.0.1 > /etc/resolv.conf)
5. aptitude update && aptitude upgrade
6. reboot

Revision history for this message
luigi.mantellini@gmail.com (comio) wrote :

I confirm that the last upstart 0.6.3-3 solves the issue for me.

thanks and best regards

luigi

Revision history for this message
switch (mjw8ight) wrote :

Hi I've been having the same problem.

I just went through the process of booting from the live cd and sudo chroot /mnt , apt-get update and apt-get upgrade.

Ubuntu 9.10 is still hanging at scripts/init-bottom.

this is the error I get.

udevd[813] inotify-add-watch(6,(null),10) failed: bad address

terminate this shell and retry.ted with status 10-udev-default.rules:4t

also when I issue the apt-get upgrade command I get the following error:

Errors were encounted while processing linux-image-2.6.31-10 generic
grub-pc
grub2
E:sub-process /usr/bin/dpkg returned an error code(1)

an additional error went something like the following:

can not write log, openpty() failed.

any help would be greatly appreciated.

Thank you

Revision history for this message
chisao (michayc) wrote :

I fixed it somehow...Many steps and I don't know if I remember them all.First...press e on recovery to edit.change ro single to rw init=/bin/bash.Then type dhclient eth0 (connect an ethernet cable first if you havent already).If it manages to connect (which it should) type aptitude update and then aptitude upgrade.It will update (mine failed a few packages because something was not mounted...I dont know exactly what happened).After this I tried to boot using the latest kernel but it froze sayin somethin about kernel panic......initramfs....vsync....I dont remember exactly.Anyway I chose the previous kernel in recovery mode run fsck, rebooted again with the previous kernel in recovery and the options were there.I chose make free space then fsck and then resume normal boot and I got in kde.I ran aptitude again aptitude autoremove apt-get autoclean apt-get autoremove aptitude update aptitude upgrade and all that stuff...rebooted with new kernel and after a few messages I got in....Hope it helps.

Revision history for this message
chisao (michayc) wrote :

I am really sorry!!!Ignore my comment right above.I meant to write it for another bug...If someone can delete it plz do....I am sorry again!!! :(

Revision history for this message
Bernd Schubert (aakef) wrote :

It hangs here after writing "fsck from until-linux 2.16"
/dev/md0: clean

I don't see how this can be considered as "Fix released". And honestly, I don't care if my system needs 5s more to boot, but I care a lot, if the boot process is not systematic anymore. With plain sysrc it was simple too see where something failed, but now? I don't have the slightest idea which script hangs and where. This against primary rule: Keep it simple!
And even X depends on upstart and not plain init? Maybe time to go looking for another distribution.

Revision history for this message
manwe (nettibug+launchpad) wrote :

I ran aptitude update + upgrade on "Mon 28.9.2009 "
and the hang problem is fixed (I did not check what were the packages that got updates)

See #430667 for more of my ramblings

Cheers,
 Manwe

ps. Thank you devs for fixing this (without me needing to reinstall!!!)

Revision history for this message
Benji (eilif) wrote :

After doing an upgrade from jaunty, I had the same problems as Bernd Schubert

"It hangs here after writing "fsck from until-linux 2.16"

I tracked it down to my /boot partition being ext2 ... because I noticed that karmic could not automatically identify the ext2 partition type, as it should. Switching /boot to ext3, my machine now boots.

Revision history for this message
Steve Langasek (vorlon) wrote :

Benji, you should file a new bug report for this unrelated issue.

Revision history for this message
Jaen Saul (slowbyte) wrote :

Got the same issue today (system hangs after /scripts/init-bottom)

Thanks to poster before, I was able to fix the issue.

My steps:
1) change recovery mode command line in GRUB from "ro single" to "rw break=init"
2) chroot /root /bin/bash
3) dhclient (to get IP/network configuration)
4) apt-get update
5) apt-get install upstart initscripts sysv-rc sysvinit-utils initramfs-tools
6) reboot (system fails)
7) reboot again (system comes up this time)

Also, have the maintainers/developers of critical system packages like upstart thought about, you know, testing? It's not that hard to set up a VM with qemu/KVM. One hour of your time could save hundreds of hours of your users' time and frustration. Thanks.

Revision history for this message
Quinn Plattel (qiet72) wrote :

Apparantly, even though a fix has been made to upstart, there is still no way to install or upgrade to karmic on sparc.

If you attempt upgrade, upstart will fail or crash the system.

If you try to install the system from CD, booting from CD will fail with a "Fast Data Access MMU miss" because not enough memory is allocated to hold both the kernel and the initrd on newer ubuntu releases - Ubuntu Hardy 8.04 was the last release to be able to boot from cd. Lucid and up have solved the memory issue of the cd bootloader, but it does not seem to load the sd driver so you cannot access any block devices even though you have the correct scsi/ide driver loaded.

The only solution left is to install Hardy and then do an LTS upgrade to Lucid - mind you that you might have an issue with apparmor so you should probably remove that package before upgrading.

Quinn

Revision history for this message
panticz.de (panticz.de) wrote :

Did you check that die file owner and permissions are correct on the root filesystem? Especially the init script to by able to start the second boot stage:

# ls -l /sbin/init
-rwxr-xr-x 1 root root 167192 Jan 18 2013 /sbin/init

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.