Comment 27 for bug 688541

On Thu, 2010-12-23 at 12:40 +0000, ingo wrote:
> @Clint:
>
> I did test your proposal in Maverick.
>
> Before editing the stop scripts:
>
> fgrep "stop on runlevel" /etc/init/*.conf
> /etc/init/acpid.conf:stop on runlevel [!2345]
> /etc/init/anacron.conf:stop on runlevel [!2345]
> /etc/init/apport.conf:stop on runlevel [!2345]
> /etc/init/atd.conf:stop on runlevel [!2345]
> /etc/init/cron.conf:stop on runlevel [!2345]
> /etc/init/cups.conf:stop on runlevel [016]
> /etc/init/dbus.conf:stop on runlevel [06]
> /etc/init/failsafe-x.conf:stop on runlevel [06]
> /etc/init/gdm.conf:stop on runlevel [016]
> /etc/init/irqbalance.conf:stop on runlevel [!2345]
> /etc/init/mountall-shell.conf:stop on runlevel [06]
> /etc/init/rc.conf:stop on runlevel [!$RUNLEVEL]
> /etc/init/rcS.conf:stop on runlevel [!S]
> /etc/init/rc-sysinit.conf:stop on runlevel
> /etc/init/rsyslog.conf:stop on runlevel [06]
> /etc/init/tty1.conf:stop on runlevel [!2345]
> /etc/init/tty2.conf:stop on runlevel [!23]
> /etc/init/tty3.conf:stop on runlevel [!23]
> /etc/init/tty4.conf:stop on runlevel [!23]
> /etc/init/tty5.conf:stop on runlevel [!23]
> /etc/init/tty6.conf:stop on runlevel [!23]
> /etc/init/udev.conf:stop on runlevel [06]
> /etc/init/ufw.conf:stop on runlevel [!023456]
>
> After editing the stop scripts:
>
> fgrep "stop on starting" /etc/init/*.conf
> /etc/init/cups.conf:stop on starting rc RUNLEVEL=[016]
> /etc/init/dbus.conf:stop on starting rc RUNLEVEL=[06]
> /etc/init/failsafe-x.conf:stop on starting rc RUNLEVEL=[06]
> /etc/init/gdm.conf:stop on starting rc RUNLEVEL=[016]
> /etc/init/mountall.conf:stop on starting rcS
> /etc/init/mountall-shell.conf:stop on starting rc RUNLEVEL=[06]
> /etc/init/rsyslog.conf:stop on starting rc RUNLEVEL=[06]
> /etc/init/udev.conf:stop on starting rc RUNLEVEL=[06]
>
> Then execute
>
> apt-get install --reinstall libc6
> and reboot:
>
> I still get the 8 orphaned inodes as reported already.
>
> Did I miss to change the other scrips as well like this?
> stop on runlevel [!2345] -> stop on stop on starting rc RUNLEVEL=[016]
>

Yes, and some of those are probably the most likely to have libc open.

If doing the same to all of the !2345's does not fix the corruption, can
you do:

apt-get install --reinstall libc6
lsof -n |grep deleted
initctl list

And paste or upload the output of that here?

> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/688541
>
> Title:
> race condition on shutdown (leads to corrupted fs)
>
> Status in “mysql-5.1” package in Ubuntu:
> Triaged
> Status in “sysvinit” package in Ubuntu:
> Triaged
>
> Bug description:
> I'm using mysql-server-5.1 on a 10.04 LTS installation.
> The mysql db is around 27GB and on a separate partition mounted as /var/lib/mysql.
>
> On shutdown I get the following error message:
>
> Checking for running unattended-upgrades: * Asking all remaining processes to terminate...
> [80G
> [74G[ OK ]
> * All processes ended within 1 seconds....
> [80G
> [74G[ OK ]
> * Deconfiguring network interfaces...
> [80G
> [74G[ OK ]
> * Deactivating swap...
> [80G
> [74G[ OK ]
> * Unmounting local filesystems...
> [80G umount2: Device or resource busy
> umount: /var/lib/mysql: device is busy.
> (In some cases useful info about processes that use
> the device is found by lsof(8) or fuser(1))
> umount2: Device or resource busy
> umount2: Device or resource busy
> umount: /tmp: device is busy.
> (In some cases useful info about processes that use
> the device is found by lsof(8) or fuser(1))
> umount2: Device or resource busy
> [74G[
> [31mfail
> [39;49m]
> mount: / is busy
> * Will now restart
> [ 3369.429751] Restarting system.
>
>
> On the next reboot the file system is corrupt and need to be fsck-ed.
>
> I think the problem is, that mysql uses an upstart job (/etc/init/mysql.conf) and has
> stop on runlevel [016]
>
> The rc.conf job is also triggered on runlevel 0 and 6, so they basically run at the same time.As
>
> When /etc/rc0.d/S20sendsigs is run, it deliberatly does not wait or kill any upstart jobs.
>
> As my mysqld process takes some time to shutdown, S40umountfs and S60umountroot are run before the mysqld has quit.
>
> Leading to the fs not being properly unmounted. It is event possible that mysqld is forcefully killed by halt in S90halt if it hasn't stopped by then.
>
> This is a serious issue, as it can (and will) lead to data loss.
>
> Other upstart jobs, like rsyslog.conf, use the same "stop on runlevel [016]" stanza, so they are probably affected too.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 10.10
> Package: mysql-server-5.1 5.1.49-1ubuntu8.1
> Uname: Linux 2.6.32-5-686 i686
> NonfreeKernelModules: michael_mic arc4 ecb lib80211_crypt_tkip aes_i586 aes_generic lib80211_crypt_ccmp sco bnep rfcomm l2cap binfmt_misc acpi_cpufreq ppdev lp cpufreq_userspace cpufreq_stats vboxnetadp cpufreq_powersave vboxnetflt cpufreq_conservative vboxdrv fuse pcmcia snd_intel8x0m snd_intel8x0 snd_ac97_codec btusb bluetooth rfkill ac97_bus yenta_socket ipw2200 snd_pcm 8139too firewire_ohci snd_seq 8139cp firewire_core sg uhci_hcd snd_timer rsrc_nonstatic libipw snd_seq_device pcmcia_core crc_itu_t parport_pc smsc_ircc2 ehci_hcd mii joydev lib80211 sr_mod parport i2c_i801 irda snd usbcore wbsd soundcore shpchp mmc_core pcspkr rng_core cdrom psmouse container crc_ccitt snd_page_alloc serio_raw pci_hotplug ac battery nls_base processor evdev ppp_generic slhc loop autofs4 ext4 mbcache jbd2 crc16 dm_mod sd_mod crc_t10dif radeon ttm ata_generic drm_kms_helper ata_piix drm i2c_algo_bit libata video thermal i2c_core scsi_mod output thermal_sys button
> Architecture: i386
> Date: Fri Dec 10 13:41:52 2010
> ProcEnviron:
> PATH=(custom, no user)
> LANG=de_DE.utf8
> SHELL=/bin/bash
> SourcePackage: mysql-5.1
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/mysql-5.1/+bug/688541/+subscribe