shutdown fails.. nfs

a) I assume you mean K50open-iscsi rather than S50open-iscsi

b) How does this fix the problem? K50open-iscsi would still get called before the nfs scripts (which are at K79 and K80).

This bug is actually much more substantial than just NFS servers as indicated above. Any service that depends on iSCSI will cause problems. For instance, if the machine above has its postfix mail queue on an iSCSI volume, postfix can't shut down since the storage has been ripped out from under it!

There are serious issues with the iscsi startup and shutdown scripts. Examples:

- There is no mechanism to unmount iscsi volumes before shutting down iscsi
- The default startup scripts try to start iscsi before networking

In fact, I would say iSCSI support on Ubuntu (and Debian for that matter) is purely "experimental" at this time. For iSCSi to be considered of any real use:

1. boot-time filesystem checks should work
2. iscsi volumes should be mounted before anything else tries to start
3. iscsi volumes should be cleanly unmounted _AFTER_ all other services have shut down

With a fresh Hardy installation I see:

- iSCSI does not start up automatically at boot time, even though it appears it should
- iSCSI gets shut down before just about everything else, crashing the system on shutdown
- No mechanism in place to check if volumes are in use before shutting down iSCSI

RedHat has had this working properly since Enterprise 3. SuSE has had this working properly since SuSE Enterprise 9. We are YEARS behind here!!

On Fri, Mar 28, 2008 at 01:04:46AM -0000, LinuxBladeGuy wrote:
> There are serious issues with the iscsi startup and shutdown scripts.

Oh, dear.

> Examples:
> - There is no mechanism to unmount iscsi volumes before shutting down iscsi

> - The default startup scripts try to start iscsi before networking

Not true (unless you're using network manager for your network
configuration). Interfaces are brought up as udev discovers them, which
is before we attempt to start iscsi.

> In fact, I would say iSCSI support on Ubuntu (and Debian for that
> matter) is purely "experimental" at this time. For iSCSi to be
> considered of any real use:
>
> 1. boot-time filesystem checks should work

Don't they?

> 2. iscsi volumes should be mounted before anything else tries to start

What's wrong with mounting them at the same time everything else is
mounted?

> 3. iscsi volumes should be cleanly unmounted _AFTER_ all other services have shut down

Correct. I wonder if simply not stopping iscsi during shutdown is a
reasonable approach. The kernel should flush everything anyway, so it
ought to be safe. Could you try that?

> With a fresh Hardy installation I see:
>
> - iSCSI does not start up automatically at boot time, even though it appears it should

Can you open a bug with some info about this?

--
Soren Hansen |
Virtualisation specialist | Ubuntu Server Team
Canonical Ltd. | http://www.ubuntu.com/

I found the cause of iSCSI not starting at boot, it's caused by having /usr on a separate file system. I have opened a new bug here:

https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/208441

I'm rebuilding my test system with /usr on the same partition as / and will see what else I can troubleshoot...

I've had a lot of experience with iSCSI on other distros and am more than happy to help test things out... We've got a pretty decent iSCSI SAN here. On the plus side, the initial performance tests I ran were extremely good, and I'm running this in VMWare!!

Not stopping iSCSI on shutdown doesn't actually fix the problem... We need to make sure that the iSCSI volumes are unmounted and then iSCSI is shut down properly.

If you watch the shutdown process, you see that all remaining processes get terminated before filesystems are unmounted. Unfortunately iscsid is one of the processes that gets killed. Result is the system hangs on shutdown...

Note that Debian has all the same problems... ;) Ask me how I know...

Now that I've done some more hammering away on this I can confirm that networking does indeed come up properly before iSCSI is started, as udev discovers the devices. I can also confirm that boot time filesystem checks are working properly. And yes, mounting iscsi volumes at the same time as everything else is perfectly fine... :)

One more tidbit to throw on here with regards to shutting down open-iscsi...

If open-iscsi is not shut down cleanly, it never flushes write caches to disk... So it absolutely must be shut down properly or we potentially loose data. I'm talking about caches between us and the physical storage - ie on the iSCSI target, or on the back end disk array itself. During shutdown open-iscsi ensures that all data is actually commited to disk.

Workaround:

In /etc/rc0.d change the symlink

K25open-iscsi

to

S80open-iscsi

Then try a shutdown test.

If successful, do something similar for /etc/rc6.d.

This bug was fixed in the package open-iscsi - 2.0.870.1-0ubuntu1

---------------
open-iscsi (2.0.870.1-0ubuntu1) jaunty; urgency=low

  * New upstream release:
   - Support 2.6.26/27 kernels (LP: #289470).
   - Fix iscsid shutdown (LP: #181188).
  * Merge from Debian. Remaining Ubuntu changes:
   - Note: Debian version isn't 870~rc3 but 869.2 which leads to a big
     .diff.gz file. Only files in debian/ have been considered for this merge
     since debian hasn't patched the source.
   - debian/control, debian/rules, debian/open-iscsi-udeb*:
     + Add open-iscsi-udeb.
   - debian/open-iscsi.dirs:
     + rename dirs to open-iscsi.dirs because of open-iscsi-udeb addition.
     + drop network/if-up.d/ directory since we're using symlinks instead.
     + utilities installed in /bin,/sbin rather than /usr/bin,/usr/sbin.
   - debian/open-iscsi.init:
     + utilities installed in /bin,/sbin rather than /usr/bin,/usr/sbin.
     + lsb log messages.
     + Don't generate initiatorname name (moved to postinst).
     + Support for being called from ifup/ifdown scripts.
     + Refactor start functions:
       - move daemon start to startdaemon function.
       - call udevadm settle rather then udevsettle (which doesn't do anything
         useful).
       - don't exit if the daemon is already running during sanitychecks.
         Instead check in startdaemon if the daemon needs to be started.
       - only start automatic targets if necessary.
       - add waitfordevices function: called during rcS, waits for all
         automatic targets to be ready. Iscsi targets are considered as
         local block devices - they don't need to be marked with _netdev in
         fstab. (LP: #227848)
       - start targets if not run from rcS.
     + Check status of iscsid daemon in addition to listing the iscsi sessions
       when status action is called.
     + Add iscsid to the list of processes that should not be killed by
       sendsigs during shutdown (LP: #192080).
     + Add starttargets, stoptargets and restarttargets to usage message.
   - debian/rules:
     + Install utilities /bin,/sbin rather than /usr/bin,/usr/sbin.
     + Start open-iscsi at S25 (waiting for devices created by ifupdown
       calls and before local filesystems are checked and mounted)
     + stop at S41 (after local filesystems are unmounted). Don't use
       umountiscsi.sh script from debian. (LP: #192080).
   - debian/initiatorname.iscsi, debian/open-iscsi.postinst:
     + Generate the random initiatorname during postinstall rather than in the
     init script.
     + Don't ship a default initiatorname.iscsi file.
   - debian/open-iscsi.postinst:
     + fix init script ordering on upgrades.
   - debian/open-iscsi.links:
     + symlink open-iscsi init script in if-up.d and if-down.d directory so
       that the iscsi subsystem is started/stopped when network interfaces
       are configured.
   - debian/open-iscsi.postrm:
     + delete iscsi configuration when the package is purged.
   - utils/iscsi_discovery: De-bashify iscsi_discovery.
   - usr/idbm.c: Fix build failure with new glibc. LP: #299843.
  * Dropped:
   - Exit without error if /sys is not available. Otherwise, it's ...