FWIW, this new 100% reproducible case is 100% reproducible for a reason, which I will get to in a minute.
So, given that in two cases where I have had a stalled boot, signalling mountall with a USR1 has caused the boot to proceed. This seems like a race somewhere.
The reason I was able to reproduce this 100% of the time on one particular given platform is because it's a PXE (i.e. netboot/netroot) system, booting from the network and mounting it's entire root (which includes /usr and /var) from an NFS server. In this scenario I found that allowing ifup to configure an interface that has already been configured by the kernel during the netboot was causing the system to hang.
As a short-term solution until I could research the real, long-term solution, I decided that given that the interface was already up even before init was even started, I would just disable it in /etc/network/interfaces. That of course also prevents the "initctl emit -n net-device-up ..." in /etc/network/if-up.d/upstart from firing.
But it would only prevent it from firing for the ethernet interface. Shouldn't it still fire for lo being ifup'd, giving mountall the USR1 it needs?
FWIW, this new 100% reproducible case is 100% reproducible for a reason, which I will get to in a minute.
So, given that in two cases where I have had a stalled boot, signalling mountall with a USR1 has caused the boot to proceed. This seems like a race somewhere.
The reason I was able to reproduce this 100% of the time on one particular given platform is because it's a PXE (i.e. netboot/netroot) system, booting from the network and mounting it's entire root (which includes /usr and /var) from an NFS server. In this scenario I found that allowing ifup to configure an interface that has already been configured by the kernel during the netboot was causing the system to hang.
As a short-term solution until I could research the real, long-term solution, I decided that given that the interface was already up even before init was even started, I would just disable it in /etc/network/ interfaces. That of course also prevents the "initctl emit -n net-device-up ..." in /etc/network/ if-up.d/ upstart from firing.
But it would only prevent it from firing for the ethernet interface. Shouldn't it still fire for lo being ifup'd, giving mountall the USR1 it needs?