Comment 36 for bug 790538

On Wed, Jun 01, 2011 at 05:11:15PM -0000, Peter Odding wrote:
> I appreciate that the problem has already been solved (I'm glad) but am
> still considering whether to nominate this as the biggest fuck-up since
> the Debian SSH/SSL key fiasco.

This is a bigger fuq-up than the SSH key fiasco. The SSH key fiasco
did not actually break anything, and it also had nothing to do with
Ubuntu, as such.

> If this truly affects every server Ubuntu install out there using
> unattended upgrades (it affected my three servers running Ubuntu
> 10.04 with unattended upgrades), this bug just broke thousands of
> backup cron jobs all over the world and the worst thing is it
> requires manual intervention from a sysadmin to get it going again
> :-S.

It it worse than you are describing. Many servers, mine included, run
everything, application-wise, from cron jobs.

I personally run five distinct categories of things:

1) My application stuff that makes money for me
2) System updates
3) Reboots
4) Backups
5) Problem detection and notification

from cron jobs. When cron stopped working, my apps would not run, the
automatic updates stopped, reboots stopped, and my notification system
stopped as well. So, problems that are self repairing normally, would
not self repair.

What this means is that not only my stuff stopped working, but also
self repair stopped, and I did not know about this problem since my
notification and problem detection system ceased to work, as well.

IOW, I was completely and totally screwed.

The only thing where I lucked out, is that my main webserver that
serves algebra.com, only does system updates on Sundays. If that one
went down, I would have lost hundreds of $$$.

I am aware that "stuff happens", and I appreciate a quick fix, but
this was a truly momentous error.

i