aiccu init.d script will race dhclient (upstart issue?)

Bug #223825 reported by floid
116
This bug affects 18 people
Affects Status Importance Assigned to Milestone
aiccu (Ubuntu)
Fix Released
Undecided
Unassigned
Nominated for Jaunty by Lars Düsing
Nominated for Lucid by Lars Düsing
Karmic
Won't Fix
Undecided
Unassigned

Bug Description

Binary package hint: aiccu

aiccu comes with a SysV-style init script. In the dark ages before upstart (and before NetworkManager?) this worked fine with a link from /etc/rc2.d/S99aiccu, but ever since, there's a race for DHCP users: aiccu won't sit around waiting if it can't reach the tunnel broker, and DHCP configuration happens late, after aiccu terminates.

Observe this snippet of syslog produced at boot (Hardy / 8.04):

Apr 26 22:43:40 badger NetworkManager: <info> Activation (eth1) Stage 1 of 5 (Device Prepare) complete.
Apr 26 22:43:40 badger NetworkManager: <info> Activation (eth1) Stage 2 of 5 (Device Configure) starting...
Apr 26 22:43:40 badger NetworkManager: <info> Activation (eth1) Stage 2 of 5 (Device Configure) successful.
Apr 26 22:43:40 badger NetworkManager: <info> Activation (eth1) Stage 3 of 5 (IP Configure Start) scheduled.
Apr 26 22:43:40 badger NetworkManager: <info> Activation (eth1) Stage 2 of 5 (Device Configure) complete.
Apr 26 22:43:40 badger NetworkManager: <info> Activation (eth1) Stage 3 of 5 (IP Configure Start) started...
Apr 26 22:43:41 badger anacron[6503]: Anacron 2.3 started on 2008-04-26
Apr 26 22:43:41 badger anacron[6503]: Normal exit (0 jobs run)
Apr 26 22:43:41 badger NetworkManager: <info> Activation (eth1) Beginning DHCP
transaction.
Apr 26 22:43:41 badger NetworkManager: <info> Activation (eth1) Stage 3 of 5 (IP Configure Start) complete.
Apr 26 22:43:41 badger NetworkManager: <info> DHCP daemon state is now 12 (succ
essfully started) for interface eth1
Apr 26 22:43:42 badger /usr/sbin/cron[6531]: (CRON) INFO (pidfile fd = 3)
Apr 26 22:43:42 badger /usr/sbin/cron[6534]: (CRON) STARTUP (fork ok)
Apr 26 22:43:42 badger /usr/sbin/cron[6534]: (CRON) INFO (Running @reboot jobs)
Apr 26 22:43:43 badger NetworkManager: <info> DHCP daemon state is now 1 (starting) for interface eth1
Apr 26 22:43:43 badger kernel: [ 51.267594] NET: Registered protocol family 17Apr 26 22:43:45 badger kernel: [ 52.114297] IPv6 over IPv4 tunneling driver
Apr 26 22:43:45 badger kernel: [ 52.114565] sit0: Disabled Privacy Extensions
Apr 26 22:43:45 badger kernel: [ 52.147518] tun: Universal TUN/TAP device driver, 1.6
Apr 26 22:43:45 badger kernel: [ 52.147523] tun: (C) 1999-2004 Max Krasnyansky <email address hidden>
Apr 26 22:43:45 badger aiccu: Couldn't resolve host tic.sixxs.net, service 3874
Apr 26 22:43:45 badger aiccu: Couldn't connect to the TIC server tic.sixxs.net
Apr 26 22:43:45 badger aiccu: Couldn't retrieve first tunnel for the above reason, aborting
Apr 26 22:43:46 badger kernel: [ 52.783534] hda-intel: Invalid position buffer, using LPIB read method instead.
Apr 26 22:43:47 badger dhclient: DHCPREQUEST of 172.16.1.33 on eth1 to 255.255.255.255 port 67
Apr 26 22:43:47 badger dhclient: DHCPACK of 172.16.1.33 from 172.16.0.1
Apr 26 22:43:47 badger avahi-daemon[5935]: Joining mDNS multicast group on inter
face eth1.IPv4 with address 172.16.1.33.
Apr 26 22:43:47 badger avahi-daemon[5935]: New relevant interface eth1.IPv4 for mDNS.
Apr 26 22:43:47 badger avahi-daemon[5935]: Registering new address record for 172.16.1.33 on eth1.IPv4.
Apr 26 22:43:47 badger NetworkManager: <info> DHCP daemon state is now 4 (reboot) for interface eth1

and aiccu.conf:

protocol tic
server tic.sixxs.net
username XXXXXXXXX
password XXXXXXXX
ipv6_interface sixxs
tunnel_id TXXXXX
verbose true
daemonize true
automatic true
requiretls false

When I have a moment I'll figure out an upstart script for it, this is just to document the issue.

Tags: upstart

Related branches

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Same issue here.
Workaround:
disable aiccu rc?.d-links: "update-rc.d -f aiccu remove" (do NOT delete /etc/init.d/aiccu)
create a shell script /etc/network/if-up.d/aiccu:
--
#!/bin/bash
/etc/init.d/aiccu start
--
chmod a+x /etc/network/if-up.d/aiccu

the same for /etc/network/if-down.d/aiccu - just replace "aiccu start" by "aiccu stop"

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Next problems I run in are: radvd, setting an IPv6 address to eth0 (radvd needs an IPv6 address within your net, not only the tunnel endpoint!)

Revision history for this message
Sebastien CELLES (s-cls) wrote :

I confirm that aiccu doesn't start when I reboot.
I hope it will soon work "out of the box" because IPv6 will become
widely used in the next years.

Revision history for this message
Philipp Kern (pkern) wrote : Re: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

On Fri, Aug 22, 2008 at 08:10:40AM -0000, Sebastien CELLES wrote:
> I confirm that aiccu doesn't start when I reboot.
> I hope it will soon work "out of the box" because IPv6 will become
> widely used in the next years.

Not with tunnel providers I hope, but natively...

Kind regards,
Philipp Kern

Revision history for this message
Gioele Barabucci (gioele) wrote :

Race still exists with Ubuntu 8.10 Intrepid.

The workaround proposed by Lars Düsing does not work anymore. Changes to the if-{up,down} infrastructure?

Revision history for this message
Steffan Jacobs (steffanjacobs) wrote :

I just tried the workaround and it does still work for me (I purged and reinstalled first).

If this is a solution, is this going to end up in an update or at least jaunty package?

Revision history for this message
Lars Düsing (lars.duesing) wrote :

I would like to see this done, too.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

it IS confirmed

Changed in aiccu (Ubuntu):
status: New → Confirmed
Revision history for this message
Michael Schnupp (michas) wrote :

Is there a general way to start aiccu whenever "internet" becomes available?
Where "internet" might be available at startup, or whenever someone plugs in the ethernet cable later, or when the user logs in and network-manager configures the wireless.

Is there any mechanism like udev, using HAL for tracking the status of "internet", or something like this?

Revision history for this message
Lars Düsing (lars.duesing) wrote :

I'm not aware of any tests if "internet" becomes available. But when network becomes available.
See my first post - /etc/network/if-up.d/*

Revision history for this message
Jeroen Massar (massar) wrote :

Given motivation (and no nasty crap coming my way) I'll try to finish up the all brand new AICCU this weekend.
This one will have a GUI on all platforms and will also resolve a lot of other tiny tidbits, including 're-connect'. One can then just run the daemon, and it will make sure that connectivity works, if connectivity is not there (yet) it will just retry a bit later to get it up and running. Testing seems to have proven that it all more or less works, but I need to test a couple of other platforms to get everything running fine before actually bringing it out to the public (and then getting flooded with mails that it does not work.... especially the "nice comments" in there always give one sooo much motivation to solve other peoples problems that those cases are not worth it). Lets see what the weekend brings ;)

Revision history for this message
Lars Düsing (lars.duesing) wrote : Re: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

Oh, thank you!
If you need testers, Drop a line

>

Revision history for this message
Gioele Barabucci (gioele) wrote :

On modern systems, you can use DBus to inquiry NetworkManager whether there is an Internet connection, or something that resembles that. Firefox asks NetworkManager the same question and updates the "Work offline" flag accordingly.

Revision history for this message
Lars Düsing (lars.duesing) wrote : AW: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

Oh ok... Never had a look into that.
But as there is a new version the days, I hope it is already irrelevant.

Revision history for this message
exactt (giesbert) wrote :

the problem still exists on karmic. regarding the switch to upstart: is the work-around still valid?

Revision history for this message
latinhawwk (latinhawk) wrote :

I can confirm that there has been no change on this. No new release to fix the problem yet.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

I will change this.
Anybody with any help on how to package ubuntu-debs?
If nobody maintains this package, I will help out, and take maintainship over. Any suggestions?

Changed in aiccu (Ubuntu):
assignee: nobody → Lars Düsing (lars.duesing)
Revision history for this message
Lars Düsing (lars.duesing) wrote :

exactt: the workaround is still vaild.

Revision history for this message
Gioele Barabucci (gioele) wrote :

How to make an Ubuntu package <https://wiki.ubuntu.com/PackagingGuide/Complete>. It is long but not hard at all.
You will also need to understand the new Upstart infrastructure <http://upstart.ubuntu.com/getting-started.html> (used on Ubuntu and Fedora).

Thank you for undertaking this task.

Revision history for this message
Lars Düsing (lars.duesing) wrote : RE: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

Thanks, Gioele. This upstart-thing was my preferred idea. That seems really the best idea.
I'll do my best the next days. :)

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Gioele Barabucci
Sent: Samstag, 31. Oktober 2009 10:16
To: <email address hidden>
Subject: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

How to make an Ubuntu package <https://wiki.ubuntu.com/PackagingGuide/Complete>. It is long but not hard at all.
You will also need to understand the new Upstart infrastructure <http://upstart.ubuntu.com/getting-started.html> (used on Ubuntu and Fedora).

Thank you for undertaking this task.

--
aiccu init.d script will race dhclient (upstart issue?)
https://bugs.launchpad.net/bugs/223825
You received this bug notification because you are a direct subscriber of the bug.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

ok. got a patch :-)
It works mostly. Only one caveat:
--
There seems to be a bug in upstart. If you stop networking after it had started, and restart after that aiccu WON'T be loaded again. Don't know why. Any hints?
--

Revision history for this message
Lars Düsing (lars.duesing) wrote :
Changed in aiccu (Ubuntu):
assignee: Lars Düsing (lars.duesing) → nobody
Revision history for this message
Lars Düsing (lars.duesing) wrote :

(There is an active ppa on https://launchpad.net/~lars.duesing/+archive/aiccu for such won't wait any more :) )

Revision history for this message
Lars Düsing (lars.duesing) wrote :

thanks <email address hidden> :)
corrected patch

Revision history for this message
John Dong (jdong) wrote :

ACK from MOTU-SRU for karmic-proposed debdiff in comment #24.

However, please reformat the original bug description in accordance to https://wiki.ubuntu.com/StableReleaseUpdates

Revision history for this message
Lars Düsing (lars.duesing) wrote :

1. aiccu won't work for any dhcp-using and/or wlan users, because aiccu will not wait until networking is up.
2. Bug is addressed by using upstart "on net-device-up" / "on net-device-down"-handlers.
3. patch is in comment #24
4. TEST CASE: install aiccu on a system which uses dhcp to get IP-adresses. aiccu runs. Reboot. aiccu starts, won't find tunnel-broker and stops then.
5. I'm not aware on any regressions. Except maybe if a user gets ubuntu running without upstart. There is a problem as described in comment #21. But this is in my opinion a rare case.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Is that ok, John Dong?
As far as I know I cannot upload anything to *-proposed. Who will do it then?

Revision history for this message
Lars Düsing (lars.duesing) wrote :

uh oh... found 2 small errors in my patch... here is a new version of the patch.
(Changed ${NAME} to aiccu, and added one more error logger-line. And corrected changelog :) )

sorry about that

Revision history for this message
exactt (giesbert) wrote :

hey lars,

thx for your work. works great so far. would you mind updating your PPA with the latest patches applied?

thx

Revision history for this message
Lars Düsing (lars.duesing) wrote :

why has maintainer switched back? *sorry*

Revision history for this message
Lars Düsing (lars.duesing) wrote :

ppa build is work in progress: https://edge.launchpad.net/~lars.duesing/+archive/aiccu/+builds?build_state=pending
(be warned, the versioning is a little bit weird, in ppa is 1.2, in karmic-proposed it will be 0.1 - I'm still learning, SORRY)

Revision history for this message
exactt (giesbert) wrote :

i just tried to upgrade to the 1.2 version. but dpkg hangs on install:

$sudo dpkg --install aiccu_20070115-9ubuntu1.2_amd64.deb
(Lese Datenbank ... 133437 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereiten zum Ersetzen von aiccu 20070115-9ubuntu1.2 (durch aiccu_20070115-9ubuntu1.2_amd64.deb) ...
stop: Job has already been stopped: aiccu
Entpacke Ersatz für aiccu ...
Richte aiccu ein (20070115-9ubuntu1.2) ...
Warning: Couldn't find global Tunnel Brokers List, please check your DNS settings and read the FAQ.
Warning: Couldn't find global Tunnel Brokers List, please check your DNS settings and read the FAQ.

(sorry, it's in german)

here are the last lines of /var/log/dpkg.log :

2009-11-05 00:57:53 startup archives install
2009-11-05 00:57:58 upgrade aiccu 20070115-9ubuntu1 20070115-9ubuntu1.2
2009-11-05 00:57:58 status half-configured aiccu 20070115-9ubuntu1
2009-11-05 01:04:20 startup packages configure
2009-11-05 01:05:31 startup archives install
2009-11-05 01:05:31 upgrade aiccu 20070115-9ubuntu1 20070115-9ubuntu1.2
2009-11-05 01:05:31 status half-configured aiccu 20070115-9ubuntu1
2009-11-05 01:05:31 status unpacked aiccu 20070115-9ubuntu1
2009-11-05 01:05:31 status half-installed aiccu 20070115-9ubuntu1
2009-11-05 01:05:31 status triggers-pending sreadahead 1.0-5
2009-11-05 01:05:31 status half-installed aiccu 20070115-9ubuntu1
2009-11-05 01:05:31 status triggers-pending sreadahead 1.0-5
2009-11-05 01:05:31 status triggers-pending man-db 2.5.6-2
2009-11-05 01:05:32 status half-installed aiccu 20070115-9ubuntu1
2009-11-05 01:05:32 status half-installed aiccu 20070115-9ubuntu1
2009-11-05 01:05:32 status unpacked aiccu 20070115-9ubuntu1.2
2009-11-05 01:05:32 status unpacked aiccu 20070115-9ubuntu1.2
2009-11-05 01:05:32 configure aiccu 20070115-9ubuntu1.2 20070115-9ubuntu1.2
2009-11-05 01:05:32 status unpacked aiccu 20070115-9ubuntu1.2
2009-11-05 01:05:32 status unpacked aiccu 20070115-9ubuntu1.2
2009-11-05 01:05:32 status unpacked aiccu 20070115-9ubuntu1.2
2009-11-05 01:05:32 status half-configured aiccu 20070115-9ubuntu1.2
2009-11-05 01:06:44 trigproc sreadahead 1.0-5 1.0-5
2009-11-05 01:06:44 status half-configured sreadahead 1.0-5
2009-11-05 01:06:44 status installed sreadahead 1.0-5
2009-11-05 01:06:44 trigproc man-db 2.5.6-2 2.5.6-2
2009-11-05 01:06:44 status half-configured man-db 2.5.6-2
2009-11-05 01:06:44 status installed man-db 2.5.6-2
2009-11-05 01:07:21 startup packages remove
2009-11-05 01:07:21 status half-configured aiccu 20070115-9ubuntu1.2
2009-11-05 01:07:22 remove aiccu 20070115-9ubuntu1.2 20070115-9ubuntu1.2
2009-11-05 01:07:22 status half-configured aiccu 20070115-9ubuntu1.2
2009-11-05 01:09:23 startup packages purge
2009-11-05 01:09:23 status half-configured aiccu 20070115-9ubuntu1.2
2009-11-05 01:09:23 remove aiccu 20070115-9ubuntu1.2 20070115-9ubuntu1.2
2009-11-05 01:09:23 status half-configured aiccu 20070115-9ubuntu1.2
2009-11-05 01:10:37 startup archives install
2009-11-05 01:10:37 upgrade aiccu 20070115-9ubuntu1.2 20070115-9ubuntu1.2
2009-11-05 01:10:37 status half-configured aiccu 20070115-9ubuntu1.2

anything else i can provide?

Revision history for this message
latinhawwk (latinhawk) wrote : Re: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

Lars Düsing wrote:
> Is that ok, John Dong?
> As far as I know I cannot upload anything to *-proposed. Who will do it then?
>
>
The original patch you placed on your ppa was working fine for me.
However, last night an upgrade was listed from your site that Karmic is
not accepting and is causing problems. Please take a look at it. Thanks.

Joe

Revision history for this message
Lars Düsing (lars.duesing) wrote :

I see the problem.
Problem is: My script stops only aiccu if networking is down. If you upgrade aiccu it won't stop because networking is still up.
I think i was too clever :)
Can you confirm that aiccu will start on next reboot?

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Ok, after some reading through source-code of upstart, browsing through upstart-wiki, and countless reboots , I found the problem:

Nov 6 22:14:12 artus init: aiccu main process (1691) became new process (1702)
Nov 6 22:14:12 artus aiccu: AICCU running as PID 1747

Nov 6 22:19:30 artus init: Handling net-device-down event
Nov 6 22:19:30 artus init: aiccu goal changed from start to stop
Nov 6 22:19:30 artus init: aiccu state changed from running to pre-stop
Nov 6 22:19:30 artus init: aiccu state changed from pre-stop to stopping
Nov 6 22:19:30 artus init: Handling stopping event
Nov 6 22:19:30 artus init: aiccu state changed from stopping to killed
Nov 6 22:19:30 artus init: Sending TERM signal to aiccu main process (1702)

upstart wants to kill a process that is not there anymore at all.
(and this is also the problem, why aiccu won't be reloaded any more... ->#21 MY FAULT :) )

so please wait a little bit, until I got over that. (and sorry for not answering sooner, I happen to have a flu [no, not that one!])

Revision history for this message
latinhawwk (latinhawk) wrote :

Lars Düsing wrote:
> I see the problem.
> Problem is: My script stops only aiccu if networking is down. If you upgrade aiccu it won't stop because networking is still up.
> I think i was too clever :)
> Can you confirm that aiccu will start on next reboot?
>
>
There is some kind of bug on it. synaptic can't complete installation
and it stall. It's necessary to force synaptic to close and the
transaction does not complete.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Thanks latinhawwk.
After looking a little bit deeper in it, I (hopefully) found the deeper problem. It is not my script, but upstart itself:
https://bugs.edge.launchpad.net/upstart/+bug/406397

That's where I have to push the brakes, and stop trying to get it in karmic, as long this bug is not done. :(

!!!!!!!!!!!!!!!!!!!!!!!!!!!
I will put a new version on my ppa, but until this bug in upstart is done, you will have trouble on anything that will stop aiccu. The only solution I have found is a reboot before trying to start aiccu again!
!!!!!!!!!!!!!!!!!!!!!!!!!!!

Sorry about that.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Minor Bugfix:
Test if script is run by net-device-down _and_ if internet connectivity still works - if yes, stop stopping aiccu.

Changed in aiccu (Ubuntu Karmic):
status: New → Incomplete
Changed in aiccu (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Lars Düsing (lars.duesing) wrote :

I put the bug on "incomplete" because it won't fix until LP: #406397 is fixed.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

New version is ready in my ppa: http://launchpad.net/~lars.duesing/+archive/aiccu.

!!!!! BE CAREFUL !!!!!
- Versioning there is a complete mess, it's my first ppa... Version in there is 20070115-9ubuntu1.3, when this patch is put into karmic-proposed or such it will be a version LESS than there (most likely 20070115-9ubuntu0.1)
- because of the bug in upstart itself I described in comment #37 it will not update properly, so you have to do it the following way:
  1. remove old aiccu ("sudo dpkg -r aiccu") - do NOT use purge ("dpkg -P") as your login information will be lost!
  2. reboot
  3. install the current version via apt-get/aptitude/dselect/synaptic

Thanks for all help!

Revision history for this message
exactt (giesbert) wrote :

hi lars,

thx for your efforts!

i have problems with your upgrade instructions. "sudo dpkg -r aiccu" hangs and after reboot i tried upgrading which hangs as well. any other instructions for a proper upgrade?

Revision history for this message
Lars Düsing (lars.duesing) wrote :

hi exacct.
Ouch. It worked with me, because of my special debug configuration of upstart (self patch). I forgot that.

1. sudo rm /etc/init/aiccu.conf
2. sudo touch /etc/init/aiccu.conf
3. reboot
4. sudo dpkg -r aiccu
5. sudo aptitude install aiccu

(tested on a newly installed virtual machine :) )

that's a real horror :(

Revision history for this message
Lars Düsing (lars.duesing) wrote :

I don't want this bug to expire....

Changed in aiccu (Ubuntu):
status: Incomplete → In Progress
Changed in aiccu (Ubuntu Karmic):
status: Incomplete → In Progress
Revision history for this message
exactt (giesbert) wrote :

hi lars,

just for the record. step 4 gave:

~$ LC_NUMERIC=en_US sudo dpkg -r aiccu
dpkg: error processing aiccu (--remove):
 Package is in a very bad inconsistent state - you should
 reinstall it before attempting a removal.
Errors were encountered while processing:
 aiccu

so i just skipped it and installed -1.3. works now. thx

Revision history for this message
latinhawwk (latinhawk) wrote :

exactt wrote:
> hi lars,
>
> just for the record. step 4 gave:
>
> ~$ LC_NUMERIC=en_US sudo dpkg -r aiccu
> dpkg: error processing aiccu (--remove):
> Package is in a very bad inconsistent state - you should
> reinstall it before attempting a removal.
> Errors were encountered while processing:
> aiccu
>
> so i just skipped it and installed -1.3. works now. thx
>
>
Where are you getting 1.3 from? Lars' PPA goes as high as 1.2. Thanks.

Revision history for this message
exactt (giesbert) wrote :

hi latinhawk,

got it from the PPA under https://launchpad.net/~lars.duesing/+archive/aiccu/+packages .

no idea why you can't get 1.3 there. i do.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

latinhawwk - it is 1.3. Maybe "sudo aptitude update" will help?

http://launchpad.net/~lars.duesing/+archive/aiccu states 1.3 is current.

Giesbert I think that is due to the blank aiccu.conf.
And sure... you can skip the remove step... Late working is bad. :)

Revision history for this message
latinhawwk (latinhawk) wrote :

Lars Düsing wrote:
> latinhawwk - it is 1.3. Maybe "sudo aptitude update" will help?
>
> http://launchpad.net/~lars.duesing/+archive/aiccu states 1.3 is current.
>
> Giesbert I think that is due to the blank aiccu.conf.
> And sure... you can skip the remove step... Late working is bad. :)
>
>
I was able to see it but it's not completing the installation. Will
cancel it and reboot to see how it goes.

Revision history for this message
latinhawwk (latinhawk) wrote :

Lars Düsing wrote:
> latinhawwk - it is 1.3. Maybe "sudo aptitude update" will help?
>
> http://launchpad.net/~lars.duesing/+archive/aiccu states 1.3 is current.
>
> Giesbert I think that is due to the blank aiccu.conf.
> And sure... you can skip the remove step... Late working is bad. :)
>
>
After several attemtps at getting 1.2 removed and always stalling I
acted as if it had completed restarted my machine and installed 1.3.
After one more reboot 1.3 is up and running. Thanks Lars.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

latinhawwk: please read https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825/comments/42
(you can leave out step 4)

Revision history for this message
Fabrice Coutadeur (fabricesp) wrote :

Hi Lars,

If I understand well your comment #39 ( https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825/comments/39 ), there is no complete fix right now. If it's the case, I'll unsubscribe u-u-s. Please subscribe again u-u-s when the fix is ready to be sponsored.

Thanks,
Fabrice

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Hi Fabrice,
There is no complete fix for that now, yes. My fix is better than the current version, but with the regression that you cannot unload or reload aiccu via the normal "service aiccu stop/restart". A fix for that would be https://bugs.edge.launchpad.net/ubuntu/+source/aiccu/+bug/223825/comments/1, but I'm not sure if this is within the guidlines of ubuntu (running a service ONLY within /etc/networks/if-up.d/ ). If this is, I would commit a patch based on that, and put it again on u-u-s.

Thanks,
Lars

Revision history for this message
Joke de Buhr (joke) wrote :

There are several packages which restart their service from within /etc/networks/if-up.d/. The ntpdate package or the openssh-server package.

Revision history for this message
Łukasz Jernaś (deejay1) wrote :

How's work on this going on?

Revision history for this message
Lars Düsing (lars.duesing) wrote : AW: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

Sorry, only usable work-around is so far: https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825/comments/1

This will be until someone fixes https://bugs.launchpad.net/upstart/+bug/406397 - which is marked invalid by now.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

One potential work-around is disabling daemonization of aiccu in its configuration file ("daemonize false") and skipping Upstart's expect fork altogether. aiccu (rather unfortunately) uses the daemonize flag to determine its logging destination, but we can capture its output and log to a file like so:

exec $BIN start $DAEMON_ARGS >> /tmp/aiccu.log 2>&1

Certainly not as flexible as the syslog, but good enough for many cases.

Anyone have contact with the aiccu devs? It might be simpler to request that they give users more control over aiccu's logging mechanisms instead of waiting for the Upstart devs to figure out a solution to the forking issue.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Attached is a basic Upstart job definition that utilizes the non-daemonize work-around. To use it, place the attached file in /etc/init/ and set the "daemonize" option to false in /etc/aiccu.conf

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Attached is an updated work-around script that emits net-device-up/net-device-down events.

Revision history for this message
Jeroen Massar (massar) wrote :

> Attached is an updated work-around script that emits net-device-up/net-device-down events.

I am really wondering if you know what the function of AICCU is, because if you did you would not be stopping it when the network goes down and starting it automatically when the network goes up.

Also obviously you did not read the README file included in the original source distribution which contains this little piece of text:

8<----------------------------------------------------------------------------------------------------
WARNING: never run AICCU from DaemonTools or a similar automated
'restart' tool/script. When AICCU does not start, it has a reason
not to start which it gives on either the stdout or in the (sys)log
file. The TIC server *will* automatically disable accounts which
are detected to run in this mode. Use 'verbose true' to see more
information which is especially handy when starting fails.
----------------------------------------------------------------------------------------------------->8

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

I would be interested to know what function of AICCU does *not* involve the presence of a network connection. I cannot find any documentation that indicates AICCU has any purpose other than setting up and tearing down AYIYA tunnels. There is no need for such a tool to run if there is no network connection, unless the tool also monitors network connectivity.

At least on my Natty system, the aiccu daemon simply terminates when no network connection is present, so it does not appear to have any ability to monitor the status of the system's network connectivity (and it probably shouldn't have that ability--that's really the system administrator's concern).

So again, I would be interested to know what function of AICCU makes it useful without a network connection.

I have in fact read the warning you mention. I'd be interested to know how it applies to my Upstart script and not the one on which Lars Dursig has been working on. My script does not have a respawn clause, so assuming AICCU behaves correctly when it's run in the foreground, it should preserve AICCU's behavior of exiting if there is a connection problem. I can confirm the script terminates if launched manually when no network connection is present.

Revision history for this message
Lars Düsing (lars.duesing) wrote : Re: [Bug 223825] Re: aiccu init.d script will race dhclient (upstart issue?)

This clause refers to auto-respawn-scripts. There is only one
possibility for such to happen in our scripts: if network itself goes
mad. And then even the aiccu-daemon itself has the mentioned troubles.
So if nobody complains, I will build a new package and will then
promote it upstream.

Thanks a lot!

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Jeroen, his upstart job does not use the 'respawn' keyword, so it will not be "automatically restarted" like daemontools does. This is only started when there is a real network interface (something I would think necessary for this daemon to work!), and stopped when the system is going down.

Despot, a few tweaks:

* Do not include the package description in the upstart job as a comment at the top. Just delete that.
* Explain your reasons for overriding the system PATH.
* Add emits stanzas for net-device-up and net-device-down

Otherwise this looks pretty good.

Changed in aiccu (Ubuntu Karmic):
status: In Progress → Won't Fix
tags: added: upstart
Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

No important reason (that I know of) for the PATH override; it was a copy/paste from another Upstart script, which was an adaptation of a System V init script. Per http://netsplit.com/2008/04/16/upstart-05-job-environment/, it shouldn't be necessary, so I removed it.

I've made the tweaks Clint recommended, the revised script is attached.

Revision history for this message
Jeroen Massar (massar) wrote :
Download full text (3.6 KiB)

> Jeroen, his upstart job does not use the 'respawn' keyword, so it will not be "automatically restarted" like daemontools does.
> This is only started when there is a real network interface (something I would think necessary for this daemon to work!),
> and stopped when the system is going down.

I agree that bringing the daemon up when the network goes up is fine, and as there is a PID file it will only run once and not start again when the interface is brought up again.

Stopping it when the network goes down though will mean that when the network is flipping (which we have seen at several people already) you will be effectively start/stopping the daemon all the time.

From another reply:
> I cannot find any documentation that indicates AICCU has any purpose other than setting up and tearing down AYIYA tunnels.

You might want to start at reading the wikipedia article which already tells you a lot more than that. It does TIC first to retrieve the tunnel configuration, then it can set up a static (which is quite useless on debian due to the great interfaces(5) :) and of course then run a heartbeat or do AYIYA.

> There is no need for such a tool to run if there is no network connection, unless the tool also monitors network connectivity.

The whole point of heartbeat and AYIYA is handling dynamic tunnels.

> At least on my Natty system, the aiccu daemon simply terminates when no network connection is present,

It exits because it is unable to retrieve the configuration. See the log file for a nice message.

> so it does not appear to have any ability to monitor the status of the system's network connectivity

When it is started and it can connect to the TIC server and retrieve it's configuration, then it will nicely inform the PoP of network connectivity state using either the heartbeat or the AYIYA protocol.

> (and it probably shouldn't have that ability--that's really the system administrator's concern).

People who are using AICCU don't care about this, they just want working connectivity and that is what it does.

> So again, I would be interested to know what function of AICCU makes it useful without a network connection.

Without a network connection (and with that an "Internet connection" not a local network) there is not much it can do, that is why it then also exits. When you though start it when you have working connectivity you can keep it running, no need to exit it, and it won't exit then by itself either.

> I have in fact read the warning you mention. I'd be interested to know how it applies to my Upstart script and not the one on which Lars Dursig has been working on.

See the above, you are also disabling it when the network goes down, as such, if the interface flips, it will
Note also that there is no clause anywhere that when that interface goes up/down is actually an interface that can do anything with internet connectivity. Maybe at least a notion of a check that there is a default route would be sane to have?

> Jeroen, his upstart job does not use the 'respawn' keyword, so it will not be "automatically restarted" like daemontools does.

As Lars said also, it will seem to automatically restart when the network goes ...

Read more...

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Excerpts from Despot's message of Wed Jul 13 01:19:28 UTC 2011:
> No important reason (that I know of) for the PATH override; it was a
> copy/paste from another Upstart script, which was an adaptation of a
> System V init script. Per http://netsplit.com/2008/04/16/upstart-05-job-
> environment/, it shouldn't be necessary, so I removed it.
>
> I've made the tweaks Clint recommended, the revised script is attached.
>
>
> ** Attachment added: "Upstart script for aiccu rev3"
> https://bugs.launchpad.net/ubuntu/+source/aiccu/+bug/223825/+attachment/2204754/+files/aiccu.conf

Cool, well done.

Looking a second time there is another problem. You stop on runlevel
[^2345] but don't start again on a transition from runlevel 1 -> 2. This
is a common mistake that is present in a lot of upstart jobs.

start on (local-filesystems and net-device-up IFACE!=lo) or runlevel
[2345]

Should take care of that.

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Excerpts from Jeroen Massar's message of Wed Jul 13 10:53:00 UTC 2011:
> > Jeroen, his upstart job does not use the 'respawn' keyword, so it will not be "automatically restarted" like daemontools does.
> > This is only started when there is a real network interface (something I would think necessary for this daemon to work!),
> > and stopped when the system is going down.
>
> I agree that bringing the daemon up when the network goes up is fine,
> and as there is a PID file it will only run once and not start again
> when the interface is brought up again.
>
> Stopping it when the network goes down though will mean that when the
> network is flipping (which we have seen at several people already) you
> will be effectively start/stopping the daemon all the time.

It does no such thing.

It stops on runlevel [^2345].. which basically means any time the system
goes into a 'rebooting', 'shutting down' or 'single user maintenance
mode' state.

Also local-filesystems only happens during bootup, so it will effectively
only be started automatically at bootup.

Revision history for this message
Jeroen Massar (massar) wrote :

Clint: you are mixing two "proposals", there is a) one for upstart, b) the network-based one

The upstart one indeed does not have that issue.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Jeroen,

Good to know this information. I'm puzzled by AICCU's behavior of shutting down on startup if it can't retrieve TIC information, since I understand you to be saying that the program won't terminate if there's an active tunnel and the computer loses connectivity (e.g. the person leaves the range of their WiFi hotspot). What is the difference, from AICCU's perspective, between having no connectivity on startup, and losing connectivity during a program run?

At any rate, the Upstart script isn't written to terminate AICCU when a network interface is taken down. The termination conditions are entering runlevels 0 or 6 (shutdown or reboot, respectively).

I'm open to suggestions on a Internet connectivity check. Currently the only sanity check is a check to make sure the interface is not the local loopback. I don't think having a default route is a very strong guarantee of connectivity, since it's quite possible for that default route to be offline. The only strong guarantee I can imagine is pinging a PoP server, but isn't that the general purpose of the heartbeat protocol? Seems like it'd be a re-invention of the wheel.

Clint,

Thanks for the recommendation. I'm not sure about this change, though, because we don't want to start the daemon unless there's a (non-loopback) network connection present. Unless I'm badly mistaken, there's no guarantee that a network connection will be present when the system switches to runlevel 2. If so, there's a good possibility that AICCU would simply terminate on startup, as Jeroen mentions. Not the end of the world, but certainly not necessary either.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Further refinements: while testing, I discovered that the AICCU daemon terminates when the system hibernates. This is not an Upstart behavior as far as I can tell, because removing the "stop on" clause does not change the behavior. The AICCU log simply says "Killed", so I suspect it's responding to a low-level SIGTERM signal or such.

No big deal, except neither the "local-filesystems" or the "runlevel" events are emitted by Upstart when the system resumes. As a result, rev 3 of the script will not restart the AICCU daemon when the system comes out of hibernation. Clint's "or runlevel [2345]" recommendation doesn't aid us either.

This seems like a fairly common scenario (I find AICCU to be most useful on my laptop). I'm unable to find any documentation on what sort of signals are emitted by Upstart when a system comes out of hibernation, so I've resorted to this simplified "start on" clause: "start on net-device-up IFACE!=lo".

Unfortunately, this start condition leaves us without a strong guarantee that local filesystems are available. So, I added a sanity check to the pre-start script that will bail out if the log file is untouchable.

This is the only way I know of to restart the daemon when the system comes out of hibernation. I'm wide open to suggestions on better way to handle the situation.

Revised script is attached.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

One more adjustment (hopefully the last). Covers the case where the log file exists but is not writable in the sanity checks.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Bah, left a temporary path in the LOGFILE variable. My kingdom for comment deletion.

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Lars, any plans to update the PPA? I'd be happy to help test.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Despot, I waited for the comments on your patch to settle.

new version is pending in my ppa ( https://launchpad.net/~lars.duesing/+archive/aiccu/ ) - it should be active within the next 24h:

aiccu (20070115-14ubuntu1ppa1) natty; urgency=low

  * Replace sysv-startup-script with upstart-script courtesy of
    Caleb Callaway <email address hidden>
 -- Lars Duesing <email address hidden> Mon, 18 Jul 2011 10:48:19 +0200

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Hi Lars,

Good idea to wait for the dust to settle. I was thinking that three days was enough time to wait, given the speed with which issues have been noted and addressed (usually within 24 hours).

Thanks for updating the PPA. I went ahead and installed it in a VM, and the new script installs as expected--huzzah! However, I see the /etc/aiccu.conf file still reads "daemonize true" For the Upstart script to work as intended, this will have to be changed to "daemonize false".

Revision history for this message
Lars Düsing (lars.duesing) wrote :

Ouch. Yes. You're right. PPA2 is out...

Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

Cool--PPA2 installs and runs nicely.

Revision history for this message
Jan Henke (jhe) wrote :

If your change to the startup script works, could please provide the patch for the package in the normal distribution?

Adding a ppa only to make this package working properly is surly not the right way. Fixing the problem in Ubuntus repositories is.
Thank you.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

PPA lives since more than half a year with no regressions.

Changed in aiccu (Ubuntu):
status: In Progress → Fix Released
Changed in aiccu (Ubuntu):
status: Fix Released → Triaged
Revision history for this message
Jan Henke (jhe) wrote :

The point is, a ppa providing a fixed version is no fix for the original packet. If your ppas version fixes this bug, the diff should be applied to the package in the ubuntu repository. Installing a ppa just to get this properly working is *not* right.

Changed in aiccu (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Caleb Callaway (enlightened-despot) wrote :

I'm hesitant to call the no-daemonize Upstart script a "fix" because it doesn't actually address the underlying problem. It seems like more of a work-around to me.

However, I think Jan's concern is very valid. Pushing the changes into the main repository might be the best option at this point: I've seen nothing that indicates the forking problem in Upstart is going to be resolved soon, and there have been no recent updates to AICCU either.

Revision history for this message
Lars Düsing (lars.duesing) wrote :

So, a full patch.
Sorry for confusion...

Revision history for this message
Lars Düsing (lars.duesing) wrote :

sorry... still working, some advancements to be done, new patch is in work...

Revision history for this message
Lars Düsing (lars.duesing) wrote :

So, patch should be fine now,
thanks to pitti and infinity @ #ubuntu-devel :-)

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package aiccu - 20070115-14.1ubuntu1

---------------
aiccu (20070115-14.1ubuntu1) precise; urgency=low

  * Fix race between DHCP setup and aiccu start by using an event-triggered
    upstart script instead of init.d (LP: #223825)
 -- Lars Duesing <email address hidden> Wed, 29 Feb 2012 08:41:59 +0100

Changed in aiccu (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote :

I've fixed a couple of things in the debdiff and uploaded it. Specifically:
*) I mentioned what the purpose in transitioning to an upstart script was in the changelog
*) I retargetted the changelog entry at Precise, the current development release
*) I dropped the debian/aiccu.maintscript addition; dh_installinit handles the removal of /etc/init.d/aiccu itself.

Thanks for your work!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.