Eucalyptus cloud controller stops working suddenly

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage. I have classified this bug as a bug in eucalyptus

When reporting bugs in the future please use apport, either via the appropriate application's "Help -> Report a Problem" menu or using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Oops.. sorry for missing the package.

Btw, this was with 1.6~bzr645-0ubuntu2.

I confirm I had exactly the same problem happen to me last week.

When looking at /var/log/eucalyptus/cloud-error.log, there was some verbiage about a corrupted index or somesuch. Do you see something similar?

Also, is it possible that you started the eucalyptus-cloud service from within an SSH session that suffer bug #407428?

Potentially some times, but signals seems to be working fine now, and it happened to me today again.

I'm attaching my cloud-ouput.log file. grep -i "corrupt" there doesn't find anything, but there are several other errors.

I am getting a similar error though it only happens for me when I actually try to run an instance. Here is the exact command line and error message.

$ euca-run-instances emi-67DC1321 -k mykey
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

Here is the only error message that is produced when I try to run the instance. It is from cloud-output.log

com.eucalyptus.ws.AuthenticationException: Missing required parameter: AWSAccessKeyId
        at com.eucalyptus.ws.handlers.HmacV2Handler.incomingMessage(HmacV2Handler.java:110)
        at com.eucalyptus.ws.handlers.MessageStackHandler.handleUpstream(MessageStackHandler.java:115)
        at com.eucalyptus.ws.server.FilteredPipeline$StageBottomHandler.handleUpstream(FilteredPipeline.java:171)
        at com.eucalyptus.ws.server.NioServerHandler.messageReceived(NioServerHandler.java:119)
        at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:114)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:385)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:459)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:443)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:381)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:342)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:329)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:330)
        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:203)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)

Trevor,

Can you have a look at bug #430093, and see if it is the same bug that you get? If yes, could you add a comment with relevant details (such as the above log snippet).

Thanks!

Marking confirmed, since multiple reports seem to indicate that this issue is widespread.

High priority, as this is core to the functionality of a working cloud.

:-Dustin

From the description of bug #436885, this doesn't feel like a duplicate at least, since the deadlock condition wasn't following any significant usage. But then, the other bug is quite short on details, so it could just be a non-relevant bit.