Can't login with SSH to Lucid UEC image - uec metadata service returns a string instead of a list of string for ssh public keys

Note sure if it also affects the cloud image on EC2.

The Lucid / 20100108.1 image works alright.

euca-get-console-output stops before "ec2: Generating public/private rsa key pair." on the affected images.

just a point of reference, I just tried
ami-f73cd19e ubuntu-images-testing-us/ubuntu-lucid-daily-i386-server-20100112.manifest.xml
and was able to ssh to it.

Confirmed regression in UEC:
20100111.1 FAIL
20100111 PASS
So it's more than probably linked to ec2-init 0.5.0

It looks like this is a difference in the metadata service on eucalyptus and ec2.
python -c 'import boto.utils; import pprint; pprint.pprint(boto.utils.get_instance_metadata())'

get_instance_metadata() is used by ec2-init 0.5.0 to crawl metadata service. The objects look different. For reference, I'll attach them here.

Please see build 0.5.0-0ubuntu3~smoser2 at https://launchpad.net/~smoser/+archive/ppa/+packages?field.name_filter=ec2-init&field.status_filter=published&field.series_filter=lucid

You can test this by booting your 20100111 instance that boots, and
- installing new ec2-init deb
- rm ~ubuntu/.ssh/authorized_keys
- reboot

As outlined in comment 7 and 8:

On ec2:

 'public-keys': {'ec2-keypair.us-east-1': ['ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCD9dlT00vOUC8Ttq6YH8RzUCVqPQl6HaSfWSTKYnZiVCpTBj1CaRZPLRLmkSB9Nziy4aRJa/LZMbBHXytQKnB1psvNknqC2UNlrXXMk+Vx5S4vg21MXYYimK4uZEY0Qz29QUiTyNsx18jpAaF4ocUpTpRhxPEBCcSCDmMbc27MU2XuTbasM2NjW/w0bBF3ZFhdH68dZICXdTxS2jUrtrCnc1D/QXVZ5kQO3jsmSyJg8E0nE+6Onpx2YRoVRSwjpGzVZ+BlXPnN5xBREBG8XxzhNFHJbek+RgK5TfL+k4yD4XhnVZuZu53cBAFhj+xPKhtisSd+YmaEq+Jt9uS0Ekd5 ec2-keypair.us-east-1',
                                           '']},
On UEC:

 'public-keys': {'mykey': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbw9Pxu9Wq7v32vEYuGlzB4qYX6bnIt+/zIe8mdO0Eyt+smus1PYMic9FGuSrc9tUJyMHeNbXRvqQEqQdfGJb1KlCuSBl7kqUQwnVZj++jYWRCmcgzxhBPtCmZAniqo8+XFIOuigydkiipYPcnQ7DEEcXNmRb3I/4XENG+caIoA5+BwaBx3+si+XiwC8C8IKs0ZQ+IDbu0RySVp9RcSiE/kuuYrdoPaToGA7mf6oQMX2RrY7aAUnVqI+nm7cK+20GP8xoHSUm3Ohg/ioDglETwxZD5kzkdgtzHoNdvD3CBq3iBJM0Nd/o1OL9qK375kLdC0XwnnuXZBCCeumWWhwIX admin@eucalyptus'},

Not as relevant here, but to point out, there are other differences in the 2 files:
 - uec has 'ancestor-ami-ids' field (probably ok)
 - ec2 has instance-action
 - uec has 'product-codes' . apparently if there are none, ec2 doesn't present that in the metadata
 - the noted public-keys difference

#12:

ancestor-ami-ids are also in ec2 if the image has ancestors.
instance-action is not supported by Eucalyptus at this time (so it is absent).

#11:
The difference between the two outputs posted seems to be a parser issue which is converting the public-key string into a list. Note the second empty string entry in the example EC2 output.

Even if the bug is in boto parsing, there is something different in UEC metadata output, compared to EC2, that makes boto behave differently ?

ec2-init (0.5.0-0ubuntu3) lucid; urgency=low

  * work around difference in uec/ec2 metadata service (LP:506332)
 -- Scott Moser < <email address hidden>> Tue, 12 Jan 2010 11:33:11 -0500

yes, obviously there is something different in the server metadata output that makes the client parse it different. by definition of api compatibility, that is a bug in the server. I agree that the parsed eucalyptus data seems more sane, but thats not something that can be changed.

Actually, this is not part of any of the EC2 specifications.

If there is a need to change the behaviour of Eucalyptus to remedy
this problem I will need more information about what differs between
the two. That is, the provided output indicates there is a
difference, but neither is the actual output and so I can't tell what
needs to be changed.

c

On Wed, Jan 13, 2010 at 6:05 AM, Scott Moser <email address hidden> wrote:
> yes, obviously there is something different in the server metadata
> output that makes the client parse it different.  by definition of api
> compatibility, that is a bug in the server.  I agree that the parsed
> eucalyptus data seems more sane, but thats not something that can be
> changed.
>
> --
> Can't login with SSH to Lucid UEC image - uec metadata service returns a string instead of a list of string for ssh public keys
> https://bugs.launchpad.net/bugs/506332
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Incomplete
> Status in “ec2-init” package in Ubuntu: Fix Released
> Status in “eucalyptus” package in Ubuntu: Confirmed
> Status in “ec2-init” source package in Lucid: Fix Released
> Status in “eucalyptus” source package in Lucid: Confirmed
>
> Bug description:
> Binary package hint: ec2-init
>
> Testing 20100112 UEC image (amd64) on UEC
> Instance starts OK, by when trying to connect through SSH, my key is refused:
>
> ttx@odyssey:~$ euca-run-instances -k mykey emi-6F0C1333 -t m1.large
> RESERVATION     r-3ABF0772      admin   admin-default
> INSTANCE        i-367506EE      emi-6F0C1333    0.0.0.0 0.0.0.0 pending mykey   2010-01-12T10:05:12.947Z        eki-9E481847    eri-803C17BC
> ttx@odyssey:~$ ssh -i mykey.priv ubuntu@192.168.0.230
> Permission denied (publickey).
>
> Running the karmic UEC image on the same UEC setup succeeds, so it's something in the lucid image.
>
>
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

We workarounded the issue in ec2-init by supporting both formats. The issue still stands in eucalyptus, but it's no longer blocking for us (so set to importance Low). It's more about other tools that might query the metadata service and could run into the same issue as we did. Even if it's not written into any EC2 spec, it's still something where Eucalyptus output differs from EC2's one.

Just to clarify, I am not objecting to fixing the issue in Eucalyptus.
 Rather, the information provided in this bug report is not enough for
me to fix it (and I would need someone to provide that information for
me as I don't have an EC2 account). The HTTP response itself would be
good enough.

thanks.
chris

On Mon, Jan 18, 2010 at 11:36 PM, Thierry Carrez
<email address hidden> wrote:
> We workarounded the issue in ec2-init by supporting both formats. The
> issue still stands in eucalyptus, but it's no longer blocking for us (so
> set to importance Low). It's more about other tools that might query the
> metadata service and could run into the same issue as we did. Even if
> it's not written into any EC2 spec, it's still something where
> Eucalyptus output differs from EC2's one.
>
> --
> Can't login with SSH to Lucid UEC image - uec metadata service returns a string instead of a list of string for ssh public keys
> https://bugs.launchpad.net/bugs/506332
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Incomplete
> Status in “ec2-init” package in Ubuntu: Fix Released
> Status in “eucalyptus” package in Ubuntu: Confirmed
> Status in “ec2-init” source package in Lucid: Fix Released
> Status in “eucalyptus” source package in Lucid: Confirmed
>
> Bug description:
> Binary package hint: ec2-init
>
> Testing 20100112 UEC image (amd64) on UEC
> Instance starts OK, by when trying to connect through SSH, my key is refused:
>
> ttx@odyssey:~$ euca-run-instances -k mykey emi-6F0C1333 -t m1.large
> RESERVATION     r-3ABF0772      admin   admin-default
> INSTANCE        i-367506EE      emi-6F0C1333    0.0.0.0 0.0.0.0 pending mykey   2010-01-12T10:05:12.947Z        eki-9E481847    eri-803C17BC
> ttx@odyssey:~$ ssh -i mykey.priv ubuntu@192.168.0.230
> Permission denied (publickey).
>
> Running the karmic UEC image on the same UEC setup succeeds, so it's something in the lucid image.
>
>
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

Okay, so what remains to be fixed in Ubuntu's Eucalyptus package for this bug?

I see a request from Chris for more information such that Eucalyptus could solve this.

I'm marking this bug incomplete, for now, as it doesn't appear there's enough information to solve the Eucalyptus part of this bug (if it exists there at all). Please update the status if you have more or better information than me.

Debdiff looks sane but is for a package in main, so subscribing sponsors.

Marking Invalid for the Eucalyptus (ubuntu) task, as it seems to be resolved elsewhere. Please re-open if this is not the case.

Thanks.

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2674

Please watch that issue for further updates.