preflight check for removing edge cluster

Assigning to Leonard for a first pass at answers, and to let us know what we will actually need to dig into.

I don't have a firm answer to any of these questions yet, but I can confirm that the very first revision of launchpadlib used edge by default.

Oh, I don't understand what "doing a rewrite in apache" means. Does it means having production simply respond to requests for edge? That *might* work.

OK, I've verified that the versions of launchpadlib in Karmic, Lucid and Maverick all use staging by default. However, that's just a default. It's possible that applications like ground control use edge by default, or are hard-coded to use edge. To say nothing of individuals' scripts. I don't think "we don't use edge by default" short-circuits the need to see what happens to launchpadlib if edge goes away.

I experimented with setting up redirects on my personal website, and pointing launchpadlib at my personal website. The first thing launchpadlib tried to do was POST to +request-token on my personal website. My redirect caused launchpadlib to raise an exception, because it's illegal for a POST that results in a redirect to be re-POSTed without user input. This means that if you don't already have a credential for edge.launchpad.net, and you use a script that tries to access edge, you will get an exception. launchpadlib won't re-POST given the redirect.

If you already have a credential, it *almost* works, but not quite. The GET for the WADL is redirected and the WADL is downloaded. But once you try to use some feature of the web service, launchpadlib notices that the base URL in the WADL is different from the "service root" URL. It gets confused and falls over.

So, redirects won't solve the problem for old versions of launchpad. But, people who use old versions of launchpad are not using edge by default.

I suspect that having some other machine answer requests for 'edge' will also not work, unless we can also serve a special copy of the WADL that claims to be the WADL for edge.launchpad.net.

On Thu, Nov 4, 2010 at 5:05 AM, Leonard Richardson
<email address hidden> wrote:
> Oh, I don't understand what "doing a rewrite in apache" means. Does it
> means having production simply respond to requests for edge? That
> *might* work.

Yes.
client -> apache 'get https://edge.launchpad...'
apache -> launchpad 'get https://launchpad...'

-Rob

Thank you, Leonard.

To follow on with comment 5, Leonard and I don't think that a naive forwarding of edge to production will work. Moreover, we're skeptical that simply serving adjusted WADL will be sufficient.

Zope typically handles virtual hosts such that the same process can respond to requests from different vhosts with the same results, except maintaining links within the requested vhost. I don't know if Launchpad still is capable of this, or if it has been customized out of that functionality, but I'm cautiously optimistic that it would be workable.

If it were, we have the only fully backwards-compatible solution we've identified so far: edge.launchpad.net (and friends, like edge.code.launchpad.net and so on) live on as vhosts against the production server, and Launchpad simply keeps URLs (including WADL, which would have to be handled specially) within the requested vhost. This is probably mostly a canonical_url dance, within the LP codebase.

On Thu, Nov 4, 2010 at 6:58 AM, Gary Poster <email address hidden> wrote:
> Thank you, Leonard.
>
> To follow on with comment 5, Leonard and I don't think that a naive
> forwarding of edge to production will work.

I'm curious what we're anticipating as failures.

> Moreover, we're skeptical that simply serving adjusted WADL will be sufficient.

Does the WADL have the hostname in it?

> Zope typically handles virtual hosts such that the same process can
> respond to requests from different vhosts with the same results, except
> maintaining links within the requested vhost.  I don't know if Launchpad
> still is capable of this, or if it has been customized out of that
> functionality, but I'm cautiously optimistic that it would be workable.

Its definitely possible - after all we serve 6 or so vhosts today. We
may hit friction in canonical_url though.

> If it were, we have the only fully backwards-compatible solution we've
> identified so far: edge.launchpad.net (and friends, like
> edge.code.launchpad.net and so on) live on as vhosts against the
> production server, and Launchpad simply keeps URLs (including WADL,
> which would have to be handled specially) within the requested vhost.
> This is probably mostly a canonical_url dance, within the LP codebase.

I'd like a plan that eventually fades to no overhead.

If I understand correctly, this is where we are at:
 - no launchpadlib versions in Ubuntu releases use edge by default
 - apps build on launchpadlib may be using edge (and in fact some probably are)
   - so lets look at the lplib usage data to identify those - so we
can estimate their impact and disappearance.
 - redirects won't work
 - rewrites in apache probably won't work
 - zope appserver changes may work

Another approach is just to shrink the edge cluster as rapidly as we
can, with a redirect for non-apis.

-Rob

Thanks for the excellent work. Edge isn't removed entirely, but we know what remains. We'll wait a release or so to let the newer launchpadlib percolate around, then turn it off.