PPA GPG signatures broken when accessing launchpad via transparent proxy

Hi

I'm looking into what I can do about the PPA server for you.

However, nz.archive.ubuntu.com is a mirror and is not run by Canonical (https://edge.launchpad.net/ubuntu/+mirror/ftp-citylink-co-nz), you should contact the admin of that site.

I'll post back here when I've got the PPA server fixed.

So, the intercepting cache admins are partly right partly wrong.

They are right that we don't serve explicit CC headers at the moment. However they are wrong that this is a problem:
Intercepting caches break the end to end rule on the internet: client software that can may how to send proper cache control headers (e.g. INM, IMS) often only does so when no cache is configured. As we don't get reports of this behaviour from other users (like me, and I have a cache at home) its highly likely that apt doesn't know how to deal with silent man in the middle attacks on HTTP.

Note that Apt already makes IMS requests for some files, but perhaps differently between Release and Release.gpg.

The underlying issue here is that for any two resources Foo, Bar on the internet there is no plain-http way to say 'I want the same version of both', and so even the most well tuned and properly behaving client and server can still run into skew.

There are three things that should be changed to really stamp this out without destroying caching:
 - our servers
 - the client software
 - the disk format

Firstly the disk format is not HTTP friendly: we have interrelated resources (Release, Release.gpg, and so on) that cannot permit skew. If we were to change that to inline-sign (perhaps by signing Release, and also providing Release.gpg for older clients, or by adding a Release.signed which would supercede Release altogether), then an old Release file wouldn't cause a spurious signature failure.

Serverside:
 for files that are immutable we don't need to do anything.
 for files that may change, we should add:
  - Expires and cc:s-max-age , with both set to cause expiry the next time the file may change. (expires is 1.0, max-age 1.1) see rfc 2616 14.9.3
 - cc: proxy-revalidate (permits caching, but caches have to ping us at each expires/max-age interval)
 we should also be sending strong etags.

Clientside:
Firstly, on a signature failure, apt should re-request the relevant signature and Release file with max-age=0, try again, and if that still fails try with no-cache. apt should do this even if there isn't a configured proxy, because there are many intercepting caches in the wild.
Secondly on all requests apt should add a INM giving the strong etag if one was supplied.

Finally, note that update-manager is totally hosed in NZ on telstra-clear at the moment with our production archive, for the same reason.

Robert,

am Fri, Nov 20, 2009 at 08:13:01PM -0000 hast du folgendes geschrieben:
> Firstly the disk format is not HTTP friendly: we have interrelated
> resources (Release, Release.gpg, and so on) that cannot permit skew. If
> we were to change that to inline-sign (perhaps by signing Release, and
> also providing Release.gpg for older clients, or by adding a
> Release.signed which would supercede Release altogether), then an old
> Release file wouldn't cause a spurious signature failure.

the Debian FTP masters recently added a Release file called InRelease which
is inline signed for exactly this purpose. I don't know if there has been
progress on that on the frontend side, though.

Kind regards,
Philipp Kern
--
 .''`. Philipp Kern Debian Developer
: :' : http://philkern.de Stable Release Manager
`. `' xmpp:phil@0x539.de Wanna-Build Admin
  `- finger <email address hidden>

A work-around was pointed out to me by a coworker: using ftp APT mirrors instead of HTTP-based ones. So far it seems to work. I guess the TelstraClear transparent proxy doesn't tamper with those.

It doesn't seem possible to access PPA repos through FTP, but it does work for the main Ubuntu repos.

Cheers,
Francois

@Rob: WRT Release, Release.Gpg and InRelease> surely you also need the
relevant Packages + Sources files to be in sync with *Release*?

Also, we explicity and globally disable Etags on our mirrors, based in
no small part on your recommendation in LP #290921. Should we now
turn it back on for indices files (Releases*, Packages*, Sources*)?

@James: Yes, the chain of files probably need to be consistent, which likely means that client side smarts are more important.

For Etags, and rechecking that bug, the problem is that apache was using the inode, and that is different on every mirror, so round robin load balancing leads to caches pulling the content again. There was a suggestion in there to use
FileETag MTime Size
which would change appropriately. Remembering that:
 - packages files never replace content, they only write new unique files.
 - sources/packages/release contents do get replaced, but replacing them would change the mtime and thus the etag

The key bit from the other bug was 'if the rsync scripts are not preserving mtime, then I'd just turn off etags'. I'm concluding from that and that they are off that the mirror scripts don't preserve mtime?

However, a quick unscientific check suggests they do:
robertc@lifeless-64:/tmp$ wget -S http://archive.ubuntu.com/ubuntu/dists/karmic-proposed/main/dist-upgrader-all/0.126.9/karmic.tar.gz.gpg 2>&1 | grep Last-Modified
  Last-Modified: Mon, 02 Nov 2009 10:27:02 GMT
robertc@lifeless-64:/tmp$ wget -S http://archive.ubuntu.com/ubuntu/dists/karmic-proposed/main/dist-upgrader-all/0.126.9/karmic.tar.gz.gpg 2>&1 | grep Last-Modified
  Last-Modified: Mon, 02 Nov 2009 10:27:02 GMT
robertc@lifeless-64:/tmp$ wget -S http://au.archive.ubuntu.com/ubuntu/dists/karmic-proposed/main/dist-upgrader-all/0.126.9/karmic.tar.gz.gpg 2>&1 | grep Last-Modified
  Last-Modified: Mon, 02 Nov 2009 10:27:02 GMT
robertc@lifeless-64:/tmp$ wget -S http://us.archive.ubuntu.com/ubuntu/dists/karmic-proposed/main/dist-upgrader-all/0.126.9/karmic.tar.gz.gpg 2>&1 | grep Last-Modified
  Last-Modified: Mon, 02 Nov 2009 10:27:02 GMT
robertc@lifeless-64:/tmp$ wget -S http://eu.archive.ubuntu.com/ubuntu/dists/karmic-proposed/main/dist-upgrader-all/0.126.9/karmic.tar.gz.gpg 2>&1 | grep Last-Modified
  Last-Modified: Mon, 02 Nov 2009 10:27:02 GMT

In which case we would be more correct with Etag MTime Size, than None.

Some work was done on the Apache side, let us know if it fixes things for you.