librarian front-end caches can't be forced to revalidate

Another case where this happens is <https://launchpadlibrarian.net/17245907/lp-gem-64x64.png>

Because these are fetched over https it's probably not something on our network: either my laptop is quite sick or it's inside of the Canonical dc.

The failure rate is imprecise but about 50% which suggests this might be one of the two front end squids(?) is sick?

I thought I'd also seen 500 errors but perhaps I imagined it.

I can reproduce the same failure in firefox and chrome so it's probably not a crazy local cache.

jpds points out that the failures are always happening from the nutmeg cache and never from banana.

Related log entry from squid:

127.0.0.1 - - [12/Jul/2010:15:13:17 +0100] "GET http://launchpadlibrarian.net/36711181/logo64.png HTTP/1.0" 404 460 "https://bugs.edge.launchpad.net/launchpad-foundations/+bug/604620" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6" TCP_NEGATIVE_HIT:NONE [Host: launchpadlibrarian.net\r\nUser-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-gb,en;q=0.5\r\nAccept-Encoding: gzip,deflate\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nReferer: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/604620\r\nCache-Control: max-age=0\r\nVia: 1.1 launchpadlibrarian.net\r\nX-Forwarded-For: 193.85.232.179\r\nX-Forwarded-Host: launchpadlibrarian.net\r\nX-Forwarded-Server: launchpadlibrarian.net\r\n] [HTTP/1.0 404 Not Found\r\nDate: Mon, 12 Jul 2010 14:09:00 GMT\r\nContent-Length: 146\r\nContent-Type: text/html\r\nServer: TwistedWeb/10.0.0\r\n\r]

Turns out these resources have actually been removed.

Arguably it's a configuration bug that even when the client sends cache-control: no-cache or max-age=0, the squids do not seem to revalidate and notice that the file is gone.

So there are two things here:
 - should clients be able to force squid to go back to the librarian
 - do we care about consistency when an item is just added/deleted in the librarian.

For the former, no, clients don't ever need to force a refresh from the librarian - the content is immutable, and we want squid to be authoritative. Contrast this with dynamic content where there is more of an argument that clients should be able to say 'hey, I know better'.

For the latter, we tolerate some slack on deletes. Were we to particularly worry about some content vanishing instantly, we could start using PURGE. For adds, we put the content in the librarian before handing out URLs, so negative misses are not an issue for us.

With the above in mind, I'm closing this as wont fix.

sounds reasonable to me