Launchpad requires the REFERER header on form submission breaking with noscript and other privacy/spam browser plugins

1) Yes, this was a conscious compromise.

2) You and I disagree on your interpretation ("Your Referer check means that your site will not be accessible from RFC 2068-compatible browsers") but I of course acknowledge the underlying fact of the RFC language. It was conscious compromise.

3) The only browser-related ways of forging REFERER headers I found (http://www.cgisecurity.com/lib/XmlHTTPRequest.shtml, for instance) appeared to be pertinent to older versions of software; and HTTP, not HTTPS. At this time, our posts are exclusively HTTPS.

Given other priorities, I do not plan to revisit the decision in the short term.

We might revisit the decision when we open up much of the site to HTTP, which may be within a few months.

Gary

We perhaps should update the wording. We only require a referrer for
POST requests I believe. I know I use a referrer blocker set to allow
referrers within a domain but not when crossing domains (the
recommended approach due to the huge number of sites that do the same
as Launchpad for similar reasons). This addresses the privacy
concerns.

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

ARG!!!!! I file a bug. I type the long text (bug description), click submit. error "No referer, please enable referer". Click Back, long bug description is gone! AAAARRRGGG! how stupid/lazy do you have to be to write such web software??? Firefox goes to great length to save form contents when you go back, but you manage to lose it. This makes it a dataloss bug.

And as said above, referers are entirely optional and it must be possible to disable them, per HTTP spec. Because they are a privacy problem, and the explicitly spec says so Software which requires them is broken. And you don't need referers, as the above comments state, and referers are actually less secure than the right solution. This is a plain bug, and a privacy problem. Fix it, yesterday!

FWIW, I run with referers disabled since a long time, and launchpad is the *only* app/site where I have problems.

Perhaps if Launchpad gets a request with no Referer it could send you to a non-ajax form to confirm the operation? Ideally this would include the text that you previously tried to enter.

The non-AJAX forms also require Referer to be set.

Ping. Please fix this. As said above, this not only prevents a perfectly standards-comformant client from commenting, it also causes data loss, because the comment cannot be resurrected!

I argued earlier that checking the Referer header is not an effective defense against CSRF, but I suspect my argument was not specific or concrete enough to be convincing. Let me try again, this time with more technical specifics:

Query: Are the Launchpad maintainers sure they know about all the attacks against the Referer header? Before answering, you may wish to read Section IV.C of this paper: http://www.cs.berkeley.edu/~devdatta/papers/websec-csf10.pdf

In particular, the attack described in that paper goes something like this:
1. Suppose the attacker owns and controls the server attacker.com.
2. The attacker posts a link https://attacker.com/badstuff on a launchpad page.
3. The victim (who is a launchpad user and is logged on to launchpad) visits that launchpad page and clicks on the link to https://attacker.com/badstuff.
4. The victim's browser requests https://attacker.com/badstuff from the attacker's server, and sends Referer: https://launchpad.net/foo in the headers.
5. The attacker's server responds with a 301 Redirect, which redirects the victim's browser to https://bugs.launchpad.net/csrfurl
6. The victim's browser receives the redirect, requests https://bugs.launchpad.net/csrfurl from the launchpad server, and sends Referer: https://launchpad.net/foo with the request.
7. The launchpad server sees a Referer header that appears valid and therefore processes the request for /csrfurl by performing some side-effecting action, under the (mistaken) assumption that the user must have requested that action.

The key observation in the attack is that browsers copy the Referer header sent in the original request across redirects. In other words, when a user at site A clicks a link to site B, and site B issues a redirect to site C, the browser sends Referer: A to site C.

This is just one of numerous attacks against Referer-checking. There have been others in the past as well (e.g., Flash-based attacks), some of which may still be pertinent today, and I fully expect other attacks may well exist, too, even if we are not aware of them. Therefore, adding some ad-hoc defense against redirects is not, in my view, right solution. Rather, I think the takeaway lesson is this: Referer headers are not an effective or reliable defense against CSRF attacks. Referer headers were not designed for this purpose. There are subtle ways to manipulate them; browser behavior can be surprising and have unexpected consequences for security mechanisms that rely upon the validity of Referer headers. It is for this reason that security experts universally (in my experience) recommend against using Referer-checking to prevent CSRF. Therefore, Launchpad's decision to rely upon Referer headers seems like one that should perhaps be revisited.

I have not attempted to mount a redirect-based attack, such as the one sketched above, against Launchpad, so I have not confirmed whether Launchpad is vulnerable. However, it appears to me that Launchpad may be creating a false tradeoff between security and usability. If my analysis is correct, it sounds like Launchpad may have chosen a solution that achieves neither security nor usability. There are better ways to d...

Redirects are not a problem. They can only result in GET requests, which cannot perform modifications in any correct Web application. The cited paper appears to be ignorant of that fact.

Well, it would be relevant to a badly designed webapp that has GET
mutators. Separately,I have vague memory of support being built for
POST + redirect, but we don't need to worry about that [yet, if ever].

Ah, but a webapp with GET mutators is probably not correct.

I think we need to revisit this because of the support cost of dealing with users using e.g. noscript.

@Ben the bug of losing your input is really bug 553946, which can occur in other situations too.

While we should revisit this in the near term (checking Referer places undesirable restrictions on user agents), this is not significant for security. Forging a Referer header on a POST requires a browser or plugin bug, at which point you are in trouble anyway.

On 2011-04-11 at 03:05 -0000 William Grant sent off:
> While we should revisit this in the near term (checking Referer places
> undesirable restrictions on user agents), this is not significant for
> security. Forging a Referer header on a POST requires a browser or
> plugin bug, at which point you are in trouble anyway.

a FF4 with noscript does not generally block referrers on ordinary POST
requests. Possibly some of the heavy script magic that lauchpag uses causes the
referrer to be removed by noscript. Anyway - people like me, that are bitten by
this launchpad annoyance CAN use other sites without trouble.

--
comment sent via my mutt

You could say what you like about "broken" plug-ins and browsers. I suggest that launchpad uses csrf tokens as one means of protection against csrf.

I haven't personally seen noscript cause problems on launchpad.net. Perhaps some one should open a separate bug about this ?

This problem angers me, too.

Requiring the referer makes no sense at all. If this is some kind of security mirage, everyone can add an Http header easily.

At least, when you give this error message, throw the user back to the same form, with all the data he tried to submit before already filled in.

This way, if he is now enabling the referrer, he does not have to type in the whole bug report (or whatever this was) again.

Dear users & readers,

Ubuntu is a world-class OS & more, & it is free.

Could we just give launchpad what they want?

Download Firefox add-on "Change Referer Button"

This add-on app keeps the user in control as to whom

they choose to reveal the Referer Header to.

Ubuntu already knows who we are, they are not nearly

as secretive as other apps or OSs, they deserve our free

support.

BavarianPH,
Ubuntu Forever

On 6/2/11, Paul Ebermann <email address hidden> wrote:
> At least, when you give this error message, throw the user back to the
> same form, with all the data he tried to submit before already filled
> in.
>
> This way, if he is now enabling the referrer, he does not have to type
> in the whole bug report (or whatever this was) again.
>
> --
> You received this bug notification because you are a direct subscriber
> of a duplicate bug (596225).
> https://bugs.launchpad.net/bugs/560246
>
> Title:
> Launchpad requires the REFERER header on form submission breaking with
> noscript and other privacy/spam browser plugins
>
> Status in Launchpad itself:
> Triaged
>
> Bug description:
> It appears that Launchpad blocks access if your browser does not send
> a Referer header. See:
>
> https://answers.launchpad.net/launchpad/+faq/1024
>
> The FAQ claims that this is done to prevent CSRF. This sounds like a
> dubious decision, for three reasons:
>
> 1) Requiring a Referer header is bad for privacy. The Referer header can
> be used to track people's actions on the web and thus is a potential privacy
> risk which I prefer to avoid. It is none of site A's business what site B I
> was browsing previously. It's not just me; you are blocking a non-trivial
> user population. Some client-side privacy tools block the Referer, for good
> and valid privacy reasons. Some privacy-sensitive users configure their
> browsers to avoid sending Referer headers. See, e.g.,
> http://www.ecommerce-blog.org/archives/neweggcoms-usability-blunder/
> Some proxies and firewalls strip the Referer from all HTTP requests, as a
> security risk and a privacy risk. By blocking those users, you are either
> harming user privacy or losing out on useful participation from people who
> care about privacy.
>
> 2) The RFC (RFC 2068) specifically envisions that users should be able
> to disable sending Referer headers, and recommends that web browsers
> provide a way so that users can enable/disable this, because of its
> privacy implications. Your Referer check means that your site will
> not be accessible from RFC 2068-compatible browsers.
>
> 3) Requiring a Referer header does not prevent CSRF. The Referer
> header is not reliable for security purposes; there are a number of
> techniques that can be used to forge Referer headers. If the Referer
> check is the only defense against CSRF, then Launchpad is probably
> vulnerable to CSRF. If Launchpad is using a proper defense against
> CSRF (e.g., double-cookie submission, CSRF tokens), then it is secure
> without the Referer check and blocking people who don't send a Referer
> header is gratuitous. I'm not sure why it would be necessary to force
> users to lower thei...

So, it would indeed be nice to revisit this, as the header isn't the full story. However we are fairly happy with our CSRF protection today, and while we're happy to improve it it is not on our current roadmap.

If someone wants to put forward a patch to change things, we'd be delighted to mentor them in getting started on LP development.

Well I hope this gets some attention some day ... !
Amusingly enough login.launchpad.net already seems to have a csrf token.

> Amusingly enough login.launchpad.net already seems to have a csrf token.

login.launchpad.net is a different system with a totally different web stack.
-Rob

It's not the users task to decide nor recognize if login.launchpad.net is part of launchpad software or not - the domain tells enough.

If, as you say, referer checks used only for POST request wouldn't do any harm, then these comments prove the opposite.

Just tried to report a bug via "ubuntu-bug" on the ocmmand and was taken to the launchpad login page. With referrer blocked I could not log in.

More annoyingly, the "login failed" page I was taken to simply said that the page was "stale". No inidcation that blocking the referrer was the cause.

> FWIW, I run with referers disabled since a long time, and launchpad is the *only* app/site where I have problems.

I can confirm this. Nothing else so far has required me to send a Referrer header. This is annoying, please fix it.

> More annoyingly, the "login failed" page I was taken to simply said that the page was "stale". No inidcation that blocking the referrer was the cause.

I concur.

Same here, the only website I ever encountered that doesn't work without a Referer :(

Actually, when trying to login with the Launchpad OpenID service and Referrer deactivated, now I get a spurious error message about a stale page or something like this (it also is badly translated to german). Could this error message please be updated to include a mention of the Referrer need, so I can activate it?

When the Firefox option network.http.sendRefererHeader is set to 1, referers are send in POST requests but not in XMLHttp POST Requests. This results in a user being able to edit, for example, the bug description by clicking the link while holding shift. It is, however, impossible for the user to use the same approach when adding a comment. Thus making it impossible for a lot of users to post comments to Launchpad.

When posting something using an XMLHttp POST Request, the message returnewd contains a full HTML page of which the source is displayed plain-text in your Yui pretty-overlay-window. I think this is a bug on itself but it will also be resolved when this bug is finally resolved after more than 2 years of annoying users with faulty 'security' mechanism's.

I recently learned some additional information that may be of some modest utility in evaluating how much we should rely upon the unforgeability of Referer headers.

In April 2010, one developer wrote "The only browser-related ways of forging REFERER headers I found appeared to be pertinent to older versions of software; and HTTP, not HTTPS."

My response was that I may not know of a specific attack, but "there have been numerous [Referer-forgery attacks] in the past as well (e.g., Flash-based attacks), some of which may still be pertinent today, and I fully expect other attacks may well exist, too, even if we are not aware of them."

I recently learned of a new vulnerability that was not publicly disclosed at the time, but has since been revealed. Even as one developer wrote in April 2010 that he believed Launchpad's reliance upon Referer headers to be safe, there was an extant vulnerability in Flash that allowed forgery of request headers. If I'm understanding things correctly, this would appear to allow forgery of arbitrary custom headers, even for POST requests (I wasn't able to find any mention about HTTP vs HTTPS). The vulnerability was not publicly reported until almost a year later. The vulnerability has since been fixed, so if users are running a fully patched version of Flash, the vulnerability should not be exploitable any longer. Nonetheless it means that, even though we didn't know it at the time, there apparently was a vulnerability that allowed forgery of request headers at the time that one developer was expressing confidence that this couldn't happen.

References: http://lists.webappsec.org/pipermail/websecurity_lists.webappsec.org/2011-February/007533.html

(Yes, this was in Flash. But many users do use Flash, and the site presumably needs to be secure for them, too.)

Someone else wrote "Redirects are not a problem. They can only result in GET requests". While at the time I believed this statement, my belief appears to have been mistaken; the vulnerability listed above apparently involved POST requests and redirects.

Of course, all of that history now, water under the bridge. But it may be relevant to an assessment of the likelihood of other such vulnerabilities being found in the future. When there have been two or three or four bugs of a certain type found so far, it is natural to suspect that there could be one more that we don't know about yet, just waiting to be found. Historically, confidence in the Referer header has been mis-placed. I don't know whether there's a solid basis for believing we've seen the last of such vulnerabilities. I leave it to you to form your own opinion about the implications of this past history.

This is a real problem. Can a develop take this up to fix it? Also please change importance to high.

Submitting my patch that fixes this problem.

Attaching a patch.

So what's happening? Why the delay? There's a fix attached, let's commit it and get on with it.

That is not a fix, it just disabled Lp's security. A fix will permit the tools to work without compromising Lp. Such as fix would require tests that show Lp is protected.

I see. What would be the test requirements to show that level of protection does not change? Do you have testcases or do you follow an auditing standart?

I could try and produce those tests.

The tests would show that Launchpad remained invulnerable to cross-site request forgery attacks. Your patch removes our protection against such attacks, so it's not an acceptable approach.

I think they would show that Lauchpad is already currently vulnerable to XSRF and nothing would change in that respect. So, is there a specific standart or ready made testcases that need to be presented to go forward with this change?

If not, and just plain speculation is enough - take my word for it - XSRF is already possible and should NOT be "solved" by checking ref header.

Can you demonstrate that with a sample attack?

Robert, that doesn't matter. Requiring referer is not an option on the web, because the HTTP spec not only says that it's optional, but specifically warns about privacy problems it causes.

See http://www.apps.ietf.org/rfc/rfc2616.html#sec-15.1.3
Quote from the HTTP spec:
"
Because the source of a link might be private information or might reveal an otherwise
private information source, it is strongly recommended that the user be able to
select whether or not the Referer field is sent. For example, a browser client could
have a toggle switch for browsing openly/anonymously, which would respectively
enable/disable the sending of Referer and From information.
"

So, you're saying I can only contribute to Ubuntu when I give up my privacy? I hope not. But that's currently the choice I have.

---

This implements a CSRF without relying on referers:
https://github.com/mozilla/django-session-csrf
HTH

On 26 March 2013 06:52, Ben Bucksch <email address hidden> wrote:
> Robert, that doesn't matter. Requiring referer is not an option on the
> web, because the HTTP spec not only says that it's optional, but
> specifically warns about privacy problems it causes.
>
> See http://www.apps.ietf.org/rfc/rfc2616.html#sec-15.1.3
> Quote from the HTTP spec:
> "
> Because the source of a link might be private information or might reveal an otherwise
> private information source, it is strongly recommended that the user be able to
> select whether or not the Referer field is sent. For example, a browser client could
> have a toggle switch for browsing openly/anonymously, which would respectively
> enable/disable the sending of Referer and From information.
> "

That recommends user agents have the capability, and I totally support
that. The use of Referer here has no privacy implications because
there is no requirement that Referer from other sites be provided,
merely that Referer from within LP be provided.

> So, you're saying I can only contribute to Ubuntu when I give up my
> privacy? I hope not. But that's currently the choice I have.

I am saying nothing of the sort, and as I show above, that isn't in
fact the case. Turn off referer for all other sites, turn it on for LP
only.

Or, as has been said before, contribute a patch to implement the
required protection in another fashion.

-Rob

> Turn off referer for all other sites, turn it on for LP only.

That's not possible. Firefox has no preference for that. Other browsers are even in worse situations.

Fact is: The spec says the referer header is purely optional.
http://www.apps.ietf.org/rfc/rfc2616.html#sec-14.35.2
Requiring it is, thus, a violation of the spec. Webapps *cannot* require Referer.

> Or, as has been said before, contribute a patch to implement the
> required protection in another fashion.

You haven't defined what exactly is "required" yet. Several people here have said that the check is not necessary for CSRF. You have only replied that the Referer check is necessary. Without high-level spec, there are no alternative implementations possible.

In my last comment, I have given an example how to implement CSRF without referer.

You provided a link to a django plugin. Launchpad is a Zope site.

In terms of a high level spec - prevent CSRF, don't break existing scripts. If you are planning on working on this, I'd be delighted to drill into what any of that means. I don't have a particular implementation in mind (but perhaps someone that is still working on Launchpad does and could advise you).

I think you meant to link to http://www.apps.ietf.org/rfc/rfc2616.html#sec-14.36 not to the Range header definition.

That said, it is *optional* whether to send it, and your browser is choosing not to send it. That doesn't make a server that chooses to reject POST requests without in be in violation of the specification, because RFC2616 says nothing about interpretation of referer only about the syntax, privacy implications and user agent considerations.

Is this discussion even relevant? Is launchpad a major target for XSRF?

Seeing as e.g. Google services work fine without REFERER header, why can't Launchpad?

> Seeing as e.g. Google services work fine without REFERER header, why can't Launchpad?

Exactly. Launchpad is pretty much the only website that I know and use that uses the referer (in a blocking way). Google, domain registrars, server hosters, all web stores, they all manage without referer and without getting hacked.

> In terms of a high level spec - prevent CSRF, don't break existing scripts.

Good. The referer check is not the only CSRF check you have in place, there are other stopgaps. For my point of view, we don't need the referer check, and others here have already stated that they consider the referer check unnecessary for CSRF prevention, possibly even dangerous. So, the patch from #32 achieves. If the patch is not acceptable, I'm afraid you'll need to be more detailed than that, about what *exactly* you're afraid about and why.

Given that you're not a django site (but luckily still a Python site), I've posted the django link not as patch, but as an example of CSRF prevention without referer.

FWIW, there is a little bit of progress here: The error message no longer comes as separate page, but as a dialog. This means that I no longer lose my comment when I forget to (or don't know to) enable referer and post. (However, the error message is now buggy: it shows a full HTML source as error message. Still, an improvement.) Losing my comment drove me mad.

> I'm afraid you'll need to be more detailed than that, about what *exactly* you're afraid about and why.

Actually, nevermind. You basically stated that if somebody wanted to convert any other CSRF protection to Zope / Launchpad, e.g. the one I posted or another, that would be acceptable.

Any takers?

I think it is worth mentioning that django enforces 'strict' referrer checking for secure (https) requests. Why is this important ? - because if a cookie backed implementation[0] is used then subdomain or other cookie 'tossing'[1] is made harder if not impossible.

[0] a random token stored in a cookie
[1] https://github.com/blog/1466-yummy-cookies-across-domains

It's shame this has been allowed to continue for 4 years. It prevents sending bug collection data to launchpad from Ubuntu Servers, for example, since servers do not have a GUI and the default CLI browser (lynx) somehow doesn't correctly pass the REFERER header after passing through the convoluted Ubuntu One log-in single-sign on process.

apport-collect therefore fails at the stage:

Authorize application to access Launchpad on your behalf

Confirm Computer Access

   The Ubuntu computer called caddy wants access to your Launchpad account. If you allow this, every application running on caddy will have read-write access to your Launchpad account, including to your private data.

   If you're using a public computer, if caddy is not the computer you're using right now, or if something just doesn't feel right about this situation, you should choose "Do Not Allow 'caddy' to Access my Launchpad Account", or close this window now. You can always try again later.

   Even if you decide to give caddy access to your Launchpad account, you can change your mind later.

   Allow caddy to access my Launchpad account:

   Until I Disable It For One Hour For One Day For One Week

Any Python developer here who want to help porting Launchpad to another CSRF package that doesn't use referers?

After speaking with wgrant on IRC (thanks!), there's a simple solution:
Chrome sends the "Origin:" HTTP header with all same-origin HTTP POST requests.
This isn't any privacy issue. It's also in line with the specs.
Launchpad would be able to check the Origin header. If it's there and matches, Launchpad wouldn't need to check the referer. This would solve the issue at least for Chrome - and Firefox, if we can make Firefox do the same. https://bugzilla.mozilla.org/show_bug.cgi?id=446344

How many years will be the launchpad.net still be hampered by this referrer necessity? For more than a decade I have disabled the referrer transfer and the whole internet works fine except for this site.

Until someone contributes a patch to do CSRF differently. LP is entirely standards compliant in the current implementation - http://tools.ietf.org/html/rfc7231#section-5.5.2 - using referrer within a site to prevent user agents being tricked into harmful actions cross-site is perfectly legitimate.

It just says "Some servers do that", not that it's legitimate. RFC 7231 upholds the warning about the damage that the Referer can do.

Change this requirement. Drop the referrer need.

Improve! I have not brought more than 10 contributors I personally know just because of this silly requirement that no other similar system with USABILITY priorities has. And probably the number of "will not contribute people" will increase in a few hours, just because of this "white need" that does not really help the community.

I also have not made eventual contributions because I would need to log in (!), and keep the referrer configuration on as I used the website after logging in. Well... no big surprise that some projects considered moving from Launchpad.

I didn't bother reading the entire thread, but relying on Referer header instead of proper cookies usage or server-side session management bothers me.

Change this requirement please. There are better ways. Drop the referrer need.

I've finally found a workaround for this.
In Firefox set "network.http.referer.spoofSource" to "true". This will allow you to use launchpad without sacrificing your privacy.

More info:
https://twitter.com/KirilsSolovjovs/status/655794897295929344
http://02.lv/f/2015/10/18/ref.png

Firefox's spoofSource is a very dangerous option that should not exist (or at least not for several years, until Origin is widely deployed).

The correct settings are network.http.referer.trimmingPolicy=2, network.http.referer.XOriginPolicy=1, and optionally network.http.sendRefererHeader=1. That configuration sends Referer only for internal links within a site, and doesn't include path information.

Launchpad can't even handle correctly this case (client not sending referer). Please see attached screenshot.

Firefox also sends "Origin" in the latest releases.

Could you stop requiring the Referer header if "Origin" is present?

Congratulations. More than one decade with this obsolete Referer mechanism is reached now.

Unbelievable. If you think requiring HTTP Referer header addresses security concerns, you should not be a programmer! And by requiring such crap, you are pushing away the very sort of customer base who gravitates to FOSS, namely, people who know about computers and who care about privacy. It's simply embarrassing. By doing this, the developers show they are unqualified.