TestBrowser Referer: header set to 'localhost'

Hi,

thanks for your report. We're sorry that it took so long to reply.

I've tried to reproduce the behavior but wasn't successful. In my case the request that is created by testbrowser didn't allow access to 'getHeader' at all.

Can you create a test case that demonstrates the behavior? If you can't get a test case, I'd be happy with explicit reproduction steps too.

Just thought I'd add a bit of info here. I was running Zope 2.9.6 and wrote a functional test using zope.testbrowser which submitted a form. During the submittal of that form the custom zope code checks request header HTTP_REFERER to figure out where to redirect when it's done. The value of that header was 'localhost' as mentioned in this bug report.

There are three places in zope/app/testing/functional.py that set default values for HTTP_REFERER:
 * BrowserTestCase.makeRequest()
 * HTTPTestCase.makeRequest()
 * HTTPCaller.__call__()

(this last one is the one that affects TestBrowser).

This is bad for two reasons:
 1. it is impossible to test behaviour when no referrer is passed.
 2. the "Referer" header is meant to be sent to a URL, not a hostname.

On 7/9/07, James Henstridge <email address hidden> wrote:
> This is bad for two reasons:
> 1. it is impossible to test behaviour when no referrer is passed.
> 2. the "Referer" header is meant to be sent to a URL, not a hostname.

It seems setting the default should simply be ripped out. testbrowser
can set the header as appropriate in the click() method of links and
submit controls.

From what I can see, TestBrowser (through mechanize) does correctly set the "Referer:" header in its requests when appropriate (following links, submitting forms, etc).

So the only case where the problem is exhibited is when TestBrowser does not send a "Referer:" header (i.e. browser.open()).

This still happens to me with Zope 2.10 (3.3).

It is definitely not just a problem with pages just opened with browser.open(). In this case, I got to this page via a series of browser.getControl(...).click()'s, landing me on a page that checked HTTP_REFERER and indeed it's set to 'localhost'.

The test in question is at http://dev.plone.org/plone/browser/plone.app.contentrules/trunk/plone/app/contentrules/tests/multipublish.txt. The try... except block near the end is masking a 404 that results from a redirect that tries to go to a non-existing /localhost page.

There are files in both Zope2 and Zope3 that produce this bug.

http://svn.zope.org/Zope/trunk/lib/python/Testing/ZopeTestCase/zopedoctest/functional.py?view=auto
http://svn.zope.org/zope.app.testing/trunk/src/zope/app/testing/functional.py?view=auto

As noted, the problem isn't in testbrowser. As a workaround, you can open the testbrowser in some URL that won't be affected by the broken HTTP_REFERER value and then use browser.getLink to navigate from there to the URL you need to test provided, of course, that a link is available on the page initially loaded.

While Ross's suggestion lets you load pages that would otherwise fail on nonsense "Referer" header values (which should probably also be classed as a bug in such pages), it does not help you test what happens when no header is sent at all.

I had an encounter with this one, when I tried to browser test Quickinstaller Control Panel of Plone. The Quickinstaller will redirect to the referrer send with the posted form request when a Product get's installed. I took a dive into the matter and here is what I found out (quoting a blog post I wrote on that matter; excuse the prose.):

There are various places inside the Zope testing framework working together to cause this weird behaviour. First there is zope.app.testing.functional which sets HTTP_REFERER to the nonsense value 'localhost'. As defined by RFC 2616, Section 14.36, the referrer is either a absolute or a relative URI. In this case it would be a relative URI, but one that simply makes no sense. Even if it was 'http://localhost/' it would not, because in most cases this would not be the correct referrer! This header value should be better left unset. After all, a web service might reckon with a suppressed referrer, but not with a misleading.

Removing those references will not solve the matter, however. The true problem is that zope.testbrowser.browser will actually never send a referrer for any submitted form. Only when it follows links, a referrer gets send along.

Zope's testbrowser builds on top the programmatic web browsing framework Mechanize. It wraps this framework completely, and unfortunately differently. The difference which is relevant here is the representation of HTML forms. Zope testbrowser offers objects for each HTML element needed for browser, i.e. links and various controls; also for forms. With the Mechanize library however the user selects a form with Browser and then fills it. The test-browser by-passes this part of the Mechanize API completely. It generates the HTTP request for posting the form with ClientForm.HTMLForm, then gives it to Browser.open of Mechanize. While this operation ultimately handles all web access, it does not set a referrer. It simply cannot: there is no way to tell whether the last call to open was related to the current call. That is why Mechanize sets the referrer only on clicks, submits and when following links, as mechanize.useragent.HTTPRefererProcessor explains.

Fixing this seems pretty easy, though I gave it no thorough testing yet. All one has to do is change zope.testbrowser.Browser._clickSubmit from

>>> self.mech_browser.open(form.click(
... id=control.id, name=control.name, label=label, coord=coord))

to

>>> self.mech_browser.form = form
>>> self.mech_browser.submit(id=control.id, name=control.name,
... label=label, coord=coord)

The only problem I see is concurrency safety, though I doubt that has any relevance here.

Given that testbrowser/mechanize are meant to simulate a user interacting with a single window/tab web browser (which is inherently a linear process), there probably isn't any concurrency issues to worry about.

If a test is simulating concurrent access to a site, it'd be using multiple browser instances.

Here is a patch for tests.py of zope.testbrowser for this issue. It was made from the version that ships with Zope 2.10.6, but with a fuzz of 3 it should apply to trunk as well.

Note that I changed almost all of the other test cases as well, because they will get a referrer where they expect none (once this is fixed at least). Since these test cases where over-specified IMO, I made heavy use of the ELLIPSIS, so that only the actually relevant part of output will be matched.

I've added a monkey patch for this and a bug viewing contents streamed to the response in collective.testcaselayer 1.2. Just include it's configure.zcml

Here's what I plan to do:

- ripping out the default referer seems reasonable
- write a set of tests that ensure how the referer evolves when using various test browser methods (from the testing browser)
- add an option to testbrowser to suppress referer headers completely

That should bring us to a situation that more closely resembles RFC 1945 (HTTP 1.0)

Fix committed for zope.app.testing 3.7 in revision 111343. The functional test helpers now do not introduce a (bad) default referer anymore.

I applied Jan Hackel's +ELLIPSIS cleanups to the zope.testbrowser trunk:

 http://svn.zope.org/zope.testbrowser/trunk/?rev=111377&view=rev

and then put his new test, along with Ross Patterson's suggested fix, on a branch:

  http://svn.zope.org/zope.testbrowser/branches/lp-93437/

The tests all pass on the branch, although I am not sure it is exactly what Christian had in mind, so I haven't merged it to the trunk.

This issue will be fixed by the release of a new version of zope.testbrowser. No fix to Zope2 itself is required.

It wasn't what I had in mind, but I also didn't notice the recommended change by Ross Patterson. Unfortunately my experimental changes from last bug day didn't make it to my working branch, so I'll just step back for now and would be happy if you merge that branch. I'll close mine.

Fix will be released with zope.testbrowser 3.9.1.