bzr+http does not send x-www-form-urlencoded data (but pretends to)

Bug #665100 reported by Adrien Saladin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
High
Vincent Ladeuil
2.2
Fix Released
Medium
Vincent Ladeuil
bzr (Ubuntu)
Fix Released
Undecided
Unassigned
Maverick
Fix Released
Undecided
Unassigned

Bug Description

Hi,

I have found what looks like a bug in http transport. POST data transmitted from bzr client to http server pretends to be x-www-form-urlencoded but looks like octet/stream.

Below is a tcp dump of a bzr+http session. Not very readable in a browser but the point is that post data contains for example ascii spaces instead of '+' and so on.

This impacts interfacing the bzr wsgi application with some web frameworks that really expects data to be form-urlencoded when client says so.

0000 50 4f 53 54 20 2f 62 61 7a 61 61 72 2f 66 6f 6f POST /bazaar/foo
0010 62 61 72 2f 2e 62 7a 72 2f 73 6d 61 72 74 20 48 bar/.bzr/smart H
0020 54 54 50 2f 31 2e 31 0d 0a 41 63 63 65 70 74 2d TTP/1.1..Accept-
0030 45 6e 63 6f 64 69 6e 67 3a 20 69 64 65 6e 74 69 Encoding: identi
0040 74 79 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 ty..Content-Leng
0050 74 68 3a 20 32 31 38 0d 0a 43 6f 6e 6e 65 63 74 th: 218..Connect
0060 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 76 65 0d ion: Keep-Alive.
0070 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 55 73 .Accept: */*..Us
0080 65 72 2d 41 67 65 6e 74 3a 20 62 7a 72 2f 32 2e er-Agent: bzr/2.
0090 31 2e 32 20 28 75 72 6c 6c 69 62 29 0d 0a 48 6f 1.2 (urllib)..Ho
00a0 73 74 3a 20 6c 6f 63 61 6c 68 6f 73 74 3a 35 30 st: localhost:50
00b0 30 30 0d 0a 50 72 61 67 6d 61 3a 20 6e 6f 2d 63 00..Pragma: no-c
00c0 61 63 68 65 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 ache..Cache-Cont
00d0 72 6f 6c 3a 20 6d 61 78 2d 61 67 65 3d 30 0d 0a rol: max-age=0..
00e0 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 Content-Type: ap
00f0 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 77 2d plication/x-www-
0100 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 64 0d form-urlencoded.
0110 0a 0d 0a ...

[server ack]

Second packet: from bzr client to http server.
0000 62 7a 72 20 6d 65 73 73 61 67 65 20 33 20 28 62 bzr message 3 (b
0010 7a 72 20 31 2e 36 29 0a 00 00 00 1c 64 31 36 3a zr 1.6).....d16:
0020 53 6f 66 74 77 61 72 65 20 76 65 72 73 69 6f 6e Software version
0030 35 3a 32 2e 31 2e 32 65 73 00 00 00 27 6c 32 39 5:2.1.2es...'l29
0040 3a 52 65 70 6f 73 69 74 6f 72 79 2e 69 6e 73 65 :Repository.inse
0050 72 74 5f 73 74 72 65 61 6d 5f 31 2e 31 39 31 3a rt_stream_1.191:
0060 2e 30 3a 65 62 00 00 00 2a 42 61 7a 61 61 72 20 .0:eb...*Bazaar
0070 70 61 63 6b 20 66 6f 72 6d 61 74 20 31 20 28 69 pack format 1 (i
0080 6e 74 72 6f 64 75 63 65 64 20 69 6e 20 30 2e 31 ntroduced in 0.1
0090 38 29 0a 62 00 00 00 3b 42 35 34 0a 0a 42 61 7a 8).b...;B54..Baz
00a0 61 61 72 20 72 65 70 6f 73 69 74 6f 72 79 20 66 aar repository f
00b0 6f 72 6d 61 74 20 32 61 20 28 6e 65 65 64 73 20 ormat 2a (needs
00c0 62 7a 72 20 31 2e 31 36 20 6f 72 20 6c 61 74 65 bzr 1.16 or late
00d0 72 29 0a 62 00 00 00 01 45 65 r).b....Ee

Related branches

Revision history for this message
Vincent Ladeuil (vila) wrote :

Could you re-try with the '-Dhttp' option added to your bzr client command ?
This should output the request/answer in ~/.bzr.log

AFAIK, bzr doesn't use x-www-form-urlencoded, ever.

Changed in bzr:
status: New → Incomplete
Revision history for this message
Adrien Saladin (adrien-saladin) wrote :

sure:

ven. 2010-10-22 19:44:09 +0200
0.038 bazaar version: 2.1.2
0.038 bzr arguments: [u'log', u'bzr+http://localhost:5000/bazaar/foo', u'-Dhttp']
0.053 looking for plugins in /home/adrien/.bazaar/plugins
0.107 looking for plugins in /usr/lib/python2.6/dist-packages/bzrlib/plugins
0.107 Plugin name fastimport already loaded
0.135 encoding stdout as sys.stdout encoding 'UTF-8'
0.170 bzr-svn: using Subversion 1.6.12 ()
0.200 * About to connect() to localhost:5000
0.201 > POST /bazaar/foo/.bzr/smart
0.201 > Content-Length: 6
> Connection: Keep-Alive
> Accept: */*
> User-Agent: bzr/2.1.2 (urllib)
> Host: localhost:5000
> Pragma: no-cache
> Cache-Control: max-age=0
> Content-Type: application/x-www-form-urlencoded

0.206 < HTTP/1.0 200 OK
0.206 < Server: PasteWSGIServer/0.5 Python/2.6.6
< Date: Fri, 22 Oct 2010 17:44:09 GMT
< Content-type: application/octet-stream
< Content-Length: 5

0.208 * About to connect() to localhost:5000
0.213 > POST /bazaar/foo/.bzr/smart
0.213 > Content-Length: 85
> Connection: Keep-Alive
> Accept: */*
> User-Agent: bzr/2.1.2 (urllib)
> Host: localhost:5000
> Pragma: no-cache
> Cache-Control: max-age=0
> Content-Type: application/x-www-form-urlencoded

0.214 < HTTP/1.0 200 OK
0.214 < Server: PasteWSGIServer/0.5 Python/2.6.6
< Date: Fri, 22 Oct 2010 17:44:09 GMT
< Content-type: application/octet-stream
< Content-Length: 76

Revision history for this message
Vincent Ladeuil (vila) wrote :

Looks like the python urllib2 library is setting this header but we should be able to set it ourselves to avoid the issue.

Changed in bzr:
status: Incomplete → Confirmed
importance: Undecided → High
Revision history for this message
Vincent Ladeuil (vila) wrote :

I just checked and it's also the case with python-2.5 and 2.4, it's amazing we never encountered this one earlier...
May be just because nobody cares (at you noted in your initial report)...

Vincent Ladeuil (vila)
Changed in bzr:
assignee: nobody → Vincent Ladeuil (vila)
Revision history for this message
Vincent Ladeuil (vila) wrote :

Since the bug has been present from day one, we may want to backport it to all our stables releases. It's very isolated *and* tested and should merge cleanly. The question is: which releases do we target here.

Changed in bzr:
status: Confirmed → In Progress
Revision history for this message
Max Bowsher (maxb) wrote :

AFAICS it is totally benign unless combined with a non-standard WSGI container doing dubious munging of the input stream. Therefore, surely it's not worth backporting to earlier than the current stable series i.e. 2.2 ?

Revision history for this message
Martin Pool (mbp) wrote : Re: [Bug 665100] Re: bzr+http does not send x-www-form-urlencoded data (but pretends to)

On 26 October 2010 13:20, Max Bowsher <email address hidden> wrote:
> AFAICS it is totally benign unless combined with a non-standard WSGI
> container doing dubious munging of the input stream. Therefore, surely
> it's not worth backporting to earlier than the current stable series
> i.e. 2.2 ?

That logic makes sense to me.

--
Martin

Vincent Ladeuil (vila)
Changed in bzr:
milestone: none → 2.3b3
status: In Progress → Fix Released
Jelmer Vernooij (jelmer)
Changed in bzr (Ubuntu):
status: New → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote : Please test proposed package

Accepted bzr into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in bzr (Ubuntu Maverick):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Martin Pool (mbp) wrote :

SRU verification:

I was going to look at what was actually sent but bug 768141 interfered.

Looking at the -Dhttp log, I see bzr is sending the right content type in maverick-proposed 2.2.4-0ubuntu1

verification done

tags: added: http verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bzr - 2.2.4-0ubuntu1

---------------
bzr (2.2.4-0ubuntu1) maverick-proposed; urgency=low

  [ Jelmer Vernooij ]
  * Update watch file to use 2.2 series.
  * New upstream release.
   + Fixes closing of leaked sockets to SSH subprocesses, which causes
     dput sftp uploads to hang. LP: #659590
   + Fixes the use of 'lp:' urls behind a http proxy. LP: #558343
   + Correctly sets the Content-Type header when http POSTing to comply
     with stricter web frameworks. LP: #665100
   + Fixes propagating tags to the master branch in a bound branch or
     heavyweight checkout. LP: #603395
   + Fixes the use of 'bzr resolve --take-other' if the file is
     involved in an unresolved text conflict. LP: #646961
   + Fixes https access with newer versions of python2.7. LP: #693880
   + Fixes crash during pack caused by a concurrent repository pack
     operation. LP: #701940
   + Fixes communication with the Launchpad web service when using
     launchpadlib >= 1.5.5. LP: #707075
   + Switches away from deprecated 'edge.launchpad.net' LP: #583667
   + Fixes resolving of content (and path) conflicts for files in subdirs.
     LP: #660935
   + Fixes nasty recursion loop while displaying branch opening error.
     LP: #687653

  [ Martin Pool ]
  * Propose for maverick SRU.
 -- Martin Pool <email address hidden> Thu, 07 Apr 2011 15:30:17 +1000

Changed in bzr (Ubuntu Maverick):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.