Launchpad itself

Uploading a tar.gz or uncompressed tar file to https://launchpad.net/products/picard/main/+pots/picard/+upload fails

Bug #44773 reported by Andrew Bennetts
8
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Medium
Christian Reis
Python
Fix Released
Unknown

Bug Description

OOPS-133C192

This appears to be a bug in Python's tarfile module. If a gzipped tar file is passed to TarFile.open using the fileobj argument, and the order of the keys of the tarfile.TarFile.OPEN_METH dictionary returns "bz2" before "gz", then "ValueError: no support for external file objects" is raised.

On my i386-architecture laptop, the dictionary-ordering luckily avoids this bug, but I'm guessing that for some reason it's different on Launchpad's production servers (64-bit?).

We can work around by detecting gzip files ourselves (with "content.startswith('\037\213')") and explicitly calling gzopen instead of open.

See original description
Revision history for this message
Andrew Bennetts (spiv) wrote :

Users can probably work around this by uploading tar.bz2 files instead. We already explicitly check for bz2, and so avoid the problematic code.

Changed in launchpad:
assignee: nobody → carlos
Stuart Bishop (stub)
description: updated
Revision history for this message
Lukáš Lalinský (luks) wrote :

Nope, switching to bz2 doesn't help.

tar.bz2:
OOPS-135A37

tar:
OOPS-135A38

Revision history for this message
Andrew Bennetts (spiv) wrote :

Well, I was half-right. tar.bz2 files do take a different code path, but that also seems to be broken.

OOPS-135A37 seems to be a new bug. tar.bz2 has worked in the past (see bug 1982), but this exception seems new. How did you generate the tar.bz2 file? Does "tar tjvf po.tar.bz2" work on it? Can you provide a link to the file?

OOPS-135A38 is the same bug that affects .tar.gz.

Revision history for this message
Andrew Bennetts (spiv) wrote :

I've filed a bug on the Python tarfile.py module: http://python.org/sf/1488634

Revision history for this message
Lukáš Lalinský (luks) wrote :

Yes, "tar tjvf po.tar.bz2" works fine. Here is the file: http://users.musicbrainz.org/~luks/po.tar.bz2

Revision history for this message
Lukáš Lalinský (luks) wrote :

I'm not sure if I just had luck, but it seems to be fixed now.

Revision history for this message
Diogo Matsubara (matsubara) wrote :

A recent occurrence OOPS-604E1842

Revision history for this message
Charles Edward Pax (pax) wrote :

I just had a bug reported on Gladex; #138983 "tar is not zipped" (https://bugs.launchpad.net/gladex/+bug/138983). Is 138983 a dupe of 44773?

Matthew Paul Thomas (mpt)
Changed in rosetta:
status: New → Confirmed
Revision history for this message
Christian Reis (kiko) wrote :

There seems to be some confusion regarding this bug, so I'll put my findings here.

1. Andrew is right that the bug in the default ordering of the dictionary affects us; the right way to fix this is to handle .gz, .tar and .bz2 files explicitly in the code, invoking the right classmethod directly, or supplying "r|xxx" to tarfile.open.

2. Python2.4's bzopen method doesn't support external file objects; they need to be files. However, if you supply them to open() they are meant to work, because they are then wrapped in a tarfile._Stream.

3. I can't seem to understand what went wrong in OOPS-135A37. It's strange: we tried to open it as a bzip2 file (note the "r|bz2" in the traceback), but the file inside the bz2 wrapper seems to have been empty. We shouldn't error out in this case, of course, but it seems that we do. But why was the file empty?

Revision history for this message
Christian Reis (kiko) wrote :

Ah, for 3, it's not that the file is empty. It's truncated. I have been able to simulate this in our tree.

Bug Watch Updater (bug-watch-updater)
Changed in python:
status: Unknown → Fix Released
Revision history for this message
Christian Reis (kiko) wrote :

Landed in RF 5637. Note that this can all be cleaned up when we move to Python2.5 and there are XXXs saying so in the code.

Changed in rosetta:
assignee: carlos → kiko
milestone: none → 1.2.2
status: Confirmed → Fix Committed
Revision history for this message
Christian Reis (kiko) wrote :

Released, though it'd be nice to clean this up when py2.5 shows up.

Changed in rosetta:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.