python-setuptools

easy_install will install a package that is already there

Bug #254035 reported by Zooko Wilcox-O'Hearn
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Tahoe-LAFS
Fix Released
Unknown
distribute
New
Undecided
Unassigned
nevow
Invalid
Unknown
python-setuptools
Unknown
Unknown
nevow (Debian)
Fix Released
Unknown
nevow (Ubuntu)
Fix Released
Undecided
Vincent Bernat
python-setuptools (Debian)
Fix Released
Unknown

Bug Description

Binary package hint: python-setuptools

This bug in Debian sid:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440

Has now been added to Ubuntu, in Hardy. I confirmed that the bug stops happening when you uninstall the python-setuptools package that comes with Ubuntu Hardy and install the same version of python-setuptools from upstream then the problem does not happen.

See original description
Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

fixed by some subset of the patches between [1718] and [1739] (inclusive on both ends).

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

I'm seeing this problem occur again, on my home machine running
debian/unstable. After I upgraded to the debian packaging of 0.6c8 (from
0.6c7), the tahoe build process started downloading and building a Nevow
tarball, even though there's a local copy already installed (with a
valid-looking .egg-info file) on sys.path .

http://allmydata.org/buildbot/builders/speed-DSL/builds/315/steps/compile/logs/stdio
is an example of the build process with setuptools-0.6c8, in which Nevow is
downloaded from divmod.org and compiled.

http://allmydata.org/buildbot/builders/speed-DSL/builds/312/steps/compile/logs/stdio
is an example of a build which failed because the local debian package of
nevow-0.9.29 was detected, which locked the target version at 0.9.29, and
setuptools then tried to download an 0.9.29 tarball (which failed because by
that point divmod.org was hosting a newer version, and so I presume they had
removed the old tarball).

http://allmydata.org/buildbot/builders/speed-DSL/builds/311/steps/compile/logs/stdio
is the first build with the new setuptools-0.6c8, doing the extra
download-and-build step.

http://allmydata.org/buildbot/builders/speed-DSL/builds/310/steps/compile/logs/stdio
is the last build with the old setuptools-0.6c7, in which it correctly
refrains from downloading or building Nevow.

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

Here's a test which could show whether setuptools/easy_install is working on your system:

Find a Python package which is already installed, such as Nevow.

Now run {{{sudo easy_install nevow}}}. If it downloads anything, then your {{{easy_install}}} is broken.

Alternately, you could get minproj-2.0.0.tar.gz.
This is a project that has {{{install_requires=["nevow"]}}} in it. If running {{{sudo ./setup.py install}}} downloads anything, then your {{{setuptools}}} is broken.

You can do this without sudo, by setting the PYTHONPATH and using --prefix...

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

On my sid workstation ('fluxx'), I did the following commands, and observed
setuptools download nevow from divmod.org, despite nevow-0.9.31 already being
installed such that 'import nevow' worked inside a python shell.

{{{
HERE = ~/trees/tahoe-mutable-205-312/t

go:
 mkdir -p $(HERE)/tmp/lib/python2.4/site-packages
 PYTHONPATH=$(HERE)/tmp/lib/python2.4/site-packages easy_install --prefix=$(HERE)/tmp nevow
}}}

I will see if I can find a machine on which this same test does *not* download anything, and then compare the setuptools/easy_install installations.

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

Here's a related question: is Nevow installed on fluxx in such a way that pkg_resources recognizes it and its version number? Try this:

{{{
python -c 'import pkg_resources;print pkg_resources.require("nevow")'
}}}
and
{{{
python -c 'import pkg_resources;print pkg_resources.require("nevow == 0.9.31")'
}}}

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

yes, I believe so:
{{{
1293:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow")'
[Nevow 0.9.31 (/var/lib/python-support/python2.4)]
1294:warner@fluxx% python -c 'import pkg_resources; print pkg_resources.require("nevow == 0.9.31")'
[Nevow 0.9.31 (/var/lib/python-support/python2.4)]
}}}

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

This has been ticketed on [http://bugs.python.org/setuptools/issue17 the setuptool s issue tracker issue17] and [http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440 debian bug report 475440].

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

For reference, this is still broken on my debian/sid system: Nevow-0.9.31 is downloaded and built even though pkg_resources.require("nevow") indicates that 0.9.31 is available in /var/lib/python-support/python2.5 .

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

we're also seeing this on a hardy box

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

Confirmed that this problem happens on our new Hardy server, and also I confirmed that it stops happening when you uninstall the python-setuptools package from Ubuntu and instead install setuptools from upstream directly. I'll update the Debian bug report and open a sibling bug report for Ubuntu...

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Binary package hint: python-setuptools

This bug in Debian sid:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440

Has now been added to Ubuntu, in Hardy. I confirmed that the bug stops happening when you uninstall the python-setuptools package that comes with Ubuntu Hardy and install the same version of python-setuptools from upstream then the problem does not happen.

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

Added a bug report for Ubuntu:

https://bugs.launchpad.net/debian/+source/python-setuptools/+bug/254035

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Here is the issue ticket on allmydata.org where we first found and diagnosed this problem, and where we are tracking the problems it is causing for allmydata.org Tahoe:

http://allmydata.org/trac/tahoe/ticket/229 # debian sid packaging of setuptools v0.6c8 broke the "Desert Island" build

Bug Watch Updater (bug-watch-updater)
Changed in allmydata.org:
status: Unknown → New
Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

Ah, it turns out that this is ''not'' caused by the Debian/Ubuntu packaging! It happens also with the upstream setuptools. I'm not sure why I thought it didn't (as earlier posted in this ticket).

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

This appears to be a bug in upstream setuptools, not in the Debian/Ubuntu
packaging. I earlier thought that I had confirmed that it did *not* happen with
pristine setuptools, but just now I tried to reproduce it and it happens with
pristine setuptools. Also a Debian developer tried to reproduce it on his
Debian-sid/lenny system and the problem did not manifest there!

I suspect that it might have something to do with
http://bugs.python.org/setuptools/issue20

Bug Watch Updater (bug-watch-updater)
Changed in python-setuptools:
status: Unknown → New
Bug Watch Updater (bug-watch-updater)
Changed in allmydata.org:
status: New → Confirmed
Revision history for this message
In , warner (warner-tahoe-trac) wrote :

still happening on a hardy box (deharo2). My workaround was to uninstall tahoe+nevow, thus allowing the buildslave to download+compile nevow on each build.

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

By the way, I suspect that #455 (setuptools sandbox isn't tight enough to hold Twisted's sand in so that it doesn't get in Nevow's eyes) might be an underlying cause of this, or at least might be related.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Okay, PJE explained on the setuptools issue tracker that the .egg-info file needs to be in the sys.path. Currently the .egg-info file in the Debian/Ubuntu package is located in:

zooko@deharo2:~$ dpkg --listfiles python-nevow | grep -i egg
/usr/share/python-support/python-nevow/Nevow-0.9.26.egg-info

But /usr/share/python-support/python-nevow is not in the sys.path. Therefore setuptools doesn't see the .egg-info file when it searches the sys.path. so it doesn't think that Nevow is installed.

Bug Watch Updater (bug-watch-updater)
Changed in nevow:
status: Unknown → New
Revision history for this message
Brian Warner (warner) wrote :

Aha, not so fast. We have two separate issues here, and neither one of them is resolved in Ubuntu:

1: python-nevow in Hardy puts the .egg-info file in the wrong place, and additionally setuptools fails to recognize that it is already installed, so 'easy_install' will redownload it. The most recent debian/sid packaging switched python-nevow to use pycentral (instead of pysupport), which puts the .egg-info file in the right place. Setuptools now does the right thing on sid. So one problem is that Hardy does not have this change from sid. This is a backporting issue.

2: python-simplejson has the same problem (setuptools fails to recognize that it is already installed, and redownloads it) on both sid and hardy. In this case, the .egg-info directory *is* on sys.path, so PJE's explanation seems to not be applicable. simplejson is packaged with python-support.

For simplejson, I did some experiments, to isolate which difference between the two packages was
causing the problem:

 * replacing the .egg-info/ directory with an .egg-info file didn't help
 * moving the .egg-info/ directory from /var/lib/python-support/python2.5/ to /usr/lib/python2.5/site-packages/ *does* help

My hypothesis is that the /var/lib directory is added to sys.path fairly late, perhaps by some site.py or *.pth file, and that setuptools is not looking in these directories properly.

So I think that there is still a problem with setuptools, possibly related to python-support.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

See http://bugs.python.org/setuptools/issue17 where the author of setuptools, PJE, is investigating whether there is also an actual setuptools bug involved.

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Okay if you add "--site-dirs=/var/lib/python-support/python2.5" then this fixes this problem for both Nevow and simplejson on Hardy. PJE is investigating fixing this in a future release of setuptools.

In the meantime, he suggests that it might be a good fix for Debian/Ubuntu to add the following to the system-wide distutils.cfg file:

zooko@deharo2:~$ cat /usr/lib/python2.5/distutils/distutils.cfg
[easy_install]
site-dirs=/var/lib/python-support/python2.5

I've tested this on my Hardy box and it does indeed cause "easy_install" to do the right thing even without the "--site-dirs=/var/lib/python-support/python2.5" option on the command-line.

Revision history for this message
Scott Kitterman (kitterman) wrote : Re: [Bug 254035] Re: easy_install will install a package that is already there

How many packages does this affect?

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

According to http://bugs.python.org/setuptools/issue17 the bug is "that easy_install does NOT search sys.path for a requirement that is
specified on the command line; it only searches "site" directories for such".

Therefore, any package which has a .egg-info which is not in the "site" directories is probably affected.

Fortunately all such packages can be fixed at once by adding the above-mentioned site-dirs config to the system-wide distutils config file.

Revision history for this message
Scott Kitterman (kitterman) wrote :

Wouldn't it make more sense to fix easy_install than to change a number of
packages?

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

This would be a change just to the setuptools package -- it would not require any change to any of the affected packages. The change would be to have the setuptools package add the following to the system-wide distutils config file:

[easy_install]
site-dirs=/var/lib/python-support/python2.5

Presumably there will be a deeper patch to setuptools/easy_install in the future, but I thought that this one would be an easy change to make right now.

If that doesn't seem like a good idea to the Debian/Ubuntu packagers then I'll try to think of an alternative fix.

Bug Watch Updater (bug-watch-updater)
Changed in nevow:
status: Unknown → New
Bug Watch Updater (bug-watch-updater)
Changed in nevow:
status: New → Invalid
Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

So, the proposal is that the Debian and Ubuntu package of setuptools v0.6c8 is changed to add the following to the systemwide distutils config file:

[easy_install]
site-dirs=/var/lib/python-support/python2.5

How does that sound to the Debian and Ubuntu python teams?

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

The same problem was causing build failures via python-openssl, since 0.6 is not being distributed anymore (or it's been removed from certain sf.net mirrors, or sf.net is having connectivity problems). In [2931] I added a workaround: just add --site-dirs=/var/lib/python-support/python2.5/ to the 'setup.py develop' command line. This should fix the problem on debian-like systems and not cause any ill effects on other platforms.

I expect that this ought to cause our buildslaves to stop trying to build nevow and openssl when they are already installed.

Revision history for this message
In , robk (robk-tahoe-trac) wrote :

so I had a very similar problem today, on the mac

it turned out that easy_install had at some point installed twisted into /Library, and somehow tahoe (specifcally make test) was pulling in twisted from /System/Library (the apple supplied version) which wound up barfing on an error it got whilst cross-importing bits of twisted from out of /Library. I removed the twisted egg etc from /Library, but also needed to add

 "--site-dirs /System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python"

to the build args in order to get tahoe's build to not try and install a different version of twisted and instead use the extant system one. I'll submit a patch to that end shortly.

Revision history for this message
In , warner (warner-tahoe-trac) wrote :

Note that we had to add a {{{--site-dirs}}} argument to get setuptools to recognize the python modules that get installed (on debian) into a /var/lib directory that is on sys.path but isn't one of the traditional places.

Take a look at setup.py for the place to add it. It should be harmless to add another one.

Revision history for this message
Matthias Klose (doko) wrote :

please use the workaround to configure packages with --site-dirs. it's unfortunate that python-support chooses not to install in sys.path.

Changed in python-support:
importance: Undecided → Low
status: New → Triaged
Bug Watch Updater (bug-watch-updater)
Changed in nevow:
status: New → Won't Fix
Changed in python-setuptools:
status: New → Won't Fix
Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Okay, the Tahoe setup.py now has a couple of work-arounds for this issue on two platforms:

http://allmydata.org/trac/tahoe/browser/setup.py?rev=20080924174255-80e44-63903e9efb0c9b8a6f58cedfbc96ecb6f352837a#L15

15 basedir = os.path.dirname(os.path.abspath(__file__))
16 pyver = "python%d.%d" % (sys.version_info[:2])
17 if sys.platform == "win32":
18 supportlib = os.path.join("support", "Lib", "site-packages")
19 else:
20 supportlib = os.path.join("support", "lib", pyver, "site-packages")
21 supportlib = os.path.join(basedir, supportlib)

Presumably we'll add similar kludges when this problem crops up on other platforms. :-/

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

Note that this means only Tahoe is safe from easy_install installing a package when it is already there. Other packages will still experience this bug. If Debian or Ubuntu folks would put

[easy_install]
site-dirs=/var/lib/python-support/python2.5

into the system-wide site-dirs then no packages would have this problem.

Revision history for this message
In , zooko (zooko-tahoe-trac) wrote :

Well, I guess this is fixed by our {{{--site-dirs}}} hack. Not very satisfying (I would really prefer it if setuptools would search the {{{sys.path}}} for requirements per http://bugs.python.org/setuptools/issue17 ), but the allmydata-tahoe-relevant portion of this issue is fixed. Right?

Bug Watch Updater (bug-watch-updater)
Changed in allmydata.org:
status: Confirmed → Fix Released
Revision history for this message
Vincent Bernat (vbernat) wrote :

This has been fixed in nevow 0.9.31-2 by switching to python-central

Changed in nevow:
assignee: nobody → bernat-debian
status: New → Fix Released
Revision history for this message
Jochen Schroeder (cyco) wrote :

I'd just like to comment that the issue with python-setuptools still isn't resolved yet. When I try to install a package that depends on python-pyparsing I get the same issue, easy_install is trying to install a new version of pyparsing from pypi. Pyparsing is located in /usr/share/pyshared. Should I report this as a pyparsing bug or python-setuptools bug

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

So let's summarize what we know of this issue. It occurs with all packages that are installed by python-support, right? And the underlying cause is that python-support is installing the package's .egg-info into a directory which may or may not be on the sys.path but which is not a "site-dir". Apparently there is a list of site-dirs, and if your .egg-info is not in a directory on that list then you can experience this bug. The maintainers of setuptools therefore suggested in [1] that if Debian/Ubuntu is going to be installing Python libraries into a directory $SOMEDIR, then they should also configure the system-wide distutils config file to add:

[easy_install]
site_dirs = $SOMEDIR

However, Matthias Klose says in [2] that he does not think that /var/lib/python-support/python$VER should be added to the site-dirs and instead that the maintainers of python-support should change the way it behaves so that Python libraries get installed into the sys.path instead.

What do the maintainers of python-support say? Are they even aware of this request? Would it be okay, oh maintainers of python-support, if we changed python-support to install Python libraries into the sys.path, as Matthias Klose suggested? Or if not, is there another way that we can fix this bug?

Thanks, folks.

[1] http://bugs.python.org/setuptools/msg402
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475440#40

Revision history for this message
Zooko Wilcox-O'Hearn (zooko) wrote :

http://bitbucket.org/tarek/distribute/issue/142/easy_install-will-install-a-package-that-is-already

Adolfo Jayme Barrientos (fitojb)
no longer affects: python-support (Ubuntu)
Bug Watch Updater (bug-watch-updater)
Changed in nevow (Debian):
status: Won't Fix → Fix Released
Changed in python-setuptools (Debian):
status: Won't Fix → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.