C extensions placed in wrong directory on karmic

On karmic, with bzr.dev r4479

jml@truth:~/src/bzr/trunk$ make
building extension modules.
python setup.py build_ext -i
running build_ext
building 'bzrlib._bencode_pyx' extension
creating build
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/bzrlib
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_bencode_pyx.c -o build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o
bzrlib/_bencode_pyx.c:2414: warning: ‘__Pyx_GetItemInt’ defined but not used
bzrlib/_bencode_pyx.c:2429: warning: ‘__Pyx_SetItemInt’ defined but not used
creating /home/jml/src/bzr/bzr.dev/bzrlib/bzrlib
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o /home/jml/src/bzr/bzr.dev/bzrlib/bzrlib/_bencode_pyx.so
pyrexc bzrlib/_btree_serializer_pyx.pyx --> bzrlib/_btree_serializer_pyx.c
/usr/lib/python2.6/dist-packages/Pyrex/Compiler/Scanning.py:39: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
building 'bzrlib._btree_serializer_pyx' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_btree_serializer_pyx.c -o build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_pyx.o
bzrlib/_btree_serializer_pyx.c: In function ‘__pyx_f_6bzrlib_21_btree_serializer_pyx_safe_string_from_size’:
bzrlib/_btree_serializer_pyx.c:187: warning: cast from pointer to integer of different size
bzrlib/_btree_serializer_pyx.c: In function ‘__pyx_f_6bzrlib_21_btree_serializer_pyx_safe_interned_string_from_size’:
bzrlib/_btree_serializer_pyx.c:239: warning: cast from pointer to integer of different size
...

On karmic with 1.16,

jml@truth:~/src/bzr/bzr.1.16$ make
building extension modules.
python setup.py build_ext -i
running build_ext
building 'bzrlib._bencode_pyx' extension
creating build
creating build/temp.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/bzrlib
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_bencode_pyx.c -o build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o
bzrlib/_bencode_pyx.c:2414: warning: ‘__Pyx_GetItemInt’ defined but not used
bzrlib/_bencode_pyx.c:2429: warning: ‘__Pyx_SetItemInt’ defined but not used
creating /home/jml/src/bzr/bzr.1.16/bzrlib/bzrlib
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o /home/jml/src/bzr/bzr.1.16/bzrlib/bzrlib/_bencode_pyx.so
building 'bzrlib._btree_serializer_c' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c bzrlib/_btree_serializer_c.c -o build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_c.o
bzrlib/_btree_serializer_c.c: In function ‘__pyx_f_19_btree_serializer_c_safe_string_from_size’:
bzrlib/_btree_serializer_c.c:144: warning: cast from pointer to integer of different size
bzrlib/_btree_serializer_c.c: In function ‘__pyx_f_19_btree_serializer_c_safe_interned_string_from_size’:
bzrlib/_btree_serializer_c.c:200: warning: cast from pointer to integer of different s...

It's affecting 1.16 and trunk then:
(from jml's paste: bzrlib/bzrlib/_bencode_pyx.so)

I'm suspecting karmic.

John, I've subscribed you because the karmic PPA builds may need some love to Do The Right Thing.

It's a shame this has gone unfixed for a few months - it seems to affect all Python C extensions, and makes development a royal pain.

2009/8/26 cortesi <email address hidden>:
> It's a shame this has gone unfixed for a few months - it seems to affect
> all Python C extensions, and makes development a royal pain.

Do you mean all of them for bzr, or all across all python projects in Karmic?
--
Martin <http://launchpad.net/~mbp/>

I'm seeing this problem with C extensions in my own projects. An extension declaration like this:

       Extension(
            "module.extension",
            sources = ["module/extension.c"],
        )

Results in an object file at "module/module/extension.so" rather than "module/extension.so". The fact that this happens with all my projects and only on Karmic seems to show that there's been some modification to the Python build mechanism that's causing this. Also, note that Python 2.5 doesn't seem to have this problem - only Python 2.6.

Regards,

Aldo

Martin,

Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
happening with the bzr package in Ubuntu.

Thanks,

James

2009/8/26 James Westby <email address hidden>:
> Martin,
>
> Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
> happening with the bzr package in Ubuntu.

Because I thought it might be a problem there. If not, I'll mark it
invalid and apologize for the noise.

--
Martin <http://launchpad.net/~mbp/>

On Wed Aug 26 22:44:40 UTC 2009 Martin Pool wrote:
> 2009/8/26 James Westby <email address hidden>:
> > Martin,
> >
> > Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
> > happening with the bzr package in Ubuntu.
>
> Because I thought it might be a problem there. If not, I'll mark it
> invalid and apologize for the noise.

No problem, I just wondered if you knew something I didn't.

I made sure to check the contents of the .deb before uploading, and
they were definitely in bzrlib/*.so as they should be.

Thanks,

James

This problem only affects in-place builds (i.e. "python setup.py build_ext
-i"), which is something that developers will stumble over while doing
development. Distribution builds don't seem to be affected.

On Thu, Aug 27, 2009 at 12:31 PM, James Westby
<<email address hidden><jw%<email address hidden>>
> wrote:

> On Wed Aug 26 22:44:40 UTC 2009 Martin Pool wrote:
> > 2009/8/26 James Westby <<email address hidden><jw%<email address hidden>>
> >:
> > > Martin,
> > >
> > > Why did you add a "bzr (Ubuntu)" task to this bug, I don't see this
> > > happening with the bzr package in Ubuntu.
> >
> > Because I thought it might be a problem there. If not, I'll mark it
> > invalid and apologize for the noise.
>
> No problem, I just wondered if you knew something I didn't.
>
> I made sure to check the contents of the .deb before uploading, and
> they were definitely in bzrlib/*.so as they should be.
>
> Thanks,
>
> James
>
> --
> C extensions placed in wrong directory
> https://bugs.launchpad.net/bugs/392355
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Bazaar Version Control System: Triaged
> Status in “bzr” package in Ubuntu: Invalid
> Status in “python-defaults” package in Ubuntu: New
>
> Bug description:
> C extensions are placed in bzrlib/bzrlib and thus are not used.
>
> $make
> setup.py build_ext -i
> running build_ext
> building 'bzrlib._bencode_pyx' extension
> creating .../iter-changes-partial-parents/bzrlib/bzrlib
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-x86_64-2.6/bzrlib/_bencode_pyx.o -o
> .../iter-changes-partial-parents/bzrlib/bzrlib/_bencode_pyx.so
> building 'bzrlib._btree_serializer_pyx' extension
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
> build/temp.linux-x86_64-2.6/bzrlib/_btree_serializer_pyx.o -o
> .../iter-changes-partial-parents/bzrlib/bzrlib/_btree_serializer_pyx.so
> ...
>

--
Aldo Cortesi
www.nullcube.com
+64 210 718 900

So to summarise, this bug appears to be impacting all python developers on karmic building extensions via setup.py. Bazaar is affected but not the problem per se. Furthermore, it isn't affecting developers on jaunty, nor the packaging of Bazaar on karmic. Is the critical priority justified?

I can dig more into this but I'll need to upgrade one of my boxes to karmic first. That's not a problem but I wasn't planning to do that prior to 2.0 shipping unless I had to.

I have a Karmic machine, and I'll try to reproduce it there.

Arguably this is not critical, though because we don't warn on missing extensions people might get a very unfoundedly bad impression if they're not built. otoh most people should run from packages. This might be shallow so if it is, we'll fix it, otherwise downgrade it.

On Mon, 2009-08-31 at 05:02 +0000, Ian Clatworthy wrote:

> I can dig more into this but I'll need to upgrade one of my boxes to
> karmic first. That's not a problem but I wasn't planning to do that
> prior to 2.0 shipping unless I had to.

I don't think this should delay 2.0 - AFAICT its not a bzr bug, just
affects bzr.

-Rob

@ian see http://blog.sourcefrog.net/2009/09/setting-up-ubuntu-chroot-for.html for what I used to reproduce this.

I can still confirm it's happening in current Karmic. However, since it can be reproduced in other packages I'm going to downgrade it in bzr. I do think the Karmic python maintainers should take it more seriously...

Hi,

It looks to me like http://bugs.python.org/issue6365 needs to be fixed on
the 2.6 branch as well.

Thanks,

James

Note http://bugs.python.org/issue6403 as well.

Thanks,

James

looking ... the patch was explicitely blocked on the 2.6 branch

python2.6 is now updated in karmic. please could somebody recheck?

2009/9/11 Matthias Klose <email address hidden>:
> python2.6 is now updated in karmic. please could somebody recheck?

Looks good to me with 2.6.2-0ubuntu5. Thanks!

--
Martin <http://launchpad.net/~mbp/>

Not a bzr upstream bug.

> Looks good to me with 2.6.2-0ubuntu5. Thanks!

closing