problems with bzr.exe vs Python-based installers

See also the comment here https://bugs.edge.launchpad.net/bzr/+bug/345998/comments/4

A related question: for Windows, the download page pushes people to pick the default, standalone installer:
- http://bazaar-vcs.org/Download offers "default installer" vs "other options" (which are the python-based ones)
- http://bazaar-vcs.org/WindowsDownloads says "This is the best choice for Bazaar users on Windows"; whereas Python-based ones are marked "Alternative".
But, when I tested the standalone installer a year ago, bzr-gtk didn't work with it, and it's still mentioned in
http://bazaar-vcs.org/WindowsDownloads "You can use most of the Bazaar plugins ( /!\ excluding some GUI plugins)".
So I don't understand: the pages are pushing for using an installer which is incompatible with such a widely used GUI plugin as bzr-gtk (we all use it at MySQL)? What is the idea around this? That people normally shouldn't be using bzr-gtk under Windows?
Is there a simple way to make bzr-gtk work with the default installer? is that already fixed? in the plans?
Thanks!

Hello, I reported the problem as part of https://bugs.launchpad.net/bzr/+bug/345998 in March. Could you please give an update on the present bug?

As Alexander writes in <https://bugs.edge.launchpad.net/loggerhead/+bug/390542/comments/8>

IIUC this error is a clear sign that some parts either of loggerhead or third-party library requires setuptools to work correctly. AFAIK, py2exed application (bzr.exe) can't work with setuptools and eggs properly. So this is dead end.

To me the options here seem to include:

 * don't use py2exe but rather build an installer by some other means that installs it as a regular python library
 * fix or workaround problems in other libraries integrating with bzrlib in a py2exe installation
 * something else?

> To me the options here seem to include:
>
> * don't use py2exe but rather build an installer by some other means that installs it as a regular python library
> * fix or workaround problems in other libraries integrating with bzrlib in a py2exe installation
> * something else?

Well, I was suggesting that loggerhead and its dependencies be installed
via a separate installer - because loggerhead is a plugin now. perhaps
thats still incompatible with setuptools, but I haven't tested.

The layout on disk would be the py2exe main install, and a second zip
contining paste, pastedeploy and so forth, as well as the loggerhead
plugin.

-Rob

Martin, why is there separate packages for each Linux distro? Why is there separate packages for each Ubuntu version?

Because they're different?

That's why there is different installers for different python versions @ win32, because there is compiled C/Pyrex extensions, and these extensions are built against specific python version. You can't workaround this fact.

About incompatibility of bzr-gtk vs bzr.exe: even before tortoisebzr and qbzr started to be packaged into official installer, qbzr provided special installer with qt libs inside. And it works very well. Anyone who want to try can do te same for bzr-gtk, IMO. The problem is: nobody want to try.

>>>>> "bialix" == Alexander Belchenko <email address hidden> writes:

<snip/>

    bialix> About incompatibility of bzr-gtk vs bzr.exe: even
    bialix> before tortoisebzr and qbzr started to be packaged
    bialix> into official installer, qbzr provided special
    bialix> installer with qt libs inside. And it works very
    bialix> well.

Just for the record, can you still build it ? How big is it ?

    bialix> Anyone who want to try can do te same for bzr-gtk,
    bialix> IMO. The problem is: nobody want to try.

It didn't get finished regretfully, but, I helped Mario Danic
start exactly that at UDS. From memory he was able to run the gtk
demo from a self-made installer as well as the gtk plugin, I
think he even published a branch...

      Vincent

Vincent Ladeuil пишет:
>>>>>> "bialix" == Alexander Belchenko <email address hidden> writes:
>
> <snip/>
>
> bialix> About incompatibility of bzr-gtk vs bzr.exe: even
> bialix> before tortoisebzr and qbzr started to be packaged
> bialix> into official installer, qbzr provided special
> bialix> installer with qt libs inside. And it works very
> bialix> well.
>
> Just for the record, can you still build it ? How big is it ?

Yes, I'm still build it, it available on qbzr's download page as
standalone installer and has size ~4MB (with PyQt 4.3.1 libs, I suspect
PyQt 4.4.3 will be a bit bigger).

PyGTK require many additional libs, but I expect resulting installer for
  bzr-gtk should be about 7MB. It's bearable for most of the people today.

> bialix> Anyone who want to try can do te same for bzr-gtk,
> bialix> IMO. The problem is: nobody want to try.
>
> It didn't get finished regretfully, but, I helped Mario Danic
> start exactly that at UDS. From memory he was able to run the gtk
> demo from a self-made installer as well as the gtk plugin, I
> think he even published a branch...
>
> Vincent
>

2009/7/1 Alexander Belchenko <email address hidden>:
> Martin, why is there separate packages for each Linux distro? Why is
> there separate packages for each Ubuntu version?
>
> Because they're different?
>
> That's why there is different installers for different python versions @
> win32, because there is compiled C/Pyrex extensions, and these
> extensions are built against specific python version. You can't
> workaround this fact.

Perhaps the bug title is unclear. Yes, of course it makes sense that
there are different builds for different python versions. The problem
as I see it is the distinction between the python-based installers and
bzr.exe. Is that really needed?

On Ubuntu, there is just one bzr package, which (in Jaunty, as of
1.16+4485+116) includes extensions built against python2.5 and 2.6,
and it can be imported as a library by plugins. It would be nice if
we could get closer to that on Windows.

--
Martin <http://launchpad.net/~mbp/>

Martin Pool пишет:
> 2009/7/1 Alexander Belchenko <email address hidden>:
>> Martin, why is there separate packages for each Linux distro? Why is
>> there separate packages for each Ubuntu version?
>>
>> Because they're different?
>>
>> That's why there is different installers for different python versions @
>> win32, because there is compiled C/Pyrex extensions, and these
>> extensions are built against specific python version. You can't
>> workaround this fact.
>
> Perhaps the bug title is unclear. Yes, of course it makes sense that
> there are different builds for different python versions. The problem
> as I see it is the distinction between the python-based installers and
> bzr.exe. Is that really needed?
>
> On Ubuntu, there is just one bzr package, which (in Jaunty, as of
> 1.16+4485+116) includes extensions built against python2.5 and 2.6,
> and it can be imported as a library by plugins. It would be nice if
> we could get closer to that on Windows.

It depends.

Linux/Unix have more or less typical layout of libs and binary files.
Many Linux distros have package managers.

Windows has none of the above. So you can't go closer to Linux there.
You have to choose different way.

If you want to have simple install for python-based bzr version then it
makes sense to look at setuptools and easy_install (though me does not
like it very much). Then user only need to install Python and maybe
setuptools and then `easy_install bzr` will do the magic.

Standalone bzr.exe is standalone program. You can put every plugin
inside it and then all required dependencies will be bundled as well. As
result you'll have ~30-60MB installer that should work for everyone.
(AFAIK TortoiseHg distribution with GTK libs inside is about 30MB).

The only one problem today is all installed plugins are enabled
implicitly. In hg user have to enable plugins manually. So hg developers
can ship standalone hg.exe with all official plugins inside.

Bazaar can ship all plugins as well, but it will require some changes to
installer to allow the user to select which plugins to install. Not the
worst way IMO.

Is bzr.exe really needed? Some facts:

1) today nobody provides windows installer for bzr-svn plugin separately
of official bzr.exe build. And building bzr-svn on Windows is hard hard
work. Do you want to drop this beautiful plugin @ windows?

2) the same applied to TortoiseBzr which should be compiled today.

3) In my observation standalone bzr.exe works a bit faster on Windows;
mostly because all pure python modules packaged into one big
library.zip, so Windows need to read only one file from disk instead of
zillion files (including py/pyc/pyo difference). This is very important
to get better performance: less files to read means faster windows
application.

4) You're starting this discussion again and again, but if you look at
download numbers (they are available now on LP) you'll see what users
prefer. Of course one can say this is because it's default installer.
Yes, of course. But I think it suits needs most of users. All other who
need something non-default have to deal with python-based version.

Of course things will be much simpler if there...

I'm asking about it because it does seem to cause some trouble to users, if you look at for instance Guilhem's comment above. I wanted this bug so at least there was a handle for the issue rather than just scattered list threads. Whether this bug is actually fixed as specified, or marked Confirmed/Low, or even if we decide it's not actually a good idea to do it, it's good to have a single url explaining the issue.

I'm not proposing to just drop either the python-based installer or bzr.exe is not a good solution. It seems clear that bzr.exe with the standard plugins provided is what most people use and want. However, people do have some problems with it, and perhaps you will add more:

 * whenever there's a choice, understanding which one to use takes some time - perhaps not very much, and perhaps it's clear enough now
 * you can't install separate plugins or other bzrlib users to be used with bzr.exe

If we use easy_setup, is that able to bring down win32 binary libraries for the appropriate Python interpreter? If so, that may be pretty good.

We could certainly have an option either in the installer or in bzr itself to enable and disable particular plugins, and beyond that I think plugins should aim to have the minimum impact if they are installed but never used. That would be useful, but it's not really a total solution, because part of the point of plugins is that you can install new ones.

It sounds to me like we can't in fact unify them, but what we should do is, in this order of priority:

 * Include bzr-gtk and loggerhead (and whatever else) into the bzr.exe installer. This will satisfy most users most of the time: they don't actually want a python-based installer, they just want particular plugins to be available.
 * Fix bzr to work with easy_setup, including installing the right binary extensions -- that gives people a way to install loggerhead or whatever, including its dependencies, if they already have a system python. It will be useful on Unix too. If we do this, will these eggs be a sufficient substitute for the python-based installers, or do they need to be shipped separately?
 * Work out how, if at all possible, to make separately-installed plugins work with bzr.exe. (Like comment #4)

Just read all comments: it looks like there still is no way to have bzr-gtk install and work if bzr was installed with the default bzr installer?

GuilhemBichot пишет:
> Just read all comments: it looks like there still is no way to have bzr-
> gtk install and work if bzr was installed with the default bzr
> installer?

Somebody started work on such feature maybe half year ago(?) and it
seems not finished it.

For what it's worth, we have had two successful setups with the standalone bzr installer and bzr-gtk and a tweak. My colleague Vladislav Vaintroub reused Kevin Light's idea of patching bzr-gtk and adding libraries to it (look in http://bazaar-vcs.org/bzr-gtk?action=show&redirect=BzrGtk : "Windows packages by Kevin Light can be found at http://d5190871.u44.websitesource.net/bzr-gtk/" ), but he did it with the bzr-gtk trunk (whereas Kevin's package has bzr-gtk 0.96):
1) install the standalone bzr installer
2) install bzr-gtk's trunk into plugins/gtk
3) at this point bzr-gtk doesn't work yet
4) patch plugins/gtk/__init__.py like done in Kevin Light's package:
=== modified file '__init__.py'
--- __init__.py 2009-09-17 14:35:47 +0000
+++ __init__.py 2009-11-17 10:57:28 +0000
@@ -34,6 +34,14 @@
 visualise Graphically visualise this branch.
 """

+import sys
+
+import os.path
+# NOTE: _lib must be ahead of bzrlib or sax.saxutils (in olive) fails
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '_lib'))
+sys.path.append(os.path.join(os.path.dirname(__file__), '_lib/gtk-2.0'))
+sys.path.append(os.path.dirname(__file__))
+
 import bzrlib
 import bzrlib.api
 from bzrlib import (

5) put, in plugins/gtk/_lib , the same files/directories (cairo, gtk-2.0, pygtk, xml etc) as in Kevin Light's package; here are what we put in _lib:
Archive: _lib.zip
  Length Date Time Name
 -------- ---- ---- ----
        0 11-17-09 11:56 _lib/
        0 11-17-09 11:56 _lib/cairo/
    61440 04-04-08 19:59 _lib/cairo/_cairo.pyd
       21 04-04-08 19:59 _lib/cairo/__init__.py
      205 07-25-09 00:18 _lib/cairo/__init__.pyc
      205 10-26-09 14:53 _lib/cairo/__init__.pyo
    23756 10-29-05 01:07 _lib/ConfigParser.py
    24588 10-27-09 14:39 _lib/ConfigParser.pyo
        0 11-17-09 11:56 _lib/gtk-2.0/
   217600 04-04-08 19:58 _lib/gtk-2.0/atk.pyd
        0 11-17-09 11:46 _lib/gtk-2.0/codegen/
    44680 04-04-08 19:58 _lib/gtk-2.0/codegen/argtypes.py
     1304 03-09-04 20:12 _lib/gtk-2.0/codegen/code-coverage.py
    69400 04-04-08 19:58 _lib/gtk-2.0/codegen/codegen.py
      389 04-04-08 19:58 _lib/gtk-2.0/codegen/createdefs.py
    21433 04-04-08 19:58 _lib/gtk-2.0/codegen/definitions.py
     5161 04-03-06 20:04 _lib/gtk-2.0/codegen/defsconvert.py
    26035 08-01-06 00:58 _lib/gtk-2.0/codegen/defsgen.py
     5793 04-04-08 19:58 _lib/gtk-2.0/codegen/defsparser.py
     6467 04-04-08 19:58 _lib/gtk-2.0/codegen/docextract.py
     2480 04-03-06 20:19 _lib/gtk-2.0/codegen/docextract_to_xml.py
    31948 04-04-08 19:58 _lib/gtk-2.0/codegen/docgen.py
    17948 04-04-08 19:58 _lib/gtk-2.0/codegen/h2def.py
      665 04-04-08 19:58 _lib/gtk-2.0/codegen/mergedefs.py
      358 12-07-01 07:50 _lib/gtk-2.0/codegen/missingdefs.py
     2803 04-04-08 19:58 _lib/gtk-2.0/codegen/mkskel.py
     9210 04-04-08 19:58 _lib/gtk-2.0/codegen/override.py
    34929 04-04-08 19:58 _lib/gtk-2.0/codegen/reversewrapper.py
     1904 08-04-05 18:44 _lib/gtk-2.0/codegen/scanvirtuals.py
     4899 04-04-08 19:58 _lib/gtk-2.0/codegen/scmexpr.py
      233 04-04-08 19:58 _lib/gtk-2.0/codegen...

GuilhemBichot пишет:
> For what it's worth, we have had two successful setups with the standalone bzr installer and bzr-gtk and a tweak. My colleague Vladislav Vaintroub reused Kevin Light's idea of patching bzr-gtk and adding libraries to it (look in http://bazaar-vcs.org/bzr-gtk?action=show&redirect=BzrGtk : "Windows packages by Kevin Light can be found at http://d5190871.u44.websitesource.net/bzr-gtk/" ), but he did it with the bzr-gtk trunk (whereas Kevin's package has bzr-gtk 0.96):
> 1) install the standalone bzr installer
> 2) install bzr-gtk's trunk into plugins/gtk
> 3) at this point bzr-gtk doesn't work yet
> 4) patch plugins/gtk/__init__.py like done in Kevin Light's package:
> === modified file '__init__.py'
> --- __init__.py 2009-09-17 14:35:47 +0000
> +++ __init__.py 2009-11-17 10:57:28 +0000
> @@ -34,6 +34,14 @@
> visualise Graphically visualise this branch.
> """
>
> +import sys
> +
> +import os.path
> +# NOTE: _lib must be ahead of bzrlib or sax.saxutils (in olive) fails
> +sys.path.insert(0, os.path.join(os.path.dirname(__file__), '_lib'))
> +sys.path.append(os.path.join(os.path.dirname(__file__), '_lib/gtk-2.0'))
> +sys.path.append(os.path.dirname(__file__))
> +
> import bzrlib
> import bzrlib.api
> from bzrlib import (

This is right way to go, but I'd suggest to make it more universal with
following construct in the patch above:

import sys

if getattr(sys, "frozen", None) is not None: # we run bzr.exe
     import os
     # NOTE: _lib must be ahead of bzrlib or sax.saxutils (in olive) fails
     sys.path.insert(0, os.path.join(os.path.dirname(__file__), '_lib'))
     sys.path.append(os.path.join(os.path.dirname(__file__),
'_lib/gtk-2.0'))
     sys.path.append(os.path.dirname(__file__))

Now it's ready to propose it as patch for bzr-gtk trunk ;-)

So it looks like after Alexander's changes, the patch could be proposed to bzr-gtk maintainers. Could someone more knowledgable than me do this, if it's a good idea? It's not even my patch, it comes from Kevin Light originally.

I've merged this patch to bzr-gtk trunk.

Does anybody knows how to get in touch with Kevin Light so he can upload his packages to lp ?

@vila he has posted previously to our list form <email address hidden>.

Does merging this to bzr-gtk imply that this bug will be fixed (at least as regards bzr-gtk), or is there more to do?

There is more to do so that bzr.exe (which does a good job regarding the handling for the plugins bundled by the installer)
can better handle additional plugins and more importantly their required libraries.

This should be easy technically speaking once an agreement is reached on how that should be done in bzr core and the windows installer.

I found Kevin via his launchpad account and we're discussing how his package can be updated in the future to make bzr-gtk
easier to install on windows.

Perhaps we should split the bzr-gtk issue out to a separate bug, because the original description of this is about the more generic issue that having two installers is suboptimal.

Did the bzr-gtk issue get merged, or is this still under discussion?