Paramiko startup delay makes bzr slow on Windows

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gareth White wrote:
> Public bug reported:
>
> On Windows, Paramiko has a 1.5 second delay on initialization (see bug
> 271791). What this means for Bazaar is that any commands that access a
> branch via bzr+ssh:// or sftp:// will have a 1.5 second delay.
>
> Also, having the bzr-svn plugin installed causes Paramiko to be
> initialized when using other protocols (see bug 487567). This means that
> any commands that use bzr://, http://, https:// will also have a 1.5
> second delay, even if you're not using svn at all. Note that bzr-svn is
> included by default in the Windows installer.
>
> What this means is that the majority of users on Windows are going to
> see slow performance (laggy commands) if they use nearly any kind of
> remote branch.
>
> I've tested this in Bazaar 2.0.2 and 2.1.0b3.
>

So it seems to depend on the platform. As running on Vista, I can see
that the resolution of time.time() is right at 1ms. So 1ms*100 = 100ms
startup time, and not 1.5s.

IIRC there is a global "multimedia timer resolution" switch that can be
set on Windows, which ends up affecting the resolution of time.time().

You can test it yourself by running this in a python shell:

def min_delta():
  t = time.time()
  t2 = t
  while t2 == t:
    t2 = time.time()
  return (t2 - t) * 1000.0

It gives the time resolution in milliseconds.

Anyway, one option is to apply the patch I made for PyCrypto to the
version that gets bundled with the installers.

Another option is to get paramiko to stop using RandomPool. Though it
requires a beta version of Pycrypto, (they took a *long* time to release
an updated pycrypto.)

Patching pycrypto seems like a reasonable route, and we may have been
doing it in the past, and I just forgot about it when we set up a new
build host. (I just checked, and this isn't true, though.)

Anyway, I'm fine applying a patch to the version of Crypto that we ship
with the installer if you think that will be reasonable.

 status confirmed
 importance high
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksqlKkACgkQJdeBCYSNAAMSdwCfR2gg0zmoqg5z3zwcUz27Qo26
qT0Anj8XG4tzQP8L/PvnzNrURaB6wCp2
=qP0J
-----END PGP SIGNATURE-----

Interesting - I'm guessing they improved the timer resolution in Vista vs. XP. Unfortunately I don't have access to a Windows box right now to test on, but last time I checked the resolution was about 10-15 ms.

Anyway, it looks like PyCrypto 2.1.0 final has just been released - see http://lists.dlitz.net/pipermail/pycrypto/2009q4/000169.html

If that fixes the delay then that may be a better long-term solution that creating a custom version of PyCrypto as it avoids the risk of people inadvertently using the wrong one and getting slow performance. On the other hand if Bazaar or Paramiko depend on PyCrypto a lot then it might be difficult to upgrade. I guess there's only one way to find out!

I've been using PyCrypto 2.1.0 with Paramiko 1.7.6 for my dev install on windows and I'm not noticing any delay but it may not be obvious. What is obvious though is the big warning printed about RandomPool being broken and deprecated.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gordon Tyler wrote:
> I've been using PyCrypto 2.1.0 with Paramiko 1.7.6 for my dev install on
> windows and I'm not noticing any delay but it may not be obvious. What
> is obvious though is the big warning printed about RandomPool being
> broken and deprecated.
>

This should be addressed by the -3 version of the installers. I'm using
the patched PyCrypto 2.0. We can certainly look at switching to 2.1, but
getting the warnings will be a bit ugly. Though I'm already getting:

c:\Python26\lib\site-packages\pycrypto-2.0.1-py2.6-win32.egg\Crypto\Hash\SHA.py:6:
DeprecationWarning: the sha module is deprecated; use the hashlib module
instead
c:\Python26\lib\site-packages\pycrypto-2.0.1-py2.6-win32.egg\Crypto\Hash\MD5.py:6:
DeprecationWarning: the md5 module is deprecated; use hashlib instead

So I suppose trading 2 warnings from pycrypto for 1 warning would be a
net win...

Note the warnings aren't shown in the packaged version, because we use
python2.5.

John
=:->

 status fixreleased
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksr4ZIACgkQJdeBCYSNAAPJFQCfa/ywO5iNDmzHvbAIFH//kTOv
YoYAn2HXCU1LY9SREgaV8+gxBN0vQlzA
=Hzyt
-----END PGP SIGNATURE-----

If we can determine how to patch Paramiko to use the correct method for getting random data instead of using RandomPool, how hard would it be to include it in installers? Or, what would be the chances of getting it into a new release of Paramiko?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gordon Tyler wrote:
> If we can determine how to patch Paramiko to use the correct method for
> getting random data instead of using RandomPool, how hard would it be to
> include it in installers? Or, what would be the chances of getting it
> into a new release of Paramiko?
>

I think it would be essentially patching one file "paramiko/rng.py"

The StrongLockingRandomPool() class would need to be patched to use the
new functionality. I don't know if we could take it the step further and
get rid of StrongLockingRandomPool entirely, since it seems that the new
random stream functionality is done differently.

I haven't looked closely, but at least it looks like Crypto now does the
entropy seeding for you, rather than expecting the calling code to pass
in more entropy from time to time.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkssB6MACgkQJdeBCYSNAAMsugCgidFbTav4lOoR1BgmYcST0RE/
J9oAoKuLKp0ypvIj2s5nK/VYcenj9sM2
=57s7
-----END PGP SIGNATURE-----