Python2.6 hmac.py TypeError: character mapping must return integer, None or unicode"

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...

> bzr: ERROR: exceptions.TypeError: character mapping must return integer,
> None or unicode
>
> Traceback (most recent call last):
> File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 896, in run_bzr_catch_errors
> return run_bzr(argv)
> File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 842, in run_bzr
> ret = run(*run_argv)
> File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 542, in run_argv_aliases
> return self.run(**all_cmd_args)
> File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/pqm/__init__.py", line 110, in run
> tree=tree)
> File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/pqm/pqm_submit.py", line 215, in submit
> SMTPConnection(config).send_email(message)
> File "/usr/lib/python2.6/dist-packages/bzrlib/smtp_connection.py", line 152, in send_email
> self._connect()
> File "/usr/lib/python2.6/dist-packages/bzrlib/smtp_connection.py", line 67, in _connect
> self._authenticate()
> File "/usr/lib/python2.6/dist-packages/bzrlib/smtp_connection.py", line 114, in _authenticate
> self._connection.login(self._smtp_username, self._smtp_password)
> File "/usr/lib/python2.6/smtplib.py", line 574, in login
> (code, resp) = self.docmd(encode_cram_md5(resp, user, password))
> File "/usr/lib/python2.6/smtplib.py", line 538, in encode_cram_md5
> response = user + " " + hmac.HMAC(password, challenge).hexdigest()
> File "/usr/lib/python2.6/hmac.py", line 72, in __init__
> self.outer.update(key.translate(trans_5C))
> TypeError: character mapping must return integer, None or unicode

^- From what I can tell, this is a bug in python's smtplib and how it
handles start_tls.

Certainly at best we are passing whatever plain-text password down into
login(), which is then responsible for the encoding, etc. Which it seems
to be failing to do.

  affects python

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmv628ACgkQJdeBCYSNAANpGQCeLjIeosDZ/w89FM8n8H3rYae1
vhEAoJGsQFLAE1NOyvfG/z2OTjaPaLEP
=NEMv
-----END PGP SIGNATURE-----

Could you try running "bzr selftest" and see if any other errors are produced? The end of the traceback is all in the Python standard library and bzrlib, so I am wondering if this is a Bazaar bug rather than a bzr-pqm bug.

this is what happens for "bzr selftest":

beuno@dell-desktop:~$ bzr selftest
testing: /usr/bin/bzr
   /usr/lib/python2.6/dist-packages/bzrlib (1.13dev python2.6.1)

/usr/lib/python2.6/dist-packages/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  from sha import *
/usr/lib/python2.6/dist-packages/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  from md5 import *
bzr: ERROR: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.Spawn.ChildExited: Launch helper exited with unknown return code 0

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 921, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 866, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/dist-packages/bzrlib/commands.py", line 547, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/dist-packages/bzrlib/builtins.py", line 3205, in run
    result = selftest(**selftest_kwargs)
  File "/usr/lib/python2.6/dist-packages/bzrlib/tests/__init__.py", line 2703, in selftest
    suite = test_suite(keep_only, starting_with)
  File "/usr/lib/python2.6/dist-packages/bzrlib/tests/__init__.py", line 3122, in test_suite
    plugin_suite = plugin.test_suite()
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugin.py", line 453, in test_suite
    return self.module.test_suite()
  File "/home/beuno/.bazaar/plugins/gtk/__init__.py", line 657, in test_suite
    result.addTest(tests.test_suite())
  File "/home/beuno/.bazaar/plugins/gtk/tests/__init__.py", line 36, in test_suite
    result.addTest(loader.loadTestsFromModuleNames(["%s.%s" % (__name__, i) for i in testmod_names]))
  File "/usr/lib/python2.6/dist-packages/bzrlib/tests/TestUtil.py", line 92, in loadTestsFromModuleNames
    result.addTests(self.loadTestsFromModuleName(name))
  File "/usr/lib/python2.6/dist-packages/bzrlib/tests/TestUtil.py", line 97, in loadTestsFromModuleName
    module = _load_module_by_name(name)
  File "/usr/lib/python2.6/dist-packages/bzrlib/tests/TestUtil.py", line 166, in _load_module_by_name
    module = __import__(mod_name)
  File "/home/beuno/.bazaar/plugins/gtk/tests/test_preferences.py", line 18, in <module>
    from bzrlib.plugins.gtk.olive import Preferences
  File "/home/beuno/.bazaar/plugins/gtk/olive/__init__.py", line 51, in <module>
    from bzrlib.plugins.gtk.annotate.gannotate import GAnnotateWindow
  File "/home/beuno/.bazaar/plugins/gtk/annotate/gannotate.py", line 31, in <module>
    from bzrlib.plugins.gtk.revisionview import RevisionView
  File "/home/beuno/.bazaar/plugins/gtk/revisionview.py", line 33, in <module>
    from bzrlib.plugins.gtk import seahorse
  File "/home/beuno/.bazaar/plugins/gtk/seahorse.py", line 33, in <module>
    crypto = dbus.Interface(bus.get_object(BUS_NAME, CRYPTO_PATH),
  File "/var/lib/python-support/python2.6/dbus/bus.py", line 244, in get_object
    follow_name_owner_changes=follow_name_owner_changes)
  File "/var/lib/python-support/python2.6/dbus/proxies.py", line 241, in __init__
    self._named_service =...

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Albisetti wrote:
> this is what happens for "bzr selftest":
>
> beuno@dell-desktop:~$ bzr selftest
> testing: /usr/bin/bzr
> /usr/lib/python2.6/dist-packages/bzrlib (1.13dev python2.6.1)
>
> /usr/lib/python2.6/dist-packages/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
> from sha import *
> /usr/lib/python2.6/dist-packages/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
> from md5 import *
> bzr: ERROR: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.Spawn.ChildExited: Launch helper exited with unknown return code 0

This is just exposing a bug in the dbus plugin, which isn't revealing
anything about bzr-pqm (unfortunately).

I think this is the "people sshing into a machine get dbus failures" bug.

Can you try running selftest without bzr-dbus installed?

Or maybe just

bzr selftest -s bp.pqm

Which would avoid loading the dbus tests, and only load the pqm ones?

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmv8YUACgkQJdeBCYSNAAN/hwCg0HFvdFvesw2L5fmtuVlxFuU1
9iIAnRnKWFBUFkDj1x1w/60i2IgIbHGJ
=vyKs
-----END PGP SIGNATURE-----

This is my 'bzr seftest' (it seems to have hung at the end, as it doesn't return me to a bash prompt)

jblount@r2d2:~/code/u1/share/colors-revisit$ bzr selftest
testing: /usr/bin/bzr
   /usr/lib/python2.6/dist-packages/bzrlib (1.12 python2.6.1)

/usr/lib/python2.6/dist-packages/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
  from sha import *
/usr/lib/python2.6/dist-packages/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  from md5 import *
XFAIL: blackbox.test_info.TestInfo.test_info_locking
OS locks are exclusive for different processes (Bug #174055)

XFAIL: blackbox.test_log.TestLogMerges.test_merges_nonsupporting_formatter
log formatters must support linear sequences now

XFAIL: branch_implementations.test_sprout.TestSprout.test_sprout_with_unicode_symlink(BzrBranchFormat5)
there is no support for symlinks to non-ASCII targets (bug #272444)

XFAIL: branch_implementations.test_sprout.TestSprout.test_sprout_with_unicode_symlink(BzrBranchFormat7)
there is no support for symlinks to non-ASCII targets (bug #272444)

XFAIL: branch_implementations.test_sprout.TestSprout.test_sprout_with_unicode_symlink(BzrBranchFormat6)
there is no support for symlinks to non-ASCII targets (bug #272444)

XFAIL: branch_implementations.test_sprout.TestSprout.test_sprout_with_unicode_symlink(RemoteBranchFormat-default)
there is no support for symlinks to non-ASCII targets (bug #272444)

XFAIL: branch_implementations.test_sprout.TestSprout.test_sprout_with_unicode_symlink(RemoteBranchFormat-v2)
there is no support for symlinks to non-ASCII targets (bug #272444)

XFAIL: test_fifo_cache.TestFIFOCache.test_cleanup_at_deconstruct
we don't call cleanups during __del__

XFAIL: test_http.TestInvalidStatusServer.test_http_get(pycurl,HTTP/1.1)
pycurl hangs if the server send back garbage

XFAIL: test_http.TestInvalidStatusServer.test_http_has(pycurl,HTTP/1.1)
pycurl hangs if the server send back garbage

XFAIL: test_http.TestAuth.test_changing_nonce(pycurl,HTTP/1.0,digest)
pycurl does not handle a nonce change

XFAIL: test_http.TestAuth.test_changing_nonce(pycurl,HTTP/1.1,digest)
pycurl does not handle a nonce change

XFAIL: test_http.TestProxyAuth.test_changing_nonce(pycurl,HTTP/1.0,digest)
pycurl does not handle a nonce change

XFAIL: test_http.TestProxyAuth.test_changing_nonce(pycurl,HTTP/1.1,digest)
pycurl does not handle a nonce change

[9522/16420 in 12m22s, 6 missing] test_http.TestProxyAuth.test_wrong_pass(pycurl,HTTP/1.1,digest)

I had a similar failure on Jaunty's python2.6; I think it was due to a
transient problem as the packages were upgraded.

Please update Jaunty, reboot, and see if this hang still occurs.

--
Martin <http://launchpad.net/~mbp/>

This is still happening to me, in order to use pqm in the manner previously mentioned I've been using a Intrepid vm.

Is there any further diagnosis I can help with?

i can't reproduce this yet

it would help if you do

  export BZR_PDB=1

then run the command; when it fails you should get into the debugger and could you please print out the relevant locals there?

(This would be a good place for having a full cgitb style traceback)

I helped Josh walk through this in the debugger and think I've discovered the problem. It looks like a bzrlib bug that didn't trigger a failure in prior versions. We should be able to fix this so it'll work for all Python versions.

The SMTPConnection._authenticate() code in bzrlib is passing unicode strings for the username and password down to smtplib. A change in the implementation of the hmac module in 2.6 stopped this from working (at least if the server wanted to do an HMAC based form of authentication).

The RFCs don't seem to mention character encodings for the username or password, so it isn't clear what smtplib should do with such strings.

I'd suggest changing SMTPConnection to encode the username/password to ASCII and see if anyone complains.

Changing task from bzr-pqm to bzr, since the bug appears to be within bzrlib rather than in how bzr-pqm uses bzrlib.

Closing the Python bug task since we are passing unicode strings to an API that expects byte strings (and the underlying protocol is defined in terms of byte strings).

Attached is a patch to fix the problem. I've also sent it to the list, so it should show up in bundle buggy soon.

+1 the patch worked for me

The fix for this bug has been merged to bzr.dev in r4203.

Is this fix likely to make it into the bzr packages in the official jaunty repos?

On Tue, 2009-04-21 at 07:16 +0000, Tom Haddon wrote:
> Is this fix likely to make it into the bzr packages in the official
> jaunty repos?

I have requested that.

Thanks,

James