UnicodeEncodeError on commit with non-BMP Unicode characters

bzr itself doesn't accept these filenames either, so it's not related to bzr-svn.

E.g. try adding a file name u'\U0005d062\U000df631' to bzr.

Okay, I see you are right. (Bzr actually will let you "bzr add" them, but dies if you try to "bzr commit" them. I just didn't go far enough.)

It looks like for some reason, it's trying to decode UTF-8 with the ascii codec, i.e.:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

Or rather, it is trying to encode an Unicode string to ASCII, instead of UTF-8.

I simplier reproduction recipe only involving bzr and Unicode character U+20000 CJK UNIFIED IDEOGRAPH-20000.

# note that the locale is UTF-8
$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US:en_GB:en
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
$ touch 𠀀
# The file is correctly in UTF-8:
$ ls | xxd
0000000: f0a0 8080 0a
$ bzr init
bzr add
Standalone tree (format: pack-0.92)
Location:
  branch root: .
# Adding works fine
$ bzr add
added "𠀀"
# But committing fails
$ bzr commit -m "Adding file"
Committing to: /tmp/bzr-bug-317644/
added 𠀀
aborting commit write group: UnicodeEncodeError('ascii', u'\U00020000', 0, 1, 'ordinal not in range(128)')
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\U00020000' in position 0: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 893, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 839, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 539, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2507, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 224, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 208, in commit
    *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/commit.py", line 376, in commit
    self.builder.finish_inventory()
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 192, in finish_inventory
    self.parents
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 670, in add_inventory
    inv_lines = self._serialise_inventory_to_lines(inv)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 1714, in _serialise_inventory_to_lines
    return self._serializer.write_inventory_to_lines(inv)
  File "/usr/lib/python2.5/site-packages/bzrlib/xml8.py", line 200, in write_inventory_to_lines
    return self.write_inventory(inv, None)
  File "/usr/lib/python2.5/site-packages/bzrlib/xml8.py", line 246, in write_inventory
    _encode_and_escape(ie.name), parent_str, parent_id,
  File "/usr/lib/python2.5/site-packages/bzrlib/xml8.py", line 104, in _encode_and_escape
    unicode_or_utf8_str)) + '"'
UnicodeEncodeError: 'ascii' codec can't encode character u'\U00020000' in position 0: ordinal not in range(128)

bzr 1.12dev on python 2.5.2 (linux2)
argum...

  status triaged
  importance medium
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

The problem seems to be here in bzrlib/xml8.py:
_unicode_re = re.compile(u'[&<>\'\"\u0080-\uffff]')

That doesn't cover all of Unicode, it just covers the BMP. This should be the following instead:
_unicode_re = re.compile(u'[&<>\'\"\u0080-\U0010ffff]')

Making that change in my tree fixes the problem for me.

Here is the patch I mentioned as a bzr bundle.

As mentioned in the merge request, it turns out to not be that simple.

It breaks down into

1) Unicode strings are UCS-4 on Linux and UCS-2 on Windows.
2) On Linux, _unicode_re = re.compile(u'[&<>\'\"\u0080-\U0010ffff]') seems to match u'\U0010abcd' just fine.
   On Windows, it only matches the first character of the 2-character form, and fails to encode.
3) As such, we *could* change the regex based on platform, but that causes other problems based on having different platforms generating different inventory texts.
4) I believe the next repo format (--development6-rich-root) will not suffer from this, because it doesn't use XML, and will just represent the paths as UTF-8.

Can someone test with --development6-rich-root, and see if commit, etc work fine? If so, I'd rather not mess around with the XML escaping, because Unicode is just a big pile of incompatibilities (from different implementers), so I'd rather stick with what has worked for us for the last couple of years.

This works fine with CHK repositories (bzr 2.0 and later), so I'm going to mark this fix released.