Handling of names in UTF-8 (Unicode)

This line:
  ie.symlink_target = data.encode('utf8')
could be changed to:
  ie.symlink_target = data
to prevent a failed conversion (¿from utf8 to utf8?) and to store the symlink really as it is. However, Bazaar doesn't support symlinks to Unicode (in fact, non-ASCII) filenames yet. This bug depends on bug 272444.

Now that bug 272444 is fixed, the change in comment 1 can be applied. I attach it as a patch. With it I could correctly import a repository that didn't work before.

Could the patch please be checked in?

While the previous patch worked before, now a decode("utf8") is needed in order to pass CHKInventory._entry_to_bytes a unicode object, not a str.

I attach an updated patch ready to check in.

To test this bug you can use this line:

cd /tmp; rm -rf tres enbzr; mkdir tres; cd tres; git init; touch més; ln -s més prova; git add més prova; git commit -a -m "link to a file with a name in utf-8"; git fast-export --all >expo; mkdir enbzr; cd enbzr; bzr init; bzr fast-import ../expo

With a larger repository I found another case that didn't work. Corrected in this new patch.

It seems there are more cases where a decode() may be needed, in particular with rename_item. I attach a new patch, but a new branch would be better since there may be many other changes.
I don't know if adding decode("utf-8") is the correct approach. With this patch v4 I could get further in the conversion of a large branch.

Sorry for so many patches -- I should use a branch. But with this one (v5) I got no Unicode errors exporting the biggest branch I have (3873 rev. with many error-prone names). It stopped in another error (bug #458260, also about file names).

Thanks Daniel. I'm looking forward to getting this sorted out. BTW, I tried this:

mkdir tres; cd tres; bzr init; touch més; ln -s més prova; bzr add més prova; bzr commit -m "link to a file with a name in utf-8"

and bzr appears to do the right thing. Running "bzr fast-export ." on the resulting branching falls over though. I suspect it needs a few tweaks like you've done on the import side.

So altogether, we need to get several things working:

1. import with unicode filenames and symlinks
2. export with unicode filenames and symlinks
3. "bzr fast-import-filter in.fi > out.fi" needs to produce an out.fi equivalent to the in.fi.

Could you put together a branch, apply your patch and push it to Launchpad? We can then work through these issues, add some tests and merge your fixes.

I created a branch at lp:~n142857/bzr-fastimport/unicode-symlinks
It has the 3 points you asked for: import and export work (I tried simple test cases for both and a complex test case for import), and the filter is neutral. As a plus, the current fast-import tests still pass.

About the 3rd point, fast-import-filter, I should mention two differences which I don't think are relevant, but just in case…:
1. git input file uses 100644 as a mode, but bzr exports 644
2. bzr produces one more blob than git in a "mv" operation. I can send a diff.

I used this script to do more complex testing with symlinks:
 cd /n; rm -rf quatre enbzr; mkdir quatre; cd quatre; git init; touch més; ln -s més prova; git add més prova; git commit -a -m "link to a file with a name in utf-8"; cp -l prova prova2; git add prova2; git commit -a -m "copied symlink"; git mv prova provab; git commit -a -m "moved symlink"; rm provab; touch més2; ln -s més2 provab; git commit -a -m "modified symlink destination"; git rm provab; git commit -a -m "deleted symlink"; git fast-export --all >expo; mkdir enbzr; cd enbzr; bzr init; bzr fast-import ../expo

But of course we need better tests.

Thanks, merged (with some tweaks). Sorry it took so long!