Localised Ubuntu start pages (8.10) have corrupted UTF-8 text

Please find attached the original translation as submitted to the ubuntu-translators list.

Confirming, the title (and contents) is broken for Russian page as well

The problem is in the "localize.sh" script. It seems that po2html does not manage correctly the accents in the po file. A work-around is the usage of HTML codes for special characters [1] in the .po file instead of the utf-8 encoded characters (e.g. "Páxina" instead of "Páxina").

[1] http://webdesign.about.com/library/bl_htmlcodes.htm

In our case the script generated a correct page, though. It is only the published page which does not seem to be encoded in UTF-8

Maybe HTML-Tidy produces such an output? I have tried to run it locally and looks like it does not detect utf-8 automatically and converts international characters into unreadable stuff...

This page an online page and is not part of the ubuntu-docs package.

Affecting the hungarian startpage too, but I guess you already know.

Affecting the Japanese startpage too.

Confirmed for the Swedish start page. It looks like something that is already UTF-8-encoded, is being transformed from something else to UTF-8-encoding once more, as every character > 127 takes up four bytes (checked with hex editor).

Working on a solution. Seems to be po2html causing the problem.

Affecting the Greek startpage too

Also the Spanish home page.

Possible solution/workaround/whatever: https://lists.ubuntu.com/archives/ubuntu-translators/2008-October/001886.html
Could somebody confirm if this is a viable way? No reply on the list yet :(. Is it just me or that problem is really _that_ difficult?

I've just send a possible fix to the mailing list ubuntu-translators. HTH

Here's the patch to apply to po2html.py

--- translate-toolkit-1.1.1/translate/convert/po2html.py.old 2008-11-05 17:18:17.000000000 +0100
+++ translate-toolkit-1.1.1/translate/convert/po2html.py 2008-11-05 17:18:50.000000000 +0100
@@ -81,7 +81,7 @@
                 htmlresult = htmlresult.replace(msgid, msgstr, 1)
         htmlresult = htmlresult.encode('utf-8')
         if tidy:
- htmlresult = str(tidy.parseString(htmlresult))
+ htmlresult = str(tidy.parseString(htmlresult, **{'char_encoding': "utf8"}))
         return htmlresult

 def converthtml(inputfile, outputfile, templatefile, wrap=None, includefuzzy=False):

I can confirm that this patch works correctly (at least for the languages I speak).

A fix has been implemented but a more long-term solution is needed. I will be working on this in the context of the ubuntu-website team, anyone interested in contributed to the solution is welcome and encouraged to join.

Strange, lots of languages (ja, ka, cs, bn) look fine now, but Hungarian and Russian not, see: http://start.ubuntu.com/8.10/index.html.ru, http://start.ubuntu.com/8.10/index.html.hu.

Confirmed for Swedish.
Still waiting for the Search string fix (which is "Sök" in Swedish)

Forget my previous comment, they are fine. Perhaps it was just my browser cache :(.