Unicode errors in mailman3

Can you provide a reproducible recipe, or test case? Is this caused by a member with non-ASCII in their name? Or a message with non-ASCII in the From field? Any more information about how to reproduce this will really help (e.g. to create a test case).

I'm trying to write a testcase for the second set of tracebacks, but I'm having a hard time understanding where the message that causes it comes from. When I open the pickle file I get:
'From nobody Sat Mar 23 09:42:34 2013\n\n'
This goes through the digest runner and tracebacks at line 269 with "TypeError: decoding Unicode is not supported"

Do you know where such a message can come from ? Are you OK with just writing a test case with this string as an entry message ?

I'm still trying to figure out the unicode errors in the digest runner, but something happens that I don't understand.
In the add_message method, around line 264, the msg.as_string() call returns an str instance, but the msg.as_string().split("\n\n") call returns a list of unicode instances. This makes no sense to me.
Could it come from the unicode-related overrides in mailman.email.message ? Could it be a bug in the python interpreter ? I'm using python 2.7.3. I can't find a way to reproduce it outside mailman.

OK, thanks to Toshio, I *finally* know what's going on here. Because of the import of "unicode_literals" from __future__ at the top of the file, the call to .split("\n\n") is actually interpreted as .split(u"\n\n"), thus the result is a list of unicode strings, which of course can't be decoded further in the following lines. To keep string literals, the string should be prefixed with "b", here's the patch:

=== modified file 'src/mailman/runners/digest.py'
--- src/mailman/runners/digest.py 2013-01-01 14:05:42 +0000
+++ src/mailman/runners/digest.py 2013-04-11 15:52:53 +0000
@@ -260,7 +260,7 @@
         # Add the payload. If the decoded payload is empty, this may be a
         # multipart message. In that case, just stringify it.
         payload = msg.get_payload(decode=True)
- payload = (payload if payload else msg.as_string().split('\n\n', 1)[1])
+ payload = (payload if payload else msg.as_string().split(b'\n\n', 1)[1])
         try:
             charset = msg.get_content_charset('us-ascii')
             payload = unicode(payload, charset, 'replace')

That's one of the errors I reported in this but, I'll investigate the other one now.

OK, I found out where the second error comes from. When there is no administrator available, the message is sent to the site_owner from the configuration file, but this entry is not converted to unicode before being set in the msgdata dict.
Thus, when it's added to the database, Storm refuses it. The following patch fixes this bug:

=== modified file 'src/mailman/handlers/owner_recipients.py'
--- src/mailman/handlers/owner_recipients.py 2013-01-01 14:05:42 +0000
+++ src/mailman/handlers/owner_recipients.py 2013-04-12 16:47:29 +0000
@@ -55,7 +55,7 @@
         # To prevent -owner messages from going into a black hole, if there
         # are no administrators available, the message goes to the site owner.
         if len(recipients) == 0:
- msgdata['recipients'] = set((config.mailman.site_owner,))
+ msgdata['recipients'] = set((unicode(config.mailman.site_owner),))
         else:
             msgdata['recipients'] = recipients
         # Don't decorate these messages with the header/footers. Eventually

I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same problem as described above.
Luckily I found this bug report and could manage to fix it with the patches above.

May I ask anybody (with more bazar-knowledge than me) to merge this two patches into the mailman 3 branch?
Thank you!

On Apr 30, 2014, at 09:38 PM, Tim Marx wrote:

>I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same
>problem as described above. Luckily I found this bug report and could manage
>to fix it with the patches above.
>
>May I ask anybody (with more bazar-knowledge than me) to merge this two
>patches into the mailman 3 branch? Thank you!

I think we just need some test cases. Tim, can you provide any details on how
you it the bug?

OK, I've written the unit tests and pushed them with the fixes in my dedicated branch: lp:~abompard/mailman/bug-1130957. The merge proposal has just been sent.