there are actually several different bugs, they just happen to provoke a similar error.
@Matthias
I didn't make it clear before, but *all* entries should be UTF-8 encoded. If there is a Unicode object, then there is a bug in the 'lines' being generated. Which is why I asked which ones were Unicode.
I'm trying to reproduce it here, but I have failed so far. I *have* been able to get the:Inconsistent delta" bug to trigger, but that was unable to give me a the UnicodeDecodeError problem that you are talking about.
I would certainly prefer it if we could reproduce this without your 5000 files, I just haven't been able to reproduce it.
@codeslinger
there are actually several different bugs, they just happen to provoke a similar error.
@Matthias
I didn't make it clear before, but *all* entries should be UTF-8 encoded. If there is a Unicode object, then there is a bug in the 'lines' being generated. Which is why I asked which ones were Unicode.
I'm trying to reproduce it here, but I have failed so far. I *have* been able to get the:Inconsistent delta" bug to trigger, but that was unable to give me a the UnicodeDecodeError problem that you are talking about.
I would certainly prefer it if we could reproduce this without your 5000 files, I just haven't been able to reproduce it.