Changes in fontforge cause Wine's marlett.ttf to have incorrect available character sets

Created attachment 9476
screenshot demonstrating the problem

All titlebar buttons are beeing drawn as an X, and they are all misaligned too.

Looks like you don't have properly created marlett.ttf font. If you compiled Wine yourself, check that you satisfied all the requirements. Check with './configure --verbose'. If this is package - contact your packager.

Created attachment 9478
image representation of marlett.ttf

this was generated using fontimage /usr/share/wine/fonts/marlett.ttf
it looks ok, so it doesnt seem like a compilation/packaging problem

i have discovered the same problem.
It seems to be a fontforge problem.
With version 20070831 the font file is created "correcly" (file size 6268 Bytes)
All newer versions generates a "wrong" font fiel (file size 6156 Bytes)

I have tested following versions of sourceforge:

        20070831 <--- generates for wine a correct font file
        20071210 <--- generates for wine a incorrect font file
 20070915 <--- generates for wine a incorrect font file
 20071002 <--- generates for wine a incorrect font file
 20071110 <--- generates for wine a incorrect font file
 20080109 (latest version) <--- generates for wine a incorrect font file

The question is, is it realy a problem with/in fontforge or with wine.

Invalid. If changing the version of fontforge is what fixes the bug, then the bug is in fontforge. Please file a report with them.

Closing.

This is really a fontforge bug. I'll mark this bug as a duplicate of
the bug 10952 with more information about its nature.

*** This bug has been marked as a duplicate of bug 10952 ***

Closing.

Thanks for the clear data, though. That looks like a
handy guide for others to avoid this problem.

Unfortunately it looks like still nobody has filed a bug for fontforge about
this problem, at least google doesn't find anything not in the debian tracker.

bug filled on the fontforge-devel-ml(also used for bug reporting) with subject:
"[BUG] incorrect generation of marlett.ttf font-file (used by wine) with newer fontforge versions"

(In reply to comment #10)
> bug filled on the fontforge-devel-ml(also used for bug reporting) with subject:
> "[BUG] incorrect generation of marlett.ttf font-file (used by wine) with newer
> fontforge versions"

Thanks!

here is the result of the bugreport on fontforge:

There is no bug in fontforge which reproduces the problem. But rather the build-system part for generating the truetype font-files relies on a behaviour of fontforge which has gone in newer version than 20070831.

So this is a bug in the wine build system.

In version 20070831 or older (i haven't tested older version then 20070831) it seems(i guess so) that fontforge can detect if the generated ttf font file should be a symbol font-file as the marlett font is. But in newer Version this detection (if there where a detection mechanism) is gone. The type of the generated font-file is determined by the file extension.
In case for the marlett symbol ttf font the right extension is .sym.ttf.

I have generated a patch(wine-fontforge-symbol-font.patch) which modifies the Makefile.in so that for the marlett.ttf file following commands are invoked:

fontforge -script ../fonts/genttf.ff marlett.sfd marlett.sym.ttf
mv marlett.sym.ttf marlett.ttf

Created attachment 10629
patch for generating correct marlett.ttf font file with newer fontforge version
than 20070831

Sounds like it needs reopening, then.

marlett.sfd already has the necessary information - Encoding: Symbol, there is
nothing do detect.

(In reply to comment #15)
> marlett.sfd already has the necessary information - Encoding: Symbol, there is
> nothing do detect.
>

From what I understand, they dont honor this token "Encoding" anymore.

(In reply to comment #16)
> (In reply to comment #15)
> > marlett.sfd already has the necessary information - Encoding: Symbol, there is
> > nothing do detect.
> >
> From what I understand, they dont honor this token "Encoding" anymore.

Right, and that's where the source of this bug is.

ok i will report this on the fontforge-devel ml

http://sourceforge.net/mailarchive/forum.php?thread_name=200801311442.39548.s.wezel%40web.de&forum_name=fontforge-devel

George Williams writes:

> The other change is that in the old days if a font was displayed in
> "Symbol" encoding, then FontForge would save it with a 3,0 (symbol) cmap
> entry. That was a misunderstanding on my part. I assumed a "Symbol"
> encoding in Adobe's sense was the same as MicroSoft's, and that was
> wrong. So in modern FontForges the font is saved with a 3,1 (unicode)
> encoding instead. If you really want a symbol (3,0) cmap vector, use
> Generate($2, "sym.ttf", 0);
> instead.

TT_PLATFORM_MICROSOFT = 3
TT_MS_ID_SYMBOL_CS = 0
TT_MS_ID_UNICODE_CS = 1

So by (3,1) George means that modern Fontforge versions create a Unicode
charmap for marlett.ttf. But that's not the problem, the problem is that
Fontforge now sets Latin1 bit in the ulCodePageRange1 fileld in the OS2
TrueType header.

here is the last response of my question from George Williams:

George Williams wrote:
> Stephan Wezel wrote:
> > Currently they still thinking it isn't a wine bug but an bug in
> > fontforge. But when you say that the old behaviour on which the wine
> > build system currently relies was wrong then i will report it on the
> > mentioned wine bugreport.
>
> That is correct.
>
> I had assumed that adobe's symbol encoding (which is what "symbol" means
> inside fontforge) was the same as Microsoft's 3,0 cmap entry. It is not.
> I presume that they made the same mistake, aided in doing so by my
> mistake.
>
> In my opinion, marlett.sfd is erroneous. It claims an adobe symbol
> encoding, which it does not, in fact, have. I don't think that could be
> created with fontforge, I think someone must have hand edited the file
> to produce that peculiar melange.

So it seems really a bug in the wine build-system part which generates the font.
Because it relays on a behavior of fontforge which was incorrect.

That still doesn't answer the question why fontforge now sets Latin1 *and*
Symbol bits in the ulCodePageRange1 fileld in the OS2 TrueType header, while
previously it only set the Symbol one.

At the very least there should be a way to tell fontforge what code page
bits should be set in the ulCodePageRange1 fileld, and relying on the file
extension looks wrong to me.

Also, there is a thing called backwards compatibility. A behaviour of
fontforge that was valid for years now suddenly called broken, making
previously valid .sfd files useless.

then write directly to George Williams, because i'm not firm enough with fontforge and generating fonts.

(In reply to comment #22)
> then write directly to George Williams, because i'm not firm enough with
> fontforge and generating fonts.

Can you please point him to this bug, and ask to comment here? That way
everybody can participate in the discussion of the problem, and that will
be tracked for future reference.

(In reply to comment #23)
> (In reply to comment #22)
> > then write directly to George Williams, because i'm not firm enough with
> > fontforge and generating fonts.
>
> Can you please point him to this bug, and ask to comment here? That way
> everybody can participate in the discussion of the problem, and that will
> be tracked for future reference.
>
I have several times mentioned this bug in my mails. But it is curious why all furthermore mails aren't in the sourceforge fontforgedevel ml archiv.

He is also a bit tired about this topic, due my bad knowledge about this topic. And how i had tried to explain your concerns why it seems to be fontforge bug.

I will try it but i hope he doesn't gent angry about me.

I have explained this three times on the fontforge mailing list.
   "I have answered three questions, and that is enough,"
   Said his father. "Don't give yourself airs!
   Do you think I can listen all day to such stuff?
   Be off, or I'll kick you down stairs.

The problem is that marlett.sfd was taking advantage of a bug in fontforge. That bug has now been fixed. marlett.sfd is rather problematic itself as it claims an adobe symbol encoding when it does not, in fact, have one. I hope someone hand-edited the sfd file, because if not there is another bug in fontforge.

At one point I assumed that MicroSoft's 3,0 cmap entry was the same as Adobe's symbol encoding (To fontforge a "symbol" encoding is Adobe's). It is not. In fact MicroSoft's 3,0 cmap entry is not a true encoding as there is no charset associated with it. At any rate FontForge no longer generates a 3,0 cmap entry for something with Adobe's Symbol encoding.

As Stephan says I am extremely tired of repeating this same point over and over. A dieu.

George, I understand your frustration on this, it must be mostly caused by
a miscommunication. Could you please answer the questions in the comments
#19 and #21?

*** Bug 10952 has been marked as a duplicate of this bug. ***

*** Bug 11538 has been marked as a duplicate of this bug. ***

I just performed a simple test: took marlett.ttf from XP (it has only symbol
bit set in ulCodePageRange1 (0x80000000)), opened it with Fontforge and saved
as marlett.sfd. Then I generated marlett.ttf from marlett.sfd using genttf.ff
from wine/fonts (Open($1; Generate($2, "ttf", 0);)

New marlett.ttf has ulCodePageRange1 set to 0x80000001, i.e. both symbol and
latin1 unicode ranges. Does that qualify as a fontforge bug?

If the latin1 bit is not set on a normal font then windows won't use it for anything useful. So FontForge pretty much always sets this bit when outputting normal fonts. When outputting symbol fonts it does not set this bit as it isn't applicable.

The root of the problem, as I keep saying (five times? six?), is that fontforge no longer generates a 3,0 cmap entry for marlett.sfd (unless you specifically request a symbol encoding). This has a number of implications, including the way the OS/2 code pages are defaulted.

If you don't want fontforge to default the setting of the code pages/unicode ranges then you can set them explicitly in Element->Font Info->OS/2->Charsets.

No this doesn't count as a fontforge bug. If you believe you have a fontforge bug please report them on the fontforge mailing list. The wine bug-tracker is not an appropriate place.

>That still doesn't answer the question why fontforge now sets Latin1 *and*
>Symbol bits in the ulCodePageRange1 fileld in the OS2 TrueType header, while
>previously it only set the Symbol one.
When I use fontforge to generate a truetype font with a symbol encoding it does *NOT* set the latin1 bit.
   Open("marlett.sfd")
   Generate("marlett.sym.ttf")

>Also, there is a thing called backwards compatibility. A behaviour of
>fontforge that was valid for years now suddenly called broken, making
>previously valid .sfd files useless.
As I pointed out in my previous post marlett is not a valid sfd file. It claims an encoding (adobe symbol) which it does not have.
  There seems to be an assumption that FontForge's symbol encoding (which is Adobe's) means the same as the symbol cmap type. That is not the case.
  The behavior you are depending on was never documented.

Now can we please leave this topic?
  The old behavior was wrong. Marlett.sfd is mildly wrong. You have a fix which works.

As I recall, I once tried to override things in "Element->Font Info->OS/2->Charsets" as an experiment, but without success. It set the latin1 bit regardless. (In any case, even if that had worked, I guess the sfd file must be patched.)

(In reply to comment #30)
> If the latin1 bit is not set on a normal font then windows won't use it for
> anything useful.

This is not true. There are other useful unicode ranges in the fonts
besides Latin1.

> So FontForge pretty much always sets this bit when outputting
> normal fonts. When outputting symbol fonts it does not set this bit as it isn't
> applicable.

What if the font developer creates a font with several unicode ranges,
including symbol?

> The root of the problem, as I keep saying (five times? six?), is that fontforge
> no longer generates a 3,0 cmap entry for marlett.sfd (unless you specifically
> request a symbol encoding). This has a number of implications, including the
> way the OS/2 code pages are defaulted.

Font character mappings and unicode ranges are different things, not related
to each other. It's perfectly legal to have a unicode character map, and
simultaneously have a symbol unicode range in the font.

> If you don't want fontforge to default the setting of the code pages/unicode
> ranges then you can set them explicitly in Element->Font Info->OS/2->Charsets.

As Ove mentioned, that doesn't work for some reason.

> No this doesn't count as a fontforge bug. If you believe you have a fontforge
> bug please report them on the fontforge mailing list. The wine bug-tracker is
> not an appropriate place.

This bug a special. I has been closed as invalid, but reopened later to better
understand the problem.

> >That still doesn't answer the question why fontforge now sets Latin1 *and*
> >Symbol bits in the ulCodePageRange1 fileld in the OS2 TrueType header, while
> >previously it only set the Symbol one.
> When I use fontforge to generate a truetype font with a symbol encoding it does
> *NOT* set the latin1 bit.
> Open("marlett.sfd")
> Generate("marlett.sym.ttf")

My impression is that the font is being created based on the information in
the font file. If there is a need in a hack to specify font encoding using
file extension (what if a developer needs several of them in the single font?)
then this looks like a limitation of the file format, and should be considered
to be fixed.

Again I'd like to point out that .ttf -> .sfd -> .ttf path leads to creation
of a not compatible font to an original one, regardless which unicode ranges
the font contains.

> >Also, there is a thing called backwards compatibility. A behaviour of
> >fontforge that was valid for years now suddenly called broken, making
> >previously valid .sfd files useless.
> As I pointed out in my previous post marlett is not a valid sfd file. It claims
> an encoding (adobe symbol) which it does not have.
> There seems to be an assumption that FontForge's symbol encoding (which is
> Adobe's) means the same as the symbol cmap type. That is not the case.
> The behavior you are depending on was never documented.
> Now can we please leave this topic?
> The old behavior was wrong. Marlett.sfd is mildly wrong. You have a fix which
> works.

I'm sorry, but I don't think that a hack with a file extension qualifies as
a fix. I'd prefer to have a real solution which doesn't prevent adding other
unicode ranges to marlett in future.

> As Ove mentioned, that doesn't work for some reason.
You are right. There's now a patch in fontforge which makes it work.

>I'm sorry, but I don't think that a hack with a file extension qualifies as
>a fix. I'd prefer to have a real solution which doesn't prevent adding other
>unicode ranges to marlett in future.
Well, the approach you are using was never supposed to work. The fact that it did was a bug.

(In reply to comment #33)
> > As Ove mentioned, that doesn't work for some reason.
> You are right. There's now a patch in fontforge which makes it work.

Thanks, once it's in a released version of fontforge I'll try to use it.

> >I'm sorry, but I don't think that a hack with a file extension qualifies as
> >a fix. I'd prefer to have a real solution which doesn't prevent adding other
> >unicode ranges to marlett in future.
> Well, the approach you are using was never supposed to work. The fact that it
> did was a bug.

George, for unknown to me reason, you are avoiding the questions, and one of
them is how to create a .ttf from .sfd *without* using a file extension hack,
i.e. is there a way to specify unicode ranges of the font in an .sfd file?

:-) I am avoiding the question because I am so tired of this argument I am not reading them. I give up. It is easier for me to reinstate the original behavor than to keep arguing. I am sorry for any problems I have caused.

(In reply to comment #35)
> :-) I am avoiding the question because I am so tired of this argument I am not
> reading them. I give up. It is easier for me to reinstate the original behavor
> than to keep arguing. I am sorry for any problems I have caused.

Now I'm confused. Am I right that instead of understanding the problem and
fixing it properly you prefer to restore previous, admittedly broken behaviour?

I think so, but only because the interaction with the wine
developers has been so dysfunctional. I've been watching,
and the conversation hasn't been exactly pleasant. Can't
say I blame him.

There might be bugs on both sides, and a calm investigation
by somebody who had enough time to look at both sides himself
would probably get to the bottom of it pretty quickly.

I believe that I did everything I could do from Wine point of view:
I described what and where the problem is. Now it's just up to fontforge
developer(s) to make .sfd -> .ttf (and probably .ttf -> .sfd) font generation
work properly without any external hacks like proposed file extensions, since
that apparently doesn't allow to create fonts with Symbol and some other
unicode range bits set in ulCodePageRange1.

If "Encoding: Symbol" is really wrong in marlett.sfd, let's remove it, but
we need a working solution/replacement which works, and preferably works
with old, current and new fontforge versions.

(For some reason George got removed from the cc: list, re-adding since he is
a key person for this bug)

George, is there anything we can do to help better understanding and fixing
this bug?

I found a TGMarlett.sfd somewhere on the web and opened it in fontforge, then generated the font as "TrueType (Symbol)" and it's working correctly in Wine.

Is that what you guys were trying to do, or is there a different way that you wanted to do it?

(In reply to comment #40)
> I found a TGMarlett.sfd somewhere on the web and opened it in fontforge, then
> generated the font as "TrueType (Symbol)" and it's working correctly in Wine.
> Is that what you guys were trying to do, or is there a different way that you
> wanted to do it?

You have omitted all the details. What version of fontforge are you using? Does
the script that Wine is using to generate marlett.ttf work for you?

(In reply to comment #41)
> (In reply to comment #40)
> > I found a TGMarlett.sfd somewhere on the web and opened it in fontforge, then
> > generated the font as "TrueType (Symbol)" and it's working correctly in Wine.
> > Is that what you guys were trying to do, or is there a different way that you
> > wanted to do it?
>
> You have omitted all the details. What version of fontforge are you using? Does
> the script that Wine is using to generate marlett.ttf work for you?
>

I apologize; I forgot.

Fontforge version: 0.0.20071110-1build2 from the Ubuntu Hardy repository.
I don't know where the script is located though, could you tell me so I could test?

(In reply to comment #42)
> Fontforge version: 0.0.20071110-1build2 from the Ubuntu Hardy repository.
> I don't know where the script is located though, could you tell me so I could
> test?

Obviously the script is in the Wine source repository. Have you read all the
comments in the bug? I'm sorry, but if you don't understand the problem(s)
behind this bug most likely there is no need to test/report anything.

I removed myself from the bug. I have done so again.

I have reverted the change you complained about. I don't see that there is anything more I can do for you. Please stop bothering me.

(In reply to comment #43)
> Obviously the script is in the Wine source repository. Have you read all the
> comments in the bug? I'm sorry, but if you don't understand the problem(s)
> behind this bug most likely there is no need to test/report anything.
>

Yes, at least twice; Obviously I missed something though.

I'm sure if there was a clear description of how to test and reproduce the
bug I wouldn't have any problems testing and possibly actually helping in
this bug. As far as I could see, there was no method, nor a way to find
the method, of testing this.

At any rate, I can see I'm unwanted here, so I'm removing myself from
the CC list. Dealing with Windows is easier than dealing with some of
the Wine developers(Though, I'm not going back to Windows).

Have a nice day, and bye.

That's two people we've driven away. This bug is cursed!

This bug has all the information about the problem, there is nothing to add
or explain more. Look for ulCodePageRange1 in the comments.

That's very unfortunate that George resigned, and decided to revert the change
instead of fixing the problem properly. Even if it's only marlett.ttf now,
it's clear that fontforge incorrectly handles unicode ranges in the fonts,
and even is not able to produce teh same consistent results when going a
ttf -> sfd -> ttf route. That means that broken marlett.ttf fonts are going
to cause us continuous headache.

Hi,

(In reply to comment #47)
> This bug has all the information about the problem, there is nothing to add
> or explain more. Look for ulCodePageRange1 in the comments.
>
> That's very unfortunate that George resigned, and decided to revert the change
> instead of fixing the problem properly. Even if it's only marlett.ttf now,
> it's clear that fontforge incorrectly handles unicode ranges in the fonts,
> and even is not able to produce teh same consistent results when going a
> ttf -> sfd -> ttf route. That means that broken marlett.ttf fonts are going
> to cause us continuous headache.

Well headaches are bad, but brokeneness is worse.
Actually there is no solution to this, which can be used as default.

This makes the life of a package maintainer more worse

(In reply to comment #48)
> Well headaches are bad, but brokeneness is worse.
> Actually there is no solution to this, which can be used as default.
> This makes the life of a package maintainer more worse

Wine depends on fontforge, and we have any control on this (it even looks
much worse taking into account George's stance in the comments above).

To me it's clear that it's the fontforge bug, why this bug stays open
is beyond my understanding. I'd suggest to keep reporting this to George,
if enough people starts doing that perhaps there is a chance it will be
fixed properly.

s/we have any control on this/we have no any control on this/

Continuing to report it to George in the same way
will just annoy him.

We need somebody to hand him a regression test case
and probably a fix.

(In reply to comment #51)
> Continuing to report it to George in the same way
> will just annoy him.
> We need somebody to hand him a regression test case
> and probably a fix.

That's not a regression, that's how fontforge handles unicode ranges of
the font in general.

Whatever. If we want them to change something,
we have to give them a good test case and a fix.

(In reply to comment #53)
> Whatever. If we want them to change something,
> we have to give them a good test case

They have it already, and I mentioned it several times in this bug: open
marlett.ttf from windows, save it to marlett.sfd, then open marlett.sfd and
save it to marlett.ttf. George insists that the .sym extension has to be
used, but that's not acceptable as I explained above.

> and a fix.

That requires an intimate knowledge of fontforge, and needs to be fixed by
a fontforge developer.

> I mentioned it several times in this bug

Not in a way that doesn't annoy them. An automated
test would be better.

We've pissed them off; now we either have to charm them
by giving them a very, very nice test case, and/or
fix it ourselves, and/or somehow otherwise get
back on their good side. Continued whining about
how they aren't fixing the bug for us won't cut it.

(In reply to comment #55)
> > I mentioned it several times in this bug
> Not in a way that doesn't annoy them. An automated
> test would be better.

It looks like 'Generate' keyword in fontforge scripting language doesn't
support creation of .sfd (fontforge own file format) files, otherwise
something like the following script would qualify as an automated test
I'd guess (and yes, the fact that Wine already has/uses a similar script
has already been pointed out):

Open("marlett.ttf");
Generate("marlett.sfd");
Open("marlett.sfd");
Generate("marlett2.ttf");

which still requires an inspection of the unicode ranges field in the OS/2
header of the generated marlett2.ttf.

> We've pissed them off; now we either have to charm them
> by giving them a very, very nice test case, and/or
> fix it ourselves, and/or somehow otherwise get
> back on their good side. Continued whining about
> how they aren't fixing the bug for us won't cut it.

If we would be pissed off by every report pointing out bugs in Wine how far
Wine would be these days?

From what I figured from what he said, George applied a fix to the unicode ranges so they can be overridden from the .sfd file, but reverted the change that required the .sym extension. If so, Wine would be fine, just need to add those overrides to the .sfd. Maybe someone could check out the current fontforge cvs and see if it works like that now...

As for creating a test case, to save the imported ttf to a sfd file, use Save(), not Generate().

> If we would be pissed off by every report pointing out
> bugs in Wine how far Wine would be these days?

Just because we have tough hides doesn't mean
we can assume everybody else does.
I'm just recognizing how George feels, and
proposing a way we can get what we want.
It may be a bit more work for us than you feel
is right, but that's life.

From a practical standpoint, Ubuntu Hardy is coming out soon, and while Wine can be patched at any time before release I believe the version of fontforge might be set. We may want to have a workaround anyway.

Then again, Ubuntu is not opposed to patching fontforge if the changes we get are minimal enough.

See Launchpad bug: https://bugs.launchpad.net/bugs/199331

So, as a package maintainer, what should I do? Someone needs to point me to a patch for Wine or a patch for fontforge that I can ask to be merged in.

(In reply to comment #60)
> So, as a package maintainer, what should I do? Someone needs to point me to a
> patch for Wine or a patch for fontforge that I can ask to be merged in.

As a package maintainer you could use an older fontforge version known to
produce correct Wine fonts.

hi,

(In reply to comment #61)
> (In reply to comment #60)
> > So, as a package maintainer, what should I do? Someone needs to point me to a
> > patch for Wine or a patch for fontforge that I can ask to be merged in.
>
> As a package maintainer you could use an older fontforge version known to
> produce correct Wine fonts.
>
You can't. You use what your distro gives you.
If this version of fontforge is broken, you can try to upgrade, and hopefully nothing else will break.
Or you stay with the issue until the next release cycle.

(In reply to comment #60)
> From a practical standpoint, Ubuntu Hardy is coming out soon, and while Wine
> can be patched at any time before release I believe the version of fontforge
> might be set. We may want to have a workaround anyway.
>
> Then again, Ubuntu is not opposed to patching fontforge if the changes we get
> are minimal enough.
>
> See Launchpad bug: https://bugs.launchpad.net/bugs/199331
>
>
> So, as a package maintainer, what should I do? Someone needs to point me to a
> patch for Wine or a patch for fontforge that I can ask to be merged in.
>

you could use my workaround until a proper fix exist:
http://bugs.winehq.org/attachment.cgi?id=10629

(In reply to comment #62)
> hi,
> (In reply to comment #61)
> > (In reply to comment #60)
> > > So, as a package maintainer, what should I do? Someone needs to point me to a
> > > patch for Wine or a patch for fontforge that I can ask to be merged in.
> >
> > As a package maintainer you could use an older fontforge version known to
> > produce correct Wine fonts.
> >
> You can't. You use what your distro gives you.

If you can't use a fontforge version different from your official distro one
then your Wine packages will be broken, and you will have to cope with all
bug reports of your users on your own.

The Debian package has worked around it since 0.9.53...

It's been reported that fontforge-20080302 creates correct marlett.ttf.

This bug can be closed as invalid.

Marking wontfix (for Wine).

Please use workarounds in packages for the meantime, and if you can please provide a proper fix for fontforge - we certainly owe them a lot.

Closing WONTFIX.

Removing deprecated 'All' Platform.