wrong width for U+00AD

[stevengj]

Hi, I was looking at your wcwidth library for comparison, since in the utf8proc library we are also implementing a similar feature (see JuliaStrings/utf8proc#2). The first disagreement that I came across between your implementation and ours was for U+00AD (soft hyphen), where you seem to give 1

>>> from wcwidth import wcwidth
>>> wcwidth(unichr(173))
1

and we give zero (a soft hyphen is used for line breaking, but is ordinarily not printed). In general, we return 0 for most characters in category Cf (formatting control characters). The wcwidth function on MacOS 10.10.2 also returns -1 (not printable) for this code point.

Am I calling your implementation incorrectly? This is for git master of wcwidth.

[stevengj]

In case it is helpful, the draft table of character widths that we are currently planning to use can be found in this CharWidths.txt gist (each line of which is codepoints; width) where non-printing characters are assigned a width of 0. This is generated automatically from the unicode 7 tables combined with font metrics from GNU unifont, as described in JuliaStrings/utf8proc#27

[jquast]

Interesting, what terminal are you testing the "character cells consumed when printed" on OSX? I too am using OSX, and on iTerm2 it displays as "a-b", consuming 3 characters, so wcwidth would be correct, here... I would need to see evidence of it not forwarding the cell when printed on at least some terminal emulators, and file bugs for the others. Just to be very clear, the purpose of wcwidth is "printable width on a terminal", and not firefox or anything else (for which such character is hidden).

Also, I don't necessarily trust the OS-provided 'wcwidth', they are typically based on very old (5-10 years old) unicode specifications. I have a program I've tested on osx and linux, both are wildly different, and in each case my version was correct: https://github.com/jquast/wcwidth/blob/master/bin/wcwidth-libc-comparator.py

The combining and wide character tables are programmatically updated by "python setup.py update", which is similar to your https://github.com/JuliaLang/utf8proc/pull/27/files#diff-3832b9cfe2fc10d35ac5c63d9b7b8133R20

There is no unicode specification reference tables for 0-width characters that I know of, so its just hardcoded here https://github.com/jquast/wcwidth/blob/master/wcwidth/wcwidth.py#L161-171

[jquast]

Using the 'Cf' category listings on iTerm2, it appears the following all consume 1 character cell, some with symbols, some simply by blanks

00AD
0600
0601
0602
0603
0604
0605
061C
06DD
070F
200E
200F
202A
202B
202C
202D
202E
2060
2061
2062
2063
2064
2066
2067
2068
2069
206A
206B
206C
206D
206E
206F
FFF9
FFFA
FFFB

And the following consume 0 cells:

180E
200B
200C
200D
FEFF

which may indeed need to be supported by wcwidth once i test a few more terminals

[stevengj]

(We don't trust the system-provided wcwidth either, for the same reason as you, which is why we compute the widths independently. However, the OSX 10.10.2 wcwidth agrees with our results when it returns a nonnegative value, so it mostly seems to have errors of omission—it returns -1 for many valid printable characters from recent Unicode standards. Moreover, U+00AD has been part of Unicode since 1993, so I would think that most wcwidth implementations would handle it properly.)

There is an interesting article on the soft hyphen, which apparently has had a controversial history, and is rendered in different ways depending on the font and the rendering system. I'm not sure what the right answer is here, but the Unicode standard seems to somewhat favor the viewpoint that it should be invisible although it leaves it up to the implementation. However, the article mentions that the Unicode FAQ does say In a terminal emulation environment, particularly in ISO-8859-1 contexts, one could display the soft hyphen as a hyphen in all circumstances and maybe that is what is done in practice.

cc: @jiahao and @StefanKarpinski.

[stevengj]

Note that the Arabic characters U+0601 etc. are defined by the unicode standard as exceptions to usual rule that Cf characters are invisible:

• Unlike most other format control characters, however, they should be rendered with a visible glyph, even in circumstances where no suitable digit or sequence of digits follows them in logical order. — Unicode Standard v.6.2.0, Section 8.2 - Arabic (p.256)

In contrast, e.g. U+200E is a left-to-right mark, and in my understanding is defined as an invisible formatting character that controls the direction of the text. Some terminals may give it a nonzero width (although the MacOS Terminal with the default font gives it zero width on my machine), but that seems like a bug in the terminal (or the font); it seems like it is better to return what the Unicode standard says rather than propagating a particular buggy implementation.

[jiahao]

I remember that article about the soft hyphen. Under "Modern Unicode semantics" it references UAX 14 for Unicode 7.0.0, §5.4, which says:

Unlike U+2010 hyphen, which always has a visible rendition, the character U+00AD soft hyphen (shy) is an invisible format character that merely indicates a preferred intraword line break position. If the line is broken at that point, then whatever mechanism is appropriate for intraword line breaks should be invoked, just as if the line break had been triggered by another hyphenation mechanism, such as a dictionary lookup.

The description in the following paragraphs suggests that the rendering of a soft hyphen is accomplished not by printing the soft hyphen itself, but rather by inserting an additional, printable hyphen glyph:

The inserted hyphen glyph can take a wide variety of shapes, as appropriate for the situation. Examples include shapes like U+2010 hyphen, U+058A armenian hyphen, U+180A mongolian nirugu, or U+1806 mongolian todo soft hyphen.

Based on this description it would seem that the character U+00AD by itself is nonprintable and should have a width of 0 or -1.

[stevengj]

Interestingly, the Unicode FAQ entry that the SHY article quoted seems to no longer exist — from that passage in the Unicode 7.0.0 standard that @jiahao quoted it seems like the Unicode consortium decided to put its foot down and and declare that the soft hyphen is definitely invisible, ISO 8859-1 be damned.

[jquast]

I really appreciate all of the resarch, @stevengj and @jiahao.

My decision is to use the common denominator across the most popular terminal emulators
for wcwidth. I might make a note of it in the readme that it deviates from the standard, as the
primary purpose of this project is how text is displayed by the most common (utf-8 capable)
terminal emulators.

I've made a checklist:

• create a bin/cf-print-test.py or amend bin/wcwidth-browser.py to display 'Cf' categoryv

Then, test the following and report:

• xterm (any)
• iTerm2 (osx)
• Terminal.app (osx)
• PuTTy (windows)
• gnome-terminal (covers all libvte-based emulators, linux)

I'm not sure how to gauge the "popular terminal emulators", this is just from memory.

sidenote: More importantly, how to factor their weight in wcwidth for any given
differences: perhaps some way to configure how the printable width
of such discrepancies may be reported if the consumer of wcwidth
knows their target audience's emulator (unfortunately all such terminals
borrow the common value "xterm" or "xterm-256color" as the OS
Environment Variable for TERM, and using the response of
the "answerback sequence" (^E) which at least PuTTY replies
to, but I'm afraid thats far out of scope for wcwidth, it would
require interaction with a terminal driver.

Finally, we can make a PR and release any update.

[jiahao]

@jquast thanks for your detailed consideration. As you had stated above, iTerm seems to have different needs from us at this point.

[jiahao]

However, I don't think it is possible to provide consistency across terminal environments without considering also the interactions with the choice of users' fonts. Many fonts simply have wrong advance widths for some code points.

Here is a simple rendering text for the fixed width fonts on my system. Consider

U+003C9 U+00302= \omega\hat =  ω̂

should render with the hat combining character on the omega.

U+00302 U+003C9 = \hat\omega =  ̂ω

should render with a hat to the left of omega.

[jquast]

You are correct, but terminal emulators don't typically care, they're the ones who handle the width of "printable cells" -- What is your system, is it a terminal emulator?

[jiahao]

The screenshots I pasted were taken from an IPython notebook rendering test HTML using those fonts. I can see the same spacing issues if I manually change the font in OSX Terminal and generate these characters in the Julia console REPL.

[jquast]

Version wcwidth 0.1.5 which includes better combining character width determination by PR #11 is available on pypi.

A terminal sequence may be emitted to illicit the terminal emulator to respond with its cursor position.

This can be used to manually display all questionable characters across different popular Font face profiles and terminal emulators, and programatically determine whether they consider it 0 width for such characters, making a report of the most common discrepenancies, weighing on the side of "most correct", resolving any.