]> git.madduck.net Git - etc/vim.git/commit

madduck's git repository

Every one of the projects in this repository is available at the canonical URL git://git.madduck.net/madduck/pub/<projectpath> — see each project's metadata for the exact URL.

All patches and comments are welcome. Please squash your changes to logical commits before using git-format-patch and git-send-email to patches@git.madduck.net. If you'd read over the Git project's submission guidelines and adhered to them, I'd be especially grateful.

SSH access, as well as push access can be individually arranged.

If you use my repositories frequently, consider adding the following snippet to ~/.gitconfig and using the third clone URL listed for each project:

[url "git://git.madduck.net/madduck/"]
  insteadOf = madduck:

Let string splitters respect `East_Asian_Width` property (#3445)
authorHong Minhee (洪 民憙) <hong@minhee.org>
Sun, 19 Mar 2023 23:09:57 +0000 (08:09 +0900)
committerGitHub <noreply@github.com>
Sun, 19 Mar 2023 23:09:57 +0000 (19:09 -0400)
commitef6e079901d53a42dfae4ab10b081ce7a73a47b5
tree32e59de839dffa7e1d70cf821a4736707cdad919
parent5c064a986c388e2be1e448c3e4b28e5f5ba7ce5a
Let string splitters respect `East_Asian_Width` property (#3445)

This patch changes the preview style so that string splitters respect
Unicode East Asian Width[^1] property.  If you are not familiar to CJK
languages it is not clear immediately.  Let me elaborate with some
examples.

Traditionally, East Asian characters (including punctuation) have
taken up space twice than European letters and stops when they are
rendered in monospace typeset.  Compare the following characters:

```
abcdefg.
글、字。
```

The characters at the first line are half-width, and the second line
are full-width.  (Also note that the last character with a small
circle, the East Asian period, is also full-width.)  Therefore, if we
want to prevent those full-width characters to exceed the maximum
columns per line, we need to count their *width* rather than the number
of characters.  Again, the following characters:

```
글、字。
```

These are just 4 characters, but their total width is 8.

Suppose we want to maintain up to 4 columns per line with the following
text:

```
abcdefg.
글、字。
```

How should it be then?  We want it to look like:

```
abcd
efg.
글、
字。
```

However, Black currently turns it into like this:

```
abcd
efg.
글、字。
```

It's because Black currently counts the number of characters in the line
instead of measuring their width. So, how could we measure the width?
How can we tell if a character is full- or half-width? What if half-width
characters and full-width ones are mixed in a line? That's why Unicode
defined an attribute named `East_Asian_Width`. Unicode grouped every
single character according to their width in fixed-width typeset.

This partially addresses #1197, but only for string splitters. The other
parts need to be fixed as well in future patches.

This was implemented by copying rich's own approach to handling wide
characters: generate a table using wcwidth, check it into source
control, and use in to drive helper functions in Black's logic. This
gets us the best of both worlds: accuracy and performance (and let's us
update as per our stability policy too!).

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
CHANGES.md
scripts/make_width_table.py [new file with mode: 0644]
src/black/_width_table.py [new file with mode: 0644]
src/black/lines.py
src/black/strings.py
src/black/trans.py
tests/data/preview/long_strings__east_asian_width.py [new file with mode: 0644]