From: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Date: Wed, 1 Dec 2021 02:01:36 +0000 (-0800)
Subject: Reduce usage of regex (#2644)
X-Git-Url: https://git.madduck.net/etc/vim.git/commitdiff_plain/5e2bb528e09df368ed7dea6b7fb9c53e799a569f?hp=5e2bb528e09df368ed7dea6b7fb9c53e799a569f

Reduce usage of regex (#2644)

This removes all but one usage of the `regex` dependency. Tricky bits included:
- A bug in test_black.py where we were incorrectly using a character range. Fix also submitted separately in #2643.
- `tokenize.py` was the original use case for regex (#1047). The important bit is that we rely on `\w` to match anything valid in an identifier, and `re` fails to match a few characters as part of identifiers. My solution is to instead match all characters *except* those we know to mean something else in Python: whitespace and ASCII punctuation. This will make Black able to parse some invalid Python programs, like those that contain non-ASCII punctuation in the place of an identifier, but that seems fine to me.
- One import of `regex` remains, in `trans.py`. We use a recursive regex to parse f-strings, and only `regex` supports that. I haven't thought of a better fix there (except maybe writing a manual parser), so I'm leaving that for now.

My goal is to remove the `regex` dependency to reduce the risk of breakage due to dependencies and make life easier for users on platforms without wheels.
---