Elaborate on what AST changes Black might perform

diff --git a/CHANGES.md b/CHANGES.md
index 5b9f574a40866c38b23c8b88f7b6d8568f2327bb..6675c484c75b173040df9c26c3d3d5beaa0ab329 100644 (file)
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -6,8 +6,9 @@
 
 - Fix crash if the user configuration directory is inaccessible. (#2158)
 
 
 - Fix crash if the user configuration directory is inaccessible. (#2158)
 

-- Clarify that _Black_ may change the AST, especially when cleaning up docstrings.
-  (#2159)
+- Clarify
+  [circumstances](https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism)
+  in which _Black_ may change the AST (#2159)

 
 #### _Packaging_
 
 
 #### _Packaging_
 

diff --git a/README.md b/README.md
index c7577fbbd041eaf2a8d4c1230ffaf9b74c7f62d7..41cea761df564c4efb9ca60d323deb5d6073ac9b 100644 (file)
--- a/README.md
+++ b/README.md
@@ -237,11 +237,11 @@ is that **until the formatter becomes stable, you should expect some formatting
 change in the future**. That being said, no drastic stylistic changes are planned,
 mostly responses to bug reports.
 
 change in the future**. That being said, no drastic stylistic changes are planned,
 mostly responses to bug reports.
 

-Also, as a temporary safety measure, _Black_ will check that the reformatted code still
-produces a valid AST that is mostly equivalent to the original. This slows it down. If
-you're feeling confident, use `--fast`. In a few contexts, Black does make changes to
-the AST: it cleans up whitespace in docstrings, adds or removes parentheses in some
-`del` statements, and may move around type comments.
+Also, as a safety measure which slows down processing, _Black_ will check that the
+reformatted code still produces a valid AST that is effectively equivalent to the
+original (see the
+[Pragmatism](https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism)
+section for details). If you're feeling confident, use `--fast`.

 
 ## The _Black_ code style
 
 
 ## The _Black_ code style
 

diff --git a/docs/the_black_code_style.md b/docs/the_black_code_style.md
index 67ccb8aab11b53f8f38467ed58522d21297bf2b0..39a452ff9a81d74abf20ce88e962e579e7a9cfae 100644 (file)
--- a/docs/the_black_code_style.md
+++ b/docs/the_black_code_style.md
@@ -464,3 +464,32 @@ exception to this rule is r-strings. It turns out that the very popular
 default by (among others) GitHub and Visual Studio Code, differentiates between
 r-strings and R-strings. The former are syntax highlighted as regular expressions while
 the latter are treated as true raw strings with no special semantics.
 default by (among others) GitHub and Visual Studio Code, differentiates between
 r-strings and R-strings. The former are syntax highlighted as regular expressions while
 the latter are treated as true raw strings with no special semantics.

+
+### AST before and after formatting
+
+When run with `--safe`, _Black_ checks that the code before and after is semantically
+equivalent. This check is done by comparing the AST of the source with the AST of the
+target. There are three limited cases in which the AST does differ:
+
+1. _Black_ cleans up leading and trailing whitespace of docstrings, re-indenting them if
+   needed. It's been one of the most popular user-reported features for the formatter to
+   fix whitespace issues with docstrings. While the result is technically an AST
+   difference, due to the various possibilities of forming docstrings, all realtime use
+   of docstrings that we're aware of sanitizes indentation and leading/trailing
+   whitespace anyway.
+
+2. _Black_ manages optional parentheses for some statements. In the case of the `del`
+   statement, presence of wrapping parentheses or lack of thereof changes the resulting
+   AST but is semantically equivalent in the interpreter.
+
+3. _Black_ might move comments around, which includes type comments. Those are part of
+   the AST as of Python 3.8. While the tool implements a number of special cases for
+   those comments, there is no guarantee they will remain where they were in the source.
+   Note that this doesn't change runtime behavior of the source code.
+
+To put things in perspective, the code equivalence check is a feature of _Black_ which
+other formatters don't implement at all. It is of crucial importance to us to ensure
+code behaves the way it did before it got reformatted. We treat this as a feature and
+there are no plans to relax this in the future. The exceptions enumerated above stem
+from either user feedback or implementation details of the tool. In each case we made
+due diligence to ensure that the AST divergence is of no practical consequence.