Elaborate on what AST changes Black might perform

diff --git a/CHANGES.md b/CHANGES.md
index 5b9f574a40866c38b23c8b88f7b6d8568f2327bb..6675c484c75b173040df9c26c3d3d5beaa0ab329 100644 (file)
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -6,8 +6,9 @@
 
 - Fix crash if the user configuration directory is inaccessible. (#2158)
 
-- Clarify that _Black_ may change the AST, especially when cleaning up docstrings.
-  (#2159)
+- Clarify
+  [circumstances](https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism)
+  in which _Black_ may change the AST (#2159)
 
 #### _Packaging_
 

diff --git a/README.md b/README.md
index c7577fbbd041eaf2a8d4c1230ffaf9b74c7f62d7..41cea761df564c4efb9ca60d323deb5d6073ac9b 100644 (file)
--- a/README.md
+++ b/README.md
@@ -237,11 +237,11 @@ is that **until the formatter becomes stable, you should expect some formatting
 change in the future**. That being said, no drastic stylistic changes are planned,
 mostly responses to bug reports.
 
-Also, as a temporary safety measure, _Black_ will check that the reformatted code still
-produces a valid AST that is mostly equivalent to the original. This slows it down. If
-you're feeling confident, use `--fast`. In a few contexts, Black does make changes to
-the AST: it cleans up whitespace in docstrings, adds or removes parentheses in some
-`del` statements, and may move around type comments.
+Also, as a safety measure which slows down processing, _Black_ will check that the
+reformatted code still produces a valid AST that is effectively equivalent to the
+original (see the
+[Pragmatism](https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism)
+section for details). If you're feeling confident, use `--fast`.
 
 ## The _Black_ code style
 

diff --git a/docs/the_black_code_style.md b/docs/the_black_code_style.md
index 67ccb8aab11b53f8f38467ed58522d21297bf2b0..39a452ff9a81d74abf20ce88e962e579e7a9cfae 100644 (file)
--- a/docs/the_black_code_style.md
+++ b/docs/the_black_code_style.md
@@ -464,3 +464,32 @@ exception to this rule is r-strings. It turns out that the very popular
 default by (among others) GitHub and Visual Studio Code, differentiates between
 r-strings and R-strings. The former are syntax highlighted as regular expressions while
 the latter are treated as true raw strings with no special semantics.
+
+### AST before and after formatting
+
+When run with `--safe`, _Black_ checks that the code before and after is semantically
+equivalent. This check is done by comparing the AST of the source with the AST of the
+target. There are three limited cases in which the AST does differ:
+
+1. _Black_ cleans up leading and trailing whitespace of docstrings, re-indenting them if
+   needed. It's been one of the most popular user-reported features for the formatter to
+   fix whitespace issues with docstrings. While the result is technically an AST
+   difference, due to the various possibilities of forming docstrings, all realtime use
+   of docstrings that we're aware of sanitizes indentation and leading/trailing
+   whitespace anyway.
+
+2. _Black_ manages optional parentheses for some statements. In the case of the `del`
+   statement, presence of wrapping parentheses or lack of thereof changes the resulting
+   AST but is semantically equivalent in the interpreter.
+
+3. _Black_ might move comments around, which includes type comments. Those are part of
+   the AST as of Python 3.8. While the tool implements a number of special cases for
+   those comments, there is no guarantee they will remain where they were in the source.
+   Note that this doesn't change runtime behavior of the source code.
+
+To put things in perspective, the code equivalence check is a feature of _Black_ which
+other formatters don't implement at all. It is of crucial importance to us to ensure
+code behaves the way it did before it got reformatted. We treat this as a feature and
+there are no plans to relax this in the future. The exceptions enumerated above stem
+from either user feedback or implementation details of the tool. In each case we made
+due diligence to ensure that the AST divergence is of no practical consequence.