Switch to a new parser

[JelleZijlstra]

Currently, Black uses a vendored version of lib2to3 for parsing. This works well for parsing Python 2 and early Python 3, but Python has now moved on to a PEG-based parser (PEP 617), and lib2to3 is no longer being maintained.

So we need a new parser. There are a few existing options that we could leverage (Parso, LibCST), but it's going to be a lot of work to do the migration. WE're doing some early brainstorming in a Google doc. This issue exists so that we have a public record that we know this is a problem.

Concrete pieces of syntax that are blocked by this new grammar include parenthesized context managers and the match statement in Python 3.10. (#2242 through #2586, #2667, #2758)

[kamahen]

The main bug is: https://bugs.python.org/issue40360 (and also https://bugs.python.org/issue36541).

I think that there's a fairly straightforward way of wrapping the new Python parser to give the necessary functionality that "Black" (and other source-level tools) need. However, it's a non-trivial amount of work, and I'm loathe to do it unless I'm sure it'll be used and that nobody else is doing the work. (There appears to be one existing wrapper, namely leoAst.py; I've looked at it a bit but it seems much more complicated than necessary and therefore could be both difficult to use and a maintenance issue.)

Some other discussion at kamahen/pykythe#27 google/yapf#825 (comment) , google/yapf#894 (comment) and elsewhere.

[ianliu]

Has treesitter been considered? It already implements a parser for python here: https://github.com/tree-sitter/tree-sitter-python and I think it allows to build formaters upon it.

[JelleZijlstra]

@ianliu interesting, I hadn't heard of that!

Looking at the Python bindings (https://github.com/tree-sitter/py-tree-sitter), it might be hard to get it to work for us:

• Installing it requires a C compiler on most platforms (there's wheels only for MacOS/3.8)
• And that doesn't even give you a Python grammar: you have to clone a repo and build the grammar at runtime.

That sounds like it would lead to a lot of people with mildly exotic systems who'd be unable to install Black if it depended on this library.