Parse error found with Hypothesmith

[Zac-HD]

weird = "pass #\r#\n"
exec(weird)                                     # works, as does `compile` etc.
black.format_str(weird, mode=black.FileMode())  # fails with `black.InvalidInput`

Operating system: Windows
Python version: 3.7.3
Black version: 19.3b0
Does also happen on master: yes (website hangs)

I found this weird little string with hypothesmith, a tool for generating Python source code with Hypothesis that I started at the PyCon NA sprints and published at the PyCon AU sprints. Currently it works based on an approximate grammar, post-processing, and filtering.

CC @ambv who I know was interested in fuzzing 😄. Next steps... I think a combination of targeted mutation (open PR upstream) and spending more than a few minutes of CPU time could turn up a lot more weird things!

[zsol]

Nice!

[zsol]

FWIW this is also a bug in upstream lib2to3. You might want to report it on http://bugs.python.org/

[MeGaGiGaGon]

This appears to have been solved at some point. If I set up the file using Alt-013 for CR and Alt-010 for LF

And after running it through black

I couldn't figure out how to type just CR/LF on the playground, so no playground link.

This also appears to have been fixed at some point in lib2to3

$ py -3.12 -c "from lib2to3.pgen2.tokenize import generate_tokens;from io import StringIO;print(*generate_tokens(StringIO('pass #\r#\n').readline))"
<string>:1: DeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
(1, 'pass', (1, 0), (1, 4), 'pass #\r#\n') (53, '#', (1, 5), (1, 6), 'pass #\r#\n') (58, '\r', (1, 6), (1, 7), 'pass #\r#\n') (53, '#', (1, 7), (1, 8), 'pass #\r#\n') (4, '\n', (1, 8), (1, 9), 'pass #\r#\n') (0, '', (2, 0), (2, 0), '')

But I couldn't find the relevant issues/PRs, and it doesn't really matter since lib2to3 is gone in 3.13.

[JelleZijlstra]

Nice, would you mind making a PR to add Zac's example as a test case so we can ensure it stays fixed?

[MeGaGiGaGon]

While trying to add a test for this, I've found that this is not solved, but also probably only observable when running directly using black's internals.
When running through a file, open is used with newline=None. This is universal newline mode, so \r is replaced with \n and formatting works.
When running through STDIN, a TextIOWrapper is used with newline=None, to the same effect.
From what I can tell, the only way to get a raw \r to be passed into black is by directly calling the internal API, which puts this issue in a weird state, since #779 hasn't been resolved yet.
I think lib2to3 has solved the problem, since even when passing in input using open("test.py", newline='').readline it still works.
This also means there is a difference between how users run black and how black's tests run black, which is a whole other can of worms.
Edit for link creation: This might also get resolved by #4536 since the issue is a bug in the parser.