bpo-40335: Correctly handle multi-line strings in tokenize error scenarios

[pablogsal]

https://bugs.python.org/issue40335

[pablogsal]

CC: @asottile Can you check if this PR runs correctly with your test suite?

[asottile]

yep pyflakes is now passing (with a minor patch, but this patch is expected):

$ git diff
diff --git a/pyflakes/test/test_api.py b/pyflakes/test/test_api.py
index 128aa69..b579ac8 100644
--- a/pyflakes/test/test_api.py
+++ b/pyflakes/test/test_api.py
@@ -449,6 +449,8 @@ def baz():
         with self.makeTempFile(source) as sourcePath:
             if PYPY:
                 message = 'end of file (EOF) while scanning triple-quoted string literal'
+            elif sys.version_info >= (3, 9):
+                message = 'invalid string prefix'
             else:
                 message = 'invalid syntax'
 

[pablogsal]

but this patch is expected

Yeah, that was on purpose 😉

[lysnikolaou]

Thanks, @pablogsal, for the fix! 😃

[gvanrossum]

Weak approve: I don't feel super confident with the ins and outs of this code, but if the tests pass, I feel it's fine.