Escaping quote characters in raw string literals causes a tokenizer error

[lpetre]

This string breaks the native parser: rf"\"{feature_name.lower()}\""

$ echo 'rf"\"{feature_name.lower()}\""' | LIBCST_PARSER_TYPE=native python -m libcst.tool print - > /dev/null
Traceback (most recent call last):
  File "<string>", line 49, in <module>
  File "<string>", line 47, in __run
  File "lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "libcst/__main__.py", line 25, in <module>
    sys.exit(fbcode_main())
  File "libcst/__main__.py", line 19, in fbcode_main
    return libcst.tool.main(
  File "libcst/tool.py", line 834, in main
    return lookup.get(args.action or None, _invalid_command)(proc_name, command_args)
  File "libcst/tool.py", line 278, in _print_tree_impl
    tree = parse_module(
  File "libcst/_parser/entrypoints.py", line 109, in parse_module
    result = _parse(
  File "libcst/_parser/entrypoints.py", line 55, in _parse
    return parse(source_str)
libcst._exceptions.ParserSyntaxError: Syntax Error @ 0:1.
tokenizer error: unexpected characters after a line continuation

rf"\"{feature_name.lower()}\""
^
$

$ echo 'rf"\"{feature_name.lower()}\""' | python -m libcst.tool print - > /dev/null
$

[lpetre]

Not sure if is the same issue, but this also breaks the native parser: f"regexp_like(path, '.*\{file_type}$')"

[lpetre]

This is another example: f"\{{\}}" which fails both libcst parsers but python3 -m py_compile parses it.

[zsol]

Right, so after digging into this a bit, there are two bugs:

1. f-strings and escaping {. I opened Escaping opening braces in f-strings causes a tokenizer error #699 and I think I have a fix for that.
2. the original example in this issue has nothing to do with f-strings, and is a tokenizer bug where it doesn't respect escaping ' and " quotes in raw strings.

Apparently it's possible to write r"\"" in Python, and this string has two characters, a backslash and a double quote.