Parsing of backslashes in custom string literals seems inconsistent

[Keno]

julia> parse(readline(STDIN))
raw"\""

:(@raw_str "\"")

julia>

julia> parse(readline(STDIN))
raw"\\""
:($(Expr(:incomplete, "incomplete: invalid string syntax")))

julia> parse(readline(STDIN))
raw"\\\""
:(@raw_str "\\\\\"")

julia> dump(ans)
Expr
  head: Symbol macrocall
  args: Array{Any}((2,))
    1: Symbol @raw_str
    2: String "\\\\\""
  typ: Any

In particular there is a confusion about what backslash actually does. It certainly escapes ", but as we see in the second example, it also escapes itself. However, in the third example it suddenly doesn't appear to escape itself anymore. In particular, it does not seem to be possible to obtain a string literal that parses to @raw_str "\\\"" at the moment.

The options I see are

1. Make raw"\\"" parse to @raw_str "\\\"" (rather than an error)
2. Make raw"\\\"" parse to @raw_str "\\\"".
3. Decouple parsing from the content and always give what was actually written, i.e. make raw"\"" parse to @raw_str "\\\"" and raw"\\\"" parse to @raw_str "\\\\\\\""
4. Disallow " in custom string literals entirely.
5. Do nothing.

FWIW, I think I prefer option 3.

Whatever the decision with respect to parsing, I think we should adjust raw_str to have the invariant that whenever print(r"<x>") is a valid expression for some sequence of characters <x>, the output of that expression is <x>. That's currently not the case in the above example:

julia> print(raw"\\\"")
\\"
julia> print(raw"\"")
"

This discrepancy was noted when raw_str was originally introduced and is documented, but it bothers me.

[JeffBezanson]

I think it should work like normal backslash escapes, but only allowing backslash and quote to be escaped.

[Keno]

Ok, so that would be option 2 in effect?

[stevengj]

I thought that the point of raw strings was that backslashes don't need to be escaped either, e.g. raw"\n" == "\\n"? So the only thing that needs to be escaped is " or maybe a backslash preceding the closing "?

i.e. essentially option 1

[stevengj]

It's pretty important not to require escaping of backslashes in custom string literals, as otherwise regex literals become much messier. (There is also the LaTeXStrings package example.)

[Keno]

Your observations are correct of course. Do you have a preference as to what to do?

[stevengj]

I don't understand option 3. If quotes aren't escaped, how can you tell where the end of the string is?

e.g. how can you distinguish raw"\" between "\\" and a string that hasn't stopped yet?

[stevengj]

I just ran into this in bramtayl/BibTeX.jl#3: it seems to be extremely hard to type \" (a backslash followed by a quote) in a raw literal:

julia> "\\\"" # the desired string, manually escaped
"\\\""

julia> raw"\"" # wrong string, no backslash
"\""

julia> raw"\\"" # does not parse
       ^C

julia> raw"\\\"" # wrong string, too many backslashes
"\\\\\""

[StefanKarpinski]

The semantics of raw string literals are not general in the sense that there are strings which cannot be input that way. The string with contents \" is one of them. If you want full generality, you need real escaping semantics.

[JeffBezanson]

I think we should fix this. If raw strings support any kind of escaping with \, it needs to be possible to escape \ itself.

[JeffBezanson]

Re-reading the thread, I see allowing lots of backslashes in strings is part of the goal of raw strings, so we shouldn't require escaping them.

Option 1 seems good, but then it's impossible to write a string ending in a backslash.

Option 4 is also pretty good: we could have no escaping at all in raw strings, and you can include quote characters using raw""" ... """. Then it's impossible to write a string containing three quotes in a row, but that seems like a pretty mild restriction.

[StefanKarpinski]

The real issue is picking a parsing strategy for custom string literals such that you can both implement normal string behavior and raw string behavior.

[JeffBezanson]

I'm not sure that's possible.