Proposal: Allow carriage return as valid newline sequence

[hukkin]

Related to some of the discussion in #835

I'd like to propose adding CR (carriage return) character that is not followed by an LF character to the list of allowed newline sequences. This is how e.g. CommonMark defines line ending.

The reasoning for the change is:

1. Avoid roundtrip instability when an LF is prefixed by more than one CR characters. EDIT: Not an issue as per .abnf

2. In Python (perhaps other languages?) the standard way of opening files supports something called Universal Newline Support, meaning that LF, CR, and CRLF are all normalised to LF. This normalisation should NOT be used with the current TOML spec because

   • CR characters in strings will be converted to LFs when not allowed to do so. EDIT: Not an issue as per .abnf
   • Files where CRs are used as newline sequence are parsed successfully even though they are invalid TOML (spec v1.0.0).

   As far as I know, all popular Python parsers and their documented API make the mistake of doing this incorrect newline normalisation.

Now for point 2, a valid counter-argument is to "just fix all Python parsers". 😄 However, perhaps other languages do similar normalisation by default or otherwise we agree that a spec change makes more sense here instead.

[hukkin]

If I interpret the .abnf correctly, CRs not followed by an LF character are already prohibited in MLB strings, in which case point 1 in the original post is actually not an issue.

I think this should be clarified in the text spec though. Currently the text spec explicitly mentions that unescaped CRs can be used in MLBs (with no mention of a following LF being required).

[eksortso]

You're right that clarification is worthwhile. But to be clear, there's no such thing as an "escaped" carriage return in TOML. There's a limited set of characters that can be escaped (with a backslash) in basic strings, and carriage returns aren't among them. (Even inside escaped end-of-line whitespace in MLBs, carriage returns are parts of newline sequences.) And other representations of carriage returns like \r and \u000d use the escape backslash but the sequences aren't considered "eacaped."

[hukkin]

But to be clear, there's no such thing as an "escaped" carriage return

Thanks for the correction.

FWIW, I currently think it may be best to reject this proposal in favor of the clarifying PR #838

[emmatyping]

Hm shouldn't TOML use the the Unicode line breaking algorithm since documents are utf-8 encoded?
https://www.unicode.org/reports/tr14/tr14-32.html#Algorithm

[hukkin]

@ethanhs Based on quick glance, it seems that the algorithm is an annex that is not part of the core spec. It also seems to do a slightly different thing than what we want, specifying mandatory and optional break opportunities for text display. E.g. how web browsers wrap text at space characters.

[pradyunsg]

I'd like to propose adding CR (carriage return) character that is not followed by an LF character to the list of allowed newline sequences. This is how e.g. CommonMark defines line ending.

Happy to accept a PR adding this.