Don't allow grammar json array to output unescaped new line in string

[ExtReMLapin]

I had this issue where the generated json was this (with a another grammar file)

{"entities": [

{"type": "Geo-Political Entity", "name": "Paris 2024"},
{"type": "Location", "name": "Paris"}],
"relations": [{"type_from": "G',
eographic Feature", "name_from": "Paris", "type_to": "Event", "name_to": "Paris 2024", "relation_type": "occurs_in"}]}

For the grammar builder I came up with this solution

string ::= "\"" (char | escapeSequence)* "\""
char ::= [^"\\]
escapeSequence ::= "\\\"" | "\\\\" | "\\n" | "\\r" | "\\t" | "\\b" | "\\f" | "\\'"

[ExtReMLapin]

Related to IntrinsicLabsAI/gbnfgen#31

[ExtReMLapin]

Oopsie woopsie, I did a mistake

--grammar "root ::= [^'\\\n]*"

Won't actually BLOCK \ characters and new lines, it will interpret \\ to block and the letter n (lowercase only)

Example :

./main -m ../models/mistral-7b-instruct-v0.2.Q5_K_M.gguf -n 4096 -p 'which month is after october ?(lowercase answer only)' -ngl 444444 --grammar "root ::= [^'\\\n]*"
Will output

The correct lowercase spelled out word for the desired term is: "the letter after o, followed by two letters" or "the eleVенth mOth". So, the word you are likely after is "November". 

And giving a try with :

"root ::= [^'\n\\]*"

Segfaults (llama_sampling_init: failed to parse grammar)

Now, my question is , from original grammar file: [^"\\] what does this thing even go ? Alright, it blocks double quotes but then how is \\ correctly interpreted if "root ::= [^'\n\\]*" fails

[ggerganov]

Hm, not sure - if you find something that works, open a PR. For now I reverted the change

[ExtReMLapin]

Thanks for reverting, i'll do more investigations, it could also be caused by my terminal interpreting the anti slashes

[ExtReMLapin]

Alright, it seems that my command line interpreter was causing the \ + lower case N interpretation, and even with that, it was not fully fixed because I forgot about carriage return.

I'll take some time to see if there are others nasty characters that can cause troubles with json strings