Enforce consistent practices for greedy patterns #3265
Description
Motivation
Greedy matching is hard and interacts in non-obvious ways with non-greedy matching. This has caused issues in the past, like the one fixed in #2680.
One best practice for grammars containing both greedy and non-greedy tokens is to ensure that there are no non-greedy tokens before any greedy token. Let's call a grammar of that form a normalized grammar or grammar in normal form.
Normalized grammars are immune to the unintuitive interactions between greedy and non-greedy patterns. Hence, they are immune to a whole range of highlighting bugs. Unfortunately, a lot of our current languages do not follow this best practice.
Description
The goals of this issue are to:
- Find out whether every grammar can feasibly be brought into normal form.
- If yes, add automated tests that enforce normalized grammars.
These goals can be achieved incrementally. The first steps should be to find, fix, and test for specific patterns where grammars only slightly deviate from normal form.