Add memoization control (for now, C only)

[gvanrossum]

Addresses #220 (comment)

Memoization is expensive. In the C generator, we now disable it except for rules with the special marker (memo) after the rule name (and type, if present). By selectively turning on memoization for a handful of rules the parser becomes faster and uses less memory.

Notes:

• Left-recursive rules always use memoization, since the implementation of left-recursion depends on it.
• The marker syntax is extensible, in case we need more per-rule flags in the future.

This also adds instrumentation to the C support code to measure how much each rule uses memoization.

Finally, it fixes two lines in scripts/test_parse_directory.py that were still using pegen/ instead of peg_parser/ as the output directory.

[gvanrossum]

This will require precarious porting to the cpython branch at https://github.com/we-like-parsers/cpython :-(

[lysnikolaou]

It's a very small nit, but I think I'd prefer to not have whitespace between the type and the flags. Looks great otherwise!

[gvanrossum]

I think I'd prefer to not have whitespace between the type and the flags.

Hm, I added that intentionally so the flags stand out a bit more -- it's a bit busy already with the type in square brackets.

I'll just land, so we can find out the consequences for other work soon.