Macro expansion and lowering

[c42f]

As requested in #312 (comment), here's a concrete description of roughly what needs doing (in reference to files over at JuliaLang/julia) to tackle the next parts of the compiler frontend.

• Prototype rewriting macro expansion in src/ast.c and src/macroexpand.scm. (However not the parts which deal with resolving GlobalRefs - those are badly placed in the compilation pipeline and should not be copied). I'm a good way through this on the caf/macro-expansion branch.
• Try out some experiments with new macro expansion. It should
  • Preserve source location information provided by the parser
  • Allow user-written macros to report errors in expressions using that precise source location information. Kinda like this https://twitter.com/c42f_/status/1678521992507543553
  • Solve Macro hygiene is hard to use correctly in nested macro expansion julia#37691
  • Finally implement a version of "automatic hygiene" as in RFC: Improvement to hygienic macros julia#6910, if possible.
  • Have some opt-in based migration path which doesn't depend on getting to Julia 2.0
• Prototype rewriting "some representative parts" of lowering from src/julia-syntax.scm. Investigate what it takes to preserve source location information in error messages.
• Use these experiments to figure out what kind of syntax tree data structures and APIs we need.
  • For data structures, will small modifications to SyntaxNode do the trick? Or do we need something else? What about storing optional semantic annotations using an ECS data structure like in Overseer.jl?
  • For APIs, are we happy with manually branching on head(expression) and relying on the ordering of AST children? Do we want pattern matching? Do we want accessors? See Tree data structures #230
  • How do we use these data structures to "talk to the user about their code"? Even after lowering has sliced and diced and rearranged the expression tree and maybe lowered it to a linearized CodeInfo-like form?
• Figure out how to proceed compatibly, supporting both Expr and SyntaxNode (or similar) through lowering. "A big switch over in Julia 2.0" will never work here. The only way to succeed in this project is to introduce the new stuff gradually in a compatible opt-in way. And maybe eventually have it on by default in Julia 2.0.
• Make some changes in Base to add hooks to replace all or parts of lowering, so a new macro expansion / lowering can be experimented with easily without need to monkey patch Base.

Having tried things and worked out some rough answers to those questions, the long grind of rewriting and testing all of lowering can go ahead.

[gafter]

User-written macros can already report errors in expressions using source location information, but only with as much resolution as is already recorded in the Expr nodes. It's a bit awkward and few packages do it properly.

[vtjnash]

However not the parts which deal with resolving GlobalRefs - those are badly placed in the compilation pipeline and should not be copied

I tried doing this recently on JuliaLang master (it is a very small change to turn this off), but it turns out a lot of packages strongly depend on this from macroexpand output, and so I gave up. Now it resolves them for macroexpand, but does not resolve them for eval.

[c42f]

does not resolve them for eval

Oh excellent! Makes sense! I hadn't read that part of the code yet :-) It's no surprise that packages depend on resolving GlobalRefs. Annoying, but not surprising.

The kinds of changes I'm talking about in this issue are huge changes anyway (like changing the type of expression literals. Whaat?!?!). So it's a good time to experiment with "improving all the things". The key to making a new frontend which people can experiment with right away (without breaking changes) will be to have it all opt-in and have really strong bidirectional interop story.

[vtjnash]

It was implemented here: JuliaLang/julia#44995

Also of possible interest: I wrote a replacement for the internal hygiene representation that implements JuliaLang/julia#6910, but currently I swap the representation back before the macroexpand.scm code (in the interest of testing it). The next step there would likely be to re-write the macroexpand.scm code to use the inverted representation (of JuliaLang/julia#6910). JuliaLang/julia@master...vtjnash:julia:jn/invert-hygiene

[c42f]

I put my current prototype up in #329. The hygiene representation I've got there is quite similar to the hygienic-scope thing which is currently emitted by the runtime code in src/ast.c. But with an added field to indicate which parts need to be automatically escaped in postprocessing after macro expansion. I'm not sure it's all watertight yet 😅

[manuelbb-upb]

Just out of curiosity: Would we have access to comments in parsed expressions then?
I have long been hoping for automatic argument docstrings like in this discussion on discourse.
I had hoped to use DocStringExtensions.interpolation to inspect and extract the documentation from the method definition expression -- without having to write an entirely new macro that wraps or replaces @doc.

The interpolation approach does not modify the expr and I would like to avoid it.
Custom comment parsing, similar to what Literate.jl does, would enable argument docstrings with valid Julia syntax:

struct DocumentedArgs 
  # caches and stuff
end

function DocStringExtensions.interpolation(arg_parser::DocumentedArgs, ex)
  # parse expression `ex` and construct documentation string
end

@doc """
Some comment containing my interpolator: $(DocumentedArgs()).
"""
function myfunc(
  a, #doc This variable would print a doc string.
  #doc The variable `b`, too.
  b,
  c # `c` Does not get a doc string because the comment does not start with "doc"
)
  nothing
end