Update link to last known work on the lowerer

[o314]

Update link to last known work on the lowerer that was proposed but not merged AFAIK in Julia circa 2019
The branch was deleted and no direct link is available anymore .
This one should be the last and most complete ref about it

[o314]

abstract type A < B end and other subtype comparisons are allowed, but only A <: B makes sense.

This is (quite strictly) recasted by the lowerer at analyze-type-sig https://github.com/JuliaLang/julia/blob/v1.8.1/src/julia-syntax.scm#L1043 by bringing back the powerful and heavy duty tree matcher @ https://github.com/JuliaLang/julia/blob/v1.8.1/src/match.scm

--
EDITED : More precisely , it seems that

• type constraints are expanded by the parser as a comparison parse
• left-hand type spec eg. typespec on self is handled by the lowerer, and the tree matcher
• right-hand type spec eg. typespec on super is handled by the lowerer, and its xxx-def-expr functions eg. the code gen part of the lowerer

[c42f]

Thanks for this. To be honest the old work on lowering may not be super relevant anymore. Some parts (tests?) might be salvaged but new lowering should probably work on the JuliaSyntax trees rather than (in addition to?) just Expr.

[o314]

Thanks for the quick feedback and merge .

but new lowering should probably work on the JuliaSyntax trees

Does this already exists somewhere, maybe nearby those promising land ? Or is it more of a prospective work ?

FWIW I have ported ~ 1K LOC of the lowerer already , more precisely 750 LOC of the lowerer and 250 LOC of the ast
It may be good to attempt a realignment and sharing of this someday

[c42f]

Does this already exists somewhere

I haven't done any work on this yet.

Hopefully we can arrange lowering to preserve precise source location information through it somehow (as in allow it to use SyntaxNode or some some similar tree data structure). But also so that it can take Expr, because macros will need to manipulate and produce Expr for compatibility.

[o314]

IMO,

• while lowering, one will have to accept to be satisfied with some locator precise only upto an union of text range (@ where flattening, tuple assignments etc.)
• Backward compat may be challenging, following the roslyn way.

Keep up the good work !

[c42f]

Thanks!

satisfied with some locator precise only upto an union of text range

Yes. Well lowering does transform constructs in a weird "discontinuous" way, where some construct in the surface level syntax can map to discontinuous sections of lowered code (in ssa form). And that kind of thing only gets worse once you hit the optimizer. There's not really a way around this, we just need to accommodate it I think :-)

Backward compat may be challenging, following the roslyn way

Do you have a link for this?

My "plan" for compatibility regarding AST data structures is to declare that the official API for manipulating them is to do pattern matching (and to provide pattern matching macros for this purpose). This may give some hope to write code which works for both Expr and SyntaxNode we will see...

[o314]

Hi !
What will happen after macro ?

• expr and syntaxnode will coexist - is it even possible ?
• only expr will exist - no more issue, but overhead upto there .
• only syntaxnode will exist

In the last case you will need to regen syntaxnode from expr

# @ https://github.com/JuliaLang/JuliaSyntax.jl/blob/main/src/syntax_tree.jl
mutable struct SyntaxNode
    source::SourceFile              # will be arbitrary if gend @ macro ?
    raw::GreenNode{SyntaxHead}      # will be arbitrary if gend @ macro ?
    position::Int                   # will be arbitrary if gend @ macro ?
    parent::Union{Nothing,SyntaxNode} # will be arbitrary if gend @ macro ?
    is_leaf::Bool                      # will be arbitrary if gend @ macro ?
    val::Any
end

80% of SyntaxNode may be outputted arbitrary at syntax_node_from_expr

--
Expr is very anchored, well quite everywhere :

• special node mapping (QuoteNode, GlobalRef ... )
  • @ https://github.com/JuliaLang/julia/blob/master/base/boot.jl#L399-L440
  • toplevel @ https://github.com/JuliaLang/julia/blob/master/src/toplevel.c#L119
  • https://github.com/JuliaLang/julia/blob/master/src/ast.c
• ccall interop @ https://github.com/JuliaLang/julia/blob/master/base/c.jl
• lowerer interop (eitheir in scheme, julia, c) with only partial match
  • jl_expr_t @ https://github.com/JuliaLang/julia/blob/master/src/julia.h#L667-L671
  • jl_astaggedvalue ? @ https://github.com/JuliaLang/julia/blob/master/src/julia.h
• reflection @ https://github.com/JuliaLang/julia/blob/master/base/expr.jl
• compiler interop @ https://github.com/JuliaLang/julia/blob/master/base/compiler/validation.jl
• interpreter interop @ https://github.com/JuliaLang/julia/blob/master/src/interpreter.c
• multithreading interop

--
Parser is 10% hacky (you point around parenthesis, i spot issues between toplevel and call @ grammar)
Balancing between parser/lowerer ; Scheme/Julia/C is not without hack too.

--
to_expr already exists today @ https://github.com/JuliaLang/JuliaSyntax.jl/blob/main/src/expr.jl and its cyclomatic complexity is very high : 75

• count(:if) = 28
• count(:else) = 10
• count(:?) = 4
• count(:return) = 7
• count(:while) = 2
• count(:for) = 3
• count(:&&) = 15
• count(:||) = 6
• count = 75

https://docs.microsoft.com/en-us/visualstudio/code-quality/code-metrics-cyclomatic-complexity?view=vs-2022#satc

--
All of those added up should lead to a strong consolidation, and IMO there is too much open gates and floating to guarantee 100% waterproof backward compatibility. No interface in julia today !
I am an 100% percent proponent of a julia frontend . The effort and roadmap is great .
May be divide the over-pessimism by two or more . But expect some overtime too once parsing is done .

---

OK for pattern matching.

• no macrotools please (too much bugs) ,
• with a pattern-lambda ? port (barely used, but simon adds some calls to them if am right)
• manually ?

We will see

[c42f]

What will happen after macro

Yet to be decided. Will need to prototype and see what works best.

too much open gates and floating to guarantee 100% waterproof backward compatibility

Nothing is ever guaranteed, but that's what testing is for. We will be carefully testing across the General registry for compatibility.

Also keep in mind that the reference parser has some odd behaviors and corner cases (described in the README) that we want to gently deprecate, if testing across the ecosystem indicates that people do not rely on them. Hyrum's Law applies here but we will do our best.

The Expr conversion code is not neat, it was written very quickly and remains a work in progress. PRs which improve the code are welcome.

Expr is very anchored, well quite everywhere

Yes Expr is very deeply embedded in many places in the runtime. It can only be replaced slowly, starting at the frontend.

for pattern matching

I'm expecting to do a custom pattern matcher, JuliaSyntax.jl needs to be standalone anyway, as it'll be vendored into Base. And I want it to be as efficient as a bunch of hand-written if statements. @BenChung has worked a bit on this though current status is unknown. However I chatted to Jeff Bezanson about this recently and he thinks we need a lower level API. (IIUC to express things that a pattern language can't easily). Nothing is fully decided here.

[BenChung]

I'm expecting to do a custom pattern matcher, JuliaSyntax.jl needs to be standalone anyway, as it'll be vendored into Base. And I want it to be as efficient as a bunch of hand-written if statements. @BenChung has worked a bit on this though current status is unknown. However I chatted to Jeff Bezanson about this recently and he thinks we need a lower level API. (IIUC to express things that a pattern language can't easily). Nothing is fully decided here.

I'm currently working on implementing part of lowering, but it's a limited subset. Right now, Julia's lowering does both a bunch of validity checking (are you trying to name a function 2+2?) as well as code generation in a completely intertwined fashion. I'm trying to pick out that first part, validity checking, to generate a sort of semantic AST that retains most of the sugar but hasn't yet been fully compiled down into the SSAesque form that the rest of lowering would produce. The other major goal of mine is to reduce the amount that expand-forms is used by applying it to a new synthetic Expr, replacing those cases with either a shared function call or simply another function.

I'm using MLStyle at the moment for pattern matching with a few custom patterns to match SyntaxNodes in terms of architecture. I have some prior work on a much more sophisticated pattern matching mechanism, but this has kind of been put on hold for reasons of expediency.

[o314]

Hard work outside takes me apart from there a certain amount of time o:
Let's say I am ok with your points 2/3 of the total time
Anyway this work is very useful and am too 100% grateful and supportive this is currently done,
with your personal and team-based diligence
Thanks for taking time to answer seriously to those questions !

[BenChung]

If anyone's interested I've posted what I've been working on here as SemanticAST. The idea is to take an Expr and turn it into a regularized AST that is of consistent and uniform structure so that consumers don't have to care about Julia's sugar while still retaining most of the surface level information.