Failures parsing General registry

[c42f]

Here's a list of failures when parsing General restricted to cases where

• JuliaSyntax.jl fails to parse a file
• the reference parser accepts the file
• or both parsers succeed, but the resulting Exprs mismatch

Parser failures

• A[x ? y : end] - AssociativeArrays_0.0.3/src/namedaxis.jl, Reduce_1.2.12/src/Reduce.jl, UnROOT_0.8.1/src/UnROOT.jl
• function (f()) where T end - DMRJtensor_0.8.7/lib/Qtensor.jl, DMRJtensor_0.8.7/lib/decompositions.jl, EnglishText_0.6.0/src/numeric.jl, ExprTools_0.1.8/test/function.jl, MemoryMutate_0.0.4/src/MemoryMutate.jl, MultivariatePolynomials_0.4.2/src/differentiation.jl, Singular_0.9.1/src/module/vector.jl, etc why people why??
• .1..0.1 - DomainSets_0.5.9/test/test_setoperations.jl, 0x01..0xcf - Rasters_0.2.0/test/sources/gdal.jl
• import Base.:* - Empirikos_0.4.5/src/ebayes_targets.jl, import A.:⋅ - MeasureTheory_0.13.2/test/runtests.jl, Pathnames_0.1.0/src/Pathnames.jl, Porta_0.1.4/src/geometry/biquaternions.jl
• Base.(:*) - UnitfulLatexify_1.5.2/src/UnitfulLatexify.jl
• function ($M).$J() end - Knet_1.4.9/src/cuarrays/reduction.jl
• @a@b - MathematicalSystems_0.11.11/test/@system.jl
• [a~b]- ModelingToolkitStandardLibrary_0.1.0/test/test_continuous.jl, NeuralPDE_4.4.0/test/NNPDE_tests.jl, etc
• @(A) - NumericalMethodsforEngineers_2.0.0/examples/ch07/WRM/Fig.7.14.jl
• -1.#comment - POMDPModels_0.4.15/src/MountainCar.jl
• A.@a(x).b - WorkerUtilities_1.0.1/test/runtests.jl
• (1+2).@x - julia_1.6.1/test/show.jl

Parses, but Expr is wrong

• @.r - no space after macro name
• @\ _ - symbol macro names
• "[1 2\n]" - hvcat rather than hcat
• A where {X, Y; Z} - parameters not in first args slot
• @a(x; a) - parameters not in first args slot
• try a end should be an error
• function (f::T{g(i)})() end and function (::g(x))() end
• .&(x)
• +0xff
• f(x for a = as, b = bs for c = cs, d = ds)
• x ≂ y ≂ z
• A[x, y; z = 1] - parameter block = vs kw
• "\"\"\"\n\$x\n b\"\"\"" - wrong deindentation
• using A: @. - not lowered to @__dot__
• A{x;y,z} and A{x;y,z=1} - parameter blocks & kw vs =
• function (:*=(f))() end - NiLang uses this

Bizarre forms which parse differently but unclear if it's worth fixing

• function (begin end) nothing end ExprTools_0.1.8/test/function.jl

Expr conversion failures

• JuliaFormatter_0.22.0/test/files/PProf.jl
• MathOptInterfaceMosek_0.5.2/examples/chainsing.jl
• MosekTools_0.11.0/examples/chainsing.jl
• SDFReader_0.3.4/test/0002.jl

Bugs in reference parser

• export @A @B should not parse - TSML_2.7.2/src/TSML.jl
• 17.328679084777833f0 is not correctly rounded?
• 1.0e+ parses as an Int128 literal
• f()isa(x,y) is not the juxtaposed multiplication f()*isa(x,y). It's infix isa syntax instead.
• [i; for i in is] is allowed by the reference parser but the presence of this semicolon is weird and broken.

The following bugs we can deal with when comparing Exprs

• 0x1.8p23f is a Float64 literal, with the trailing f ignored (also 0x1p1f0)
• The macrocall in "@f(a=1) do\nend" is not the same as the call in @f(a=1)
• global (x,y) is the same as global x,y in reference parser
• "\"\"\"\n a\n \n b\"\"\"" parses to "a\n \nb" in reference parser - lines with partial whitespace are handled inconsistently
• (a; b,) - should be a frankentuple not a block. Occurs a single time in QuadraticFormsMGHyp_0.5.1/src/QuadraticFormsMGHyp.jl and appears to be a syntax mistake.
• "\"\"\"\n \$y\n \n \"\"\"\n" - partial whitespace
• "f(g() = x)" - extra block

Intentional failures due to being stricter than reference parser

• import A as (a.b) - julia_1.6.1/test/syntax.jl
• Juxtaposition of strings with more keywords is disallowed. For example "str"for i=is end

[c42f]

Couple of issues which might be bugs in the reference parser:

• global (x,y)
• "@f(a=1) do\nend" parsed differently from @f(a=1)
• "\"\"\"\n a\n \n b\"\"\"" strips the mismatched whitespace weirdly

[oscardssmith]

Related to the Float64 literal bug you found, 0x1p1f0 is also valid (but 0x1p1f1 isn't)

[c42f]

Thanks yes the float literal bug is pretty weird. I tried to understand how this happens a while back. IIRC it was a combination of (a) a bug in the reference lexer allowing extra sufficies and (b) reusing the flisp reader for numeric literals which does allow f suffix to parse Float32 literals. Something like that... it was subtle.

We had a call about the issues mentioned in #134 (comment) and decided they're bugs. It's a little unclear what to do about the string processing issue. But clear that the existing behavior is weird.

[c42f]

@oscardssmith I've found a weird disagreement when parsing the float constant 17.328679084777833f0, thought you might be interested :)

I think the following indicate this literal gets incorrectly rounded in the reference parser:

julia> abs(big"17.328679084777833" - JuliaSyntax.parse(Expr, "17.328679084777833f0")) <
       abs(big"17.328679084777833" - JuliaSyntax.fl_parse("17.328679084777833f0"))
true

julia> x = JuliaSyntax.fl_parse("17.328679084777833f0")
17.328678f0

# looks like the error is roughly 1/2 ulps of a Float64
julia> abs(big"17.328679084777833" - x) - abs(big"17.328679084777833" - nextfloat(x))
1.937500000000000000000000000000000000000000000000000000000000163954795499142396e-15