Cleanup nested attributes (docs, validation, shared_attributes, updating, infrastructure)

[ffreyer]

Description

I noticed that I basically just implemented the backend for nested attributes in #5482. They worked with Attributes in the old @recipe ... do scene ... end syntax, but not in @recipe ... begin ... end or @Block. As a result they didn't integrate with DocumentedAttributes, i.e. didn't have docstrings, validation and mixins. There were also some integration with other features missing, like shared_attributes(), nested attributes/compute graph passthrough or nested attribute updating with merging via plot.nested = Attributes(...).

While working on this I also got sucked into a bunch of tangential things:

First, while we have been using new recipes for around 2 years, old recipes still existed without any deprecation warning and as the thing that ?@recipe documents. So I updated the docstring and added a deprecation warning.

Second, old, new and block recipes each had their own attribute infrastructure. This means a bunch of repetition between new recipes and block recipes and bunch of old vs new paths with old vs new recipes. I reworked all of this to feed into the same @DocumentedAttributes infrastructure so it can then be handled by the same processing. (Well, more or less. Plots initialize their attributes in two parts, first graph init with passthrough and kwargs, then theme resolution, while blocks do it all in one step. To not slow down blocks, these bits are still a little different.)

This somewhat naturally lead to reworking the attribute init code of plots, which was a bit messy before imo and got extra messy after adding nested attributes because I added a bunch of extra branching and recursion.

Third was me getting frustrated with performance after the pr benchmarks starting looking worse. I had been thinking for a while that I should maybe rework DocumentedAttributes to do some of the nesting work ComputeGraph at @DocumentedAttributes resolution time and then save things in a flattened format. I tried and got ~20% faster create benchmarks than the previous commit, so I went ahead with. That meant another rework

At some point I remembered that SpecApi exists. I updated BlockSpec to accept arguments (for block/complex recipes) and updated (and unified) the block and plot infrastructure to work with nested attributes.

---

What changed

Recipes

Old Recipes now warn about being outdated during @recipe evaluation and convert their attribute callback to a new recipe style block, before feeding it to the @DocumentedAttributes infrastructure. I tried to make this fully compatible, but I feel like that's impossible when you're converting a user define callback...

• ?@recipe got update to document the "new" style of recipes @recipe ... begin ... end.
• Validation now validates nested attributes and works with old recipes.
• plot.outer = Attributes(...) should now allow for nested attributes to be updated in bulk.
• shared_attributes() is now compatible with nested attributes + new infrastructure

Block Recipes

No longer changes attribute_names to force validation to accept kwargs. That now happens in block_kwargs.

More or less all of the old attribute processing and docs handling has been removed and replaced by @DocumentedAttributes. This should be internal, though I wouldn't be surprised if the removed functions affect some other packages...

Legend

Legends used to overwrite Legend() for argument processing which required block_defaults to fill out attributes early so that entry groups could be initialized. This blocked the removal of block_defaults so I reworked it to process arguments in initialize_block!().

I ran into a bunch of issues with the newer MeshElement etc. and more generally the legend element attribute infrastructure here which I also (tried to) fix. I think it would be worth it to rework this further with nested attributes, to replace attributes like Legend.markerstrokewidth with Legend.scatter.strokewidth and remove all the renaming it required. But that's probably something for another pr...

@DocumentedAttributes (user facing)

"New" recipes and block recipes now feed into the @DocumentedAttributes directly. As such they have the same user facing interface. (Old recipes still have their own interface/syntax, which gets translated.)
Nested attributes are supported with @attributes blocks, which can also have a docstring:

@DocumentedAttributes begin
    "outer docs"
    outer = @attributes begin
        "inner docs"
        inner = 1
    end
end

Mixins (the ... parts) can now be used in nested scopes. (For compat with old recipes they can also run be variables now, though esc rules may make them impossible to use outside of that context.)

@recipe MyPlot begin
	Scatter = @attributes begin
		documented_attributes(Scatter)...
	end
end

@inherit now supports callbacks (old recipe, block compat), multiple nested inherits (block compat) and inheriting from nested theme entries (block compat). There is also compat for inherit() (blocks) and map() (blocks, old recipes compat earlier) and theme(scene, key) here.

@recipe MyPlot begin
	# not new: (scene_key can be Symbol as well)
	attrib = @inherit scene_key
	attrib = @inherit scene_key 1
	# new: (keys can be Symbols as well)
	attrib = @inherit foo scene_key 1
	attrib = @inherit (parent, child)
	attrib = @inherit (parent, child) 1
	attrib = @inherit foo (parent, child) 1
	attrib = @inherit(scene_key, @inherit other)
	# compat:
	attrib = inherit(scene, :scene_key, 1)
	attrib = theme(scene, :scene_key)
	attrib = map(foo, inherit(scene, :scene_key, 1))
	# indirectly/unofficially supported:
	attrib = Inherit([f], (:scene_key,), [fallback])
end

Another minor note here is that adding no fallback now results in NoFallback() rather than nothing so that nothingis now a valid fallback value.

Types annotations can now be added to attribute names. These are only used by blocks, but can be added in plot recipes too.

@Block MyBlock begin
	@attributes begin
		attrib::Float32 = 1	
	end
end

DocumentedAttributes (internal)

I burned down the house and build a new one. DocumentedAttributes are no longer a wrapper around a Dict. Instead they are a collection of flattened arrays with ComputePipeline.NestedSearchTree encoding the nesting.

struct DocumentedAttributes
	# Nesting information, maps valid paths to attribute indices
    nesting::NestedSearchTree

	# maps Symbol(join(keys, '.')) to attribute indices
    merged_key_to_index::Dict{Symbol, Int}

    # per attribute information
    merged_keys::Vector{Symbol}
    defaults::Vector{Any}
    default_expr::Vector{String}
    leaf_docstring::Vector{Union{Nothing, String}}

    # per nesting key
    nested_docstring::Vector{Union{Nothing, String}}

    # attribute index to type index
    type_index::Vector{Int}

    # type index -> type
    types::Vector{Type}
    # type index -> attribute indices using that type
    type_indices::Vector{Vector{Int}}

    # attribute indices that inherit, to skip type inference
    inherit::Vector{Int}
end

Generation of DocumentedAttributes now includes some state to encode the nesting (basically the recursion stack you'd have with recursive functions) and uses _begin_nesting_layer!(), _end_nesting_layer!(), _add_attribute!() and _merge_attributes!() to build the structure. Parsing still more or less runs through the same functions, but relies much more on MacroTools.

Plot Infrastructure

Plots now step through:

1. Plot(): extract attributes in first argument
2. build_plot(): argument init
3. add_attributes!(): branching for cycle
4. init_graph!(): apply/add documented_attributes. kwargs and passthrough attributes

The latter calls into new DocumentedAttributes infrastructure:

1. prepare_graph_for_attributes!(): copies nesting information into compute graph and adds headless compute nodes for all missing attributes
2. add_from_kwargs!(): adds Inputs or connections to parent compute nodes for each headless node based on kwargs
3. connect_parents!(): connects headless nodes to appropriate nodes in parent graph
4. if plot cycling is defined, initialize cycle inputs for remaining headless nodes via add_prepared_input!()
5. add_remaining_inputs!() adds inputs for the remaining headless nodes. These may contain Inherit(...) as this makes no effort to resolve theming. (There is no scene here, so why bother?)

Steps 2-5 all rely on a build_callback function defined in add_attributes!() to create edge callback functions. With a bit of help from @nospecialize the attribute initialization gets away with (almost?) no specialization/type inference at all. Only kwargs check for Observable vs Computed vs value.

Theming is resolved in add_theme!() (like before) by another function of the same name. Instead of stepping through kwargs, an overwrite theme, documented attributes and then the default theme for each attribute, it builds a vector of resolved defaults with resolve_defaults first and then applies those. So it does one pass through kwargs, filling relevant bits of the vector, then overwrite, etc. This should help a bit with data locality and reduce type inference/checks for nested attributes (if entry isa Attributes; recurse). This also dodges type inference/specialization of defaults. (They are ::Any typed in Input so knowing their type is useless.)

Block Infrastructure

To initialize block attributes _block calls:

1. resolve_defaults to get initial values for attributes
2. add_attributes!() with those defaults, as a user-overwrite path (I added this in the Axis compute graph rework)
3. prepare_graph_for_attributes!() which initializes nodes in groups of the same type here. This should reduce the amount of type inference (once per unique type instead of once per attribute)
4. add_remaining_block_inputs!() which runs through steps 2, 4, 5 and add_theme!() in one go.

SpecApi Infrastructure

Both PlotSpec and BlockSpec now rely on bacth_update*!() functions to collect a list of attributes to update. This should do the same thing as before, but in a nested-attribute compatible way.

Note that this code is not optimized. It may be beneficial to flatten out kwargs in both specs to avoid recursion in batch_update*, and it may be beneficial to save the resolved defaults to avoid lookup_default calls when kwargs get unset.

Documentation

The $ATTRIBUTES thing and help_attributes that got left behind for old recipes in #5389 is now gone, as old recipes integrate with DocumentedAttirbutes.

Pretty much all of the code around attribute docs got reworked to work with the new DocumentedAttributes format. I.e. generating code for the attribute section in ?Plot and the online docs as well as the field_docs for ?Plot.attribute and help(Plot, :attribute).

In ?Plot, groups are now defined based on nested paths, e.g. if :outer is part of a group, every inner attribute is also part of the group. Those nested attributes are shown as outer.(a, inner.(a, b), c). For ungrouped attributes I got a bit lazy, so they just show as their merged keys outer.a, outer.inner.a, outer.inner.b, outer.c.

In online docs generate mostly the same output. Nested Attributes create the same kind of section as unnested ones, but with their merged key as a title, i.e. "#### `outer.inner.a`".

I believe attribute examples should parse and integrate fine if they use the merged key as their header. I.e. "### `outer.inner.a`".

help() now accepts multiple keys to trace a nesting path. If that path is complete (points to a leaf attribute) it displays the same thing as it used to with the merged key as the attribute name. If the path is incomplete (i.e. there is more nesting), it displays (for example):

?Plot.a
  a = @attributes begin ... end

  <a docstring>

  Nested Attribute a contains:

    •  .a = @attributes begin ... end: No docstring available
    •  .b = 5: No docstring available
    •  .c = @inherit sin :q @inherit((:m, :n), 5): a.c nested inherit

  See help(NewRecipe, :inner[, ...]) for detailed documentation on nested attributes.

Removed Functions/structs

• attribute_names, _attribute_docs (both)
• plot_attributes (plots)
• default_attribute_values, attribute_types, attribute_default_expressions, block_defautlts (blocks)
• _attribute_list, repl_docstring, REPL.fielddoc (blocks, replaced by shared docs infrastructure)
• make_attr_dict_expr, extract_attributes!, inherit (blocks, replace by @DocumentedAttributes infrastructure)
• removed help_attributes (old recipes, superseeded by help with DocumentedAttributes integration)
• $ATTRIBUTES DocStringExtension thing (old recipe)
• AttributeMetadata and associated functions (fairly useless for new system)
• LegendOverride wrapper (doesn't seem necessary and internal)
• apply_theme!() (plots, unused)

Cleanup

• removed a refimg file about recipes that hasn't been used in 6 years, added some Makie tests instead
• removed attribute_docs() functions (unused, I think these were vibe coded in improve documentation #5389)
• update old to new recipes: PlotList

TODO

• check that plot.outer = Attributes(a = 1) works and merges correctly
• document DocumentedAttributes
• update Makie-shipped recipes to "new" syntax
• fix online docs
• update changelog
• cleanup typos
• document @inherit with callbacks

[MakieBot]

Benchmark Results

SHA: 0f0ae76ff1c10d5d2a359dc0e104508cddd9a599

Warning

These results are subject to substantial noise because GitHub's CI runs on shared machines that are not ideally suited for benchmarking.

[ffreyer]

For context, validate_node_value(value) causes type inference which costs about 2µs of ~45µs total in @benchmark scatter!(scene, 1:5) setup=(scene = Scene()).
Iirc I added this initially to make sure add_input!() is going down the correct paths. Under normal usage it should never trigger. The calls in Computed are never reached.