Optimize compilation speed and memory consumption (DeclEngine)

[ironcev]

Description

This PR improves compiler's memory consumption and compilation speed by removing duplicated entries from the DeclEngine. The PR also removes many of the unnecessary cloning of various type declarations.

When compiling o2 order-book contract:

• the memory consumption reduces from 3.85 GB to 1.06 GB,
• the compilation time reduces from ~6.1 to ~4.1 seconds.

Compilation time improvements are also reported by CodeSpeed:

	Benchmark	BASE	HEAD	Efficiency
⚡	compile	5.4 s	4.6 s	+18.74%
⚡	open_all_example_workspace_members	8.6 s	7.4 s	+16.8%

When compiling the o2 order-book contract, the number of declarations on the DeclEngine reduced as following (only biggest three slabs shown):

Slab	Length/Capacity Before	Length/Capacity After
function_slab	464_769 / 524_288	110_517 / 131_072
struct_slab	87_852 / 131_072	84_530 / 131_072
enum_slab	25_892 / 32_768	23_125 / 32_768

Approach

To efficiently develop large Sway applications like o2 we need to reduce compiler memory consumption and compilation time. DeclEngine was holding large number of duplicated declarations, redundantly inserted during different compiler phases, e.g., monomorphization. Implementing SemanticDefinitions RFC will remove all of those duplicates by changing how monomorphization work.

Implementing SemanticDefinitions will require a long(er) roadmap. To get more performant compiler immediately this PR checks if declarations to be inserted are actually changed compared to originals that are already in the DeclEngine.

Note that this practice was already used in some cases, where HasChanges was checked after calling subst, but these cases were very rare.

Side-effects

Assuming how intertwined type checking, type unification, type inference, and monomorphization is, I've expected more side-effect of removing the duplicates, but it turned out there were only two:

• DCA got improved and in two cases (order-bool contract and unused_return_value test) reported two additional warnings that are valid warnings and were not reported before.
• recursion detection got broken. An attempt to analyze the issue and maybe fix it is described in DeclEngine insert-elimination in monomorphize_method breaks recursion detection #7658. As described in that issue, recursion detection must not depend on duplicates being inserted. Essentially, the current implementation relies on an unwanted behavior. Because the probability of having recursive methods is rare, and there is no miscompilation (compiler overflows), this issue can be analyzed and fixed in a separate PR so that we can immediately have the benefits of not-inserting duplicates in the method_application.

Aside from these two changes in the compilation process, the compilation output (bytecode) remained the same.

Next Steps

Additional improvements can be done in subsequent PRs.

• HasChanges enum is still defined in subst_types. It should be moved to some common place. [Code quality]
• Notifying changes to caller is done in three different ways: returning HasChanges, returning bool, or not returning information at all. We want to have HasChanges everywhere as an established pattern. [Code quality]
• HasChanges should be marked as #[must_use] with appropriate hint message. This will help to not forget to check the change and also to easy discover additional places where we, e.g., insert into DeclEngine although there are no changes. [Code quality, Performance]
• As the above table shows, we still have a large number of duplicates in the DeclEngine. A part of the reason is that we still insert same monomorphized types if they are created by monomorphizing the same parent with the same generic arguments but in different places in code. We can explore approaches to remove these duplicates as well. E.g, have hashing/cashing inside of the DeclEngine to avoid duplicates, while avoiding at the same time hashing cost of large decls like TyFunctionDecl? Isn't that what TyFunctionSignature is doing? [Performance]

Checklist

• I have linked to any relevant issues.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation where relevant (API docs, the reference, and the Sway book).
  • If my change requires substantial documentation changes, I have requested support from the DevRel team
• I have added tests that prove my fix is effective or that my feature works.
• I have added (or requested a maintainer to add) the necessary Breaking* or New Feature labels where relevant.
• I have done my best to ensure that my PR adheres to the Fuel Labs Code Review Standards.
• I have requested a review from the relevant team or maintainers.

[cursor]

PR Summary

High Risk
Wide changes to core type-checking and DeclEngine behavior; documented regression disables recursive-call detection and can cause stack overflow instead of compile errors (#7658).

Overview
Reduces DeclEngine churn by threading HasChanges through monomorphization, type substitution, replace_decls, and constant-expression updates, and only insert/replace when a pass actually mutates a declaration.

get_method_safe_to_unify, trait/impl item lowering, method and function application, trait-map resolution, turbofish bindings, and match scrutinee monomorphization now reuse the original DeclRef when the cloned work is unchanged. monomorphize_with_mod_path (renamed) and TypeCheckContext::monomorphize return HasChanges to drive that.

impl_self_recursive e2e is disabled with a note that this optimization breaks recursion detection (#7658) until fixed. Const-generics e2e coverage adds struct assoc-fn/method tests; snapshot and minor test harness tweaks accompany the change.

^{Reviewed by Cursor Bugbot for commit 1bda649. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codspeed-hq]

Merging this PR will improve performance by 19.86%

⚡ 3 improved benchmarks
✅ 22 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	tokens_for_program	656.5 µs	528.7 µs	+24.19%
⚡	compile	5.4 s	4.6 s	+18.68%
⚡	open_all_example_workspace_members	8.6 s	7.3 s	+16.83%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ironcev/optimize-decl-engine-interactions (1bda649) with master (493e7b7)}

[ironcev]

👍

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1bda649. Configure here.}

[JoshuaBatty]

💪 Nice finds and fixes. Love it. Well done!