Skip to content

Skip unchanged trimmable typemap assembly emission - #12599

Merged
simonrozsival merged 8 commits into
mainfrom
simonrozsival-incremental-typemap-emission
Sep 3, 2026
Merged

simonrozsival merged 8 commits into
mainfrom
simonrozsival-incremental-typemap-emission

Conversation

@simonrozsival

Copy link
Copy Markdown
Member

Stacked on #12598.

Summary

  • Build the final per-assembly typemap model first and persist a SHA-256 fingerprint for every output.
  • Skip expensive PE emission when the model fingerprint and output DLL are unchanged.
  • Include the generator module identity, runtime version, shared-universe mode, and every emitter-consumed model field in the fingerprint.
  • Fingerprint the root typemap separately from its ordered per-assembly reference list, so it regenerates when assemblies are added or removed but not for unrelated per-assembly content changes.
  • Keep cross-assembly alias peers in the owner assembly model, so changing an alias invalidates the owner even when the owner's source assembly did not change.
  • Preserve the existing default full-detail scanner behavior and byte-identical generated output.

Incremental benchmark

Release scratch harness over TestFixtures.dll, median of 10 batches of 100 runs. Both paths perform assembly scanning and model construction; the incremental path skips unchanged per-assembly and root PE emission.

Path Wall time Allocations
Full emission 424.1 ms 256.7 MiB
Unchanged incremental 249.1 ms 156.8 MiB
Change -41.3% -38.9%

Correctness

  • Standalone Microsoft.Android.Sdk.TrimmableTypeMap.Tests: 779 passed.
  • Added regressions for unchanged-output skipping, byte-identical callback emission, cross-assembly alias owner invalidation, and root reference-list invalidation.
  • Added task coverage that verifies a second run reports skipped _Mono.Android.TypeMap PE emission and preserves the output timestamp.
  • The modified targets file parses as valid XML.

Full-build task/integration validation was unavailable because this worktree has no local Android SDK build (bin/Debug/dotnet/dotnet / bin/Release/dotnet/dotnet). A direct build attempt reached missing bootstrap/local environment prerequisites (including no usable JDK), so no full SDK build was started.

simonrozsival and others added 2 commits August 31, 2026 13:20
Reduce scanner and emitter allocations by caching metadata, avoiding unused MCW method scans, and eliminating intermediate buffers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Persist versioned model fingerprints so incremental builds only emit typemap PE assemblies whose final model changed, while retaining alias-owner and root-reference invalidation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 31, 2026 12:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Review tier: Lite
Findings: 1 Medium severity

New issues introduced by this change (1)
Severity Finding
Medium severity src/​Xamarin.Android.Build.Tasks/​Tasks/​GenerateTrimmableTypeMap.cs — ⚠️ Operational implications — ReadTypeMapFingerprints() treats the fingerprints file as…
What changed in this PR

Adds an incremental-build path to trimmable typemap generation by fingerprinting the typemap content model and skipping PE emission when fingerprints and on-disk outputs are unchanged. This fits into the build pipeline by reducing repeated work in GenerateTrimmableTypeMap while preserving byte-identical outputs when generation does occur.

Changes:

  • Introduces per-assembly and root typemap incremental fingerprints (including generator identity + emitter-consumed model fields) and a callback to decide whether to emit each output.
  • Persists typemap fingerprints to disk and wires the file into the MSBuild task + targets so subsequent builds can skip unchanged PE emission.
  • Adds/extends unit + task tests to validate skipping behavior, byte identity, and invalidation scenarios (alias ownership + assembly-set changes).
File Description
tests/​Microsoft.Android.Sdk.TrimmableTypeMap.Tests/​Generator/​TrimmableTypeMapGeneratorTests.cs Adds generator-level tests covering skip/all-skip, byte identity, alias invalidation, and root regeneration on assembly-set changes.
src/​Xamarin.Android.Build.Tasks/​Tests/​Xamarin.Android.Build.Tests/​Tasks/​GenerateTrimmableTypeMapTests.cs Extends task test to verify second run skips PE emission (and preserves output timestamp) when fingerprints are persisted.
src/​Xamarin.Android.Build.Tasks/​Tasks/​GenerateTrimmableTypeMap.cs Persists and consults typemap fingerprints to skip PE emission; updates assembly writing flow to support “skipped but still output” assemblies.
src/​Xamarin.Android.Build.Tasks/​Microsoft.Android.Sdk/​targets/​Microsoft.Android.Sdk.TypeMap.Trimmable.targets Wires the fingerprints file into the task and ensures it’s tracked for cleaning via @(FileWrites).
src/​Microsoft.Android.Sdk.TrimmableTypeMap/​TrimmableTypeMapGenerator.cs Adds an optional callback for incremental emission decisions; computes per-assembly/root fingerprints before PE emission.
src/​Microsoft.Android.Sdk.TrimmableTypeMap/​Generator/​TypeMapAssemblyGenerator.cs Refactors generation to separate model creation/fingerprinting from PE emission.
src/​Microsoft.Android.Sdk.TrimmableTypeMap/​Generator/​MetadataHelper.cs Implements incremental fingerprint computation for per-assembly and root typemap outputs.

Base automatically changed from simonrozsival-optimize-trimmable-typemap to main August 31, 2026 15:40
Persist versioned model fingerprints so incremental builds only emit typemap PE assemblies whose final model changed, while retaining alias-owner and root-reference invalidation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@jonathanpeppers
jonathanpeppers force-pushed the simonrozsival-incremental-typemap-emission branch from fa94514 to 51e0fdf Compare August 31, 2026 15:40
@simonrozsival

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12599

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Reject

Found 1 error in the incremental-clean integration. The model-first fingerprinting and selective PE emission are otherwise well-structured, with good focused coverage for fingerprint stability, alias ownership, root invalidation, and byte-identical output.

CI is still in progress: 42 checks have passed, with the Java.Interop macOS lane running and the aggregate check queued. No failures are currently reported.

Generated by Android PR Reviewer for #12599 · gpt56 · 155.6 AIC · ⌖ 19.9 AIC · ⊞ 25.7K
Comment /review to run again

Keep the fingerprint cache registered during no-op builds and treat unreadable or malformed cache contents as a regeneration signal.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simonrozsival

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12599

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Verdict: Needs changes

I found no high-confidence correctness defects in the implementation. The fingerprints cover the emitter-consumed model, root membership invalidation, generator identity, runtime version, and shared-universe mode; skipped assemblies remain represented in the expected output set. I left one inline testing suggestion for the missing-output recovery path.

CI: Azure DevOps build 1575463 is red in Windows MSBuild lanes 1, 6, and 8 and macOS lanes 2, 3, and 10. The Azure CLI could not initialize its required writable profile in this environment, so I could not attribute those failures from the timeline/logs. These failures should be resolved or confirmed unrelated before merge.

Findings: 0 bugs · 0 warnings · 1 suggestion

Generated by Android PR Reviewer for #12599 · gpt56 · 290.9 AIC · ⌖ 9 AIC · ⊞ 25.7K
Comment /review to run again

simonrozsival and others added 3 commits September 2, 2026 13:42
Resolve custom view rooting conflicts and preserve the default MainActivity source in the incremental emission test.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@simonrozsival simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Sep 2, 2026
@jonathanpeppers

Copy link
Copy Markdown
Member

/review

@github-actions

github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12599

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — The per-assembly and root fingerprints cover the current emitter inputs, skipped outputs remain represented in task/MSBuild outputs, cache failures safely force regeneration, and CI is green.

Findings: 0 errors, 0 warnings, 1 suggestion. The inline suggestion asks for broader fingerprint-invalidation coverage to protect this manually maintained emitter contract as the model evolves.

Generated by Android PR Reviewer for #12599 · gpt56 · 233.2 AIC · ⌖ 8.96 AIC · ⊞ 25.7K
Comment /review to run again

@simonrozsival
simonrozsival merged commit 0b775b0 into main Sep 3, 2026
44 checks passed
@simonrozsival
simonrozsival deleted the simonrozsival-incremental-typemap-emission branch September 3, 2026 13:17
simonrozsival added a commit that referenced this pull request Sep 10, 2026
Dependent PR on top of #12599 (`simonrozsival-incremental-typemap-emission`).

An independent retry of optimization ideas an earlier experimental pass rejected. Every candidate was benchmarked parent-vs-candidate in alternating fresh processes against the real `Microsoft.Android.Ref.36` `Mono.Android.dll` plus fixture workloads, on both the changed and unchanged incremental paths. Three changes cleared the bar; three did not and are **not** in this PR (details below).

Generated output is **byte-identical to the #12599 base** — all typemap DLLs and Java sources hash-compared in shared and per-assembly universe modes on both workloads.

## Retained

### 1. Cache the fixed signatures repeated for every emitted peer

`CreateInstance` and the activation constructor have the same signature on every peer, but each emission rebuilt it through nested SRM signature callbacks. Cache both, and let `PEAssemblyBuilder` accept a pre-encoded `BlobHandle`, retaining the existing encoder and IL/max-stack paths.

### 2. Compute both typemap fingerprints in a single streaming pass

The content fingerprint (which seeds the emitted assembly's deterministic MVID) and the incremental-build fingerprint each walked the entire model into a `BinaryWriter` over a `MemoryStream` and then hashed the buffer. Those two buffered walks were the single largest allocation source in the generator.

Both are now serialised from one walk and streamed straight into SHA-256 through a small reusable buffer (`FingerprintWriter`). Fields shared by the two fingerprints are UTF-8 encoded once and appended to both hashes, so the byte stream each hash sees — and therefore every fingerprint value and generated MVID — is unchanged.

### 3. Return generated assemblies without a second buffer

Emitting an assembly serialised the PE image into a `BlobBuilder` and then copied every byte again into a `MemoryStream`. `GeneratedAssembly.Content` is now a read-only, seekable stream over the chunks the serialiser already produced (`BlobBuilderStream`), typed as `Stream` instead of `MemoryStream`.

That is all the build task needs: it hashes the stream and copies it to disk through `Files.CopyIfStreamChanged`, so atomic last-known-good replacement is unchanged and `GenerateTrimmableTypeMap` needed no edits.

## Measurements

These tables isolate **changes 2 and 3**. The measurement parent is this branch with only change 1 applied — i.e. the signature-encoder caching is present on *both* sides, so it does not contribute to the deltas below and is not separately quantified here.

Real `Microsoft.Android.Ref.36` `Mono.Android.dll` + `Java.Interop.dll` as framework inputs. *fixtures* = the repo test fixtures (8 962 peers, 405 JCWs, 5.3 MB output); *bigapp* = 1 500 synthetic user ACW types over `Activity`/`Fragment`/`View`/`Service`/listener bases (10 320 peers, 1 813 JCWs, 7.2 MB output).

Wall time — 12 alternating fresh-process rounds per cell, 3 timed iterations each (n = 36):

| workload | path | parent median | candidate median | Δ | parent min | candidate min | Δ |
|---|---|---|---|---|---|---|---|
| fixtures | changed | 180.5 ms | 174.5 ms | −3.3 % | 156.2 ms | 136.6 ms | −12.5 % |
| fixtures | unchanged | 108.2 ms | 107.8 ms | −0.4 % | 87.9 ms | 86.0 ms | −2.2 % |
| bigapp | changed | 291.0 ms | 285.8 ms | −1.8 % | 236.6 ms | 228.0 ms | −3.6 % |
| bigapp | unchanged | 172.2 ms | 166.1 ms | −3.6 % | 139.9 ms | 138.0 ms | −1.4 % |

No cell regresses. Total allocated bytes (`GC.GetTotalAllocatedBytes(precise: true)`) and peak committed GC bytes:

| workload | path | universe | alloc parent → cand | Δ | peak parent → cand | Δ |
|---|---|---|---|---|---|---|
| fixtures | changed | shared | 109.7 → 80.6 MB | −26.5 % | 73.1 → 57.6 MB | −21.2 % |
| fixtures | changed | per-assembly | 109.2 → 80.1 MB | −26.6 % | 72.7 → 54.8 MB | −24.6 % |
| fixtures | unchanged | shared | 62.9 → 46.7 MB | −25.8 % | 52.9 → 44.5 MB | −15.9 % |
| fixtures | unchanged | per-assembly | 62.4 → 46.2 MB | −26.0 % | 53.0 → 44.2 MB | −16.6 % |
| bigapp | changed | shared | 182.5 → 140.4 MB | −23.1 % | 86.0 → 66.3 MB | −22.9 % |
| bigapp | changed | per-assembly | 181.5 → 139.4 MB | −23.2 % | 83.1 → 65.3 MB | −21.4 % |
| bigapp | unchanged | shared | 112.6 → 88.6 MB | −21.3 % | 42.9 → 48.8 MB | +13.8 % |
| bigapp | unchanged | per-assembly | 111.5 → 87.6 MB | −21.4 % | 42.6 → 48.2 MB | +13.1 % |

The one non-improving cell is *committed* GC bytes on the bigapp unchanged path. It is a GC budget artifact, not retention: allocations there drop 24 MB, fewer gen0 collections run, and the collector keeps a larger gen0 budget committed. Live managed bytes after a forced full collection are 0.1 MB for both parent and candidate, and wall time on that cell improves 3.6 %.

Allocation traces (`Microsoft-Windows-DotNETRuntime` allocation sampling, bigapp changed path, 3 iterations) — top frames by sampled bytes:

| parent | candidate |
|---|---|
| `MetadataHelper.ComputeIncrementalFingerprint` 49.0 MB | `BlobBuilder..ctor` 11.5 MB *(SRM-internal)* |
| `MetadataHelper.WriteTypeRef` 27.0 MB | `BlobBuilder.AllocateChunk` 5.7 MB *(SRM-internal)* |
| `PEAssemblyBuilder.WritePE` 21.5 MB | `TypeMapAssemblyEmitter.EmitProxyType` 4.5 MB |
| `MetadataHelper.ComputeContentFingerprint` 13.1 MB | `AssemblyIndex.Build` 4.4 MB |
| `MetadataHelper.WriteNativeRegistration` 9.0 MB | `MergeCrossAssemblyAliases` 3.7 MB |

Every frame changes 2 and 3 target has dropped out of the profile entirely.

## Attempted and rejected

Reverted because they did not clear the bar of a reproducible wall-time win or a substantial allocation/peak-memory win at neutral wall time.

**Base-constructor-chain caching.** Cached generic-substitution-independent registered-constructor metadata (and decoded parameter signatures) per declaring type on `AssemblyIndex`, applying generic substitutions at consumption, and hoisted the derived-constructor signature decode out of the base-constructor comparison loop. Consistently ~1 % faster in every statistic across five independent measurement sets, but that is under the run-to-run noise floor, allocations moved only 146.5 → 145.9 MB (−0.4 %), and peak committed regressed slightly because the cache retains arrays. A control build that removed `CollectBaseConstructorChain` outright showed the entire path is worth only ~4.5 % (250.0 → 243.4 ms median), so the ceiling here is small.

**Metadata reading strategy.** Compared `PEStreamOptions.Default`, `PrefetchMetadata`, `PrefetchEntireImage`, `FileOptions.RandomAccess`, and a 1-byte `FileStream` buffer on the 18.5 MB `Mono.Android.dll`. Medians across four alternating rounds were 249.1 / 248.6 / 254.6 / 251.3 / 249.9 ms — all within noise, with identical managed allocation. No real `Mono.Android.dll` win, so nothing changed.

**Further SRM signature/IL callback allocations.** Beyond the caching kept as change 1, I also tried delegate-free `BeginSignature`/`EndSignature` scratch encoding plus a `BlobHandle` overload of the exception-region `EmitBody`, converting the hottest per-method path (`EmitUcoMethod`, both its wrapper and callback signatures) to SRM's `out`-parameter `Parameters` overload. Byte-identical output — but only after reordering the two signature encodings, because interning them in the opposite order changes the blob heap layout. Worth 140.4 → 138.8 MB (−1.1 %) with no wall-time change. The remaining delegate families are spread across many call sites that each carry the same interning-order hazard, so the reward did not justify the fragility.

## Testing

- Standalone `Microsoft.Android.Sdk.TrimmableTypeMap.Tests`: **806 passed, 0 failed** (781 on the #12599 base, +25 new).
- New `FingerprintWriterTests` pin the streaming writer against `BinaryWriter`/`MemoryStream` byte-for-byte, including empty, non-ASCII, and values spanning the scratch and sink buffers, and re-run the previous buffered fingerprint serialisation verbatim over a model built from the real fixtures to assert both fingerprints are unchanged.
- New `BlobBuilderStreamTests` cover multi-chunk reads, partial reads across chunk boundaries, re-reads after rewind (the hash-then-copy pattern the build task uses), all three seek origins, argument validation, write rejection, and that each generated assembly stream still opens as a valid PE image.
- Generated output hash-compared against the #12599 base: 4 DLLs + 405 Java sources (fixtures, shared and per-assembly universes) and 4 DLLs + 1 813 Java sources (bigapp) — all identical.
- The `GenerateTrimmableTypeMap` consumption pattern (`Files.CopyIfStreamChanged` + `Dispose`) was compiled against the new `Stream`-typed property to confirm the task needs no change.

**No local SDK is available in this environment** (`bin/Debug/dotnet/dotnet` and `bin/Release/dotnet/dotnet` are absent, and `make prepare` cannot run here — building `Xamarin.Android.Build.Tasks` fails on the Java.Interop `javac` bootstrap and the missing Android SDK/aapt2 bootstrap tasks). Full-build tests — `Xamarin.Android.Build.Tests` and `Microsoft.Android.Sdk.TrimmableTypeMap.IntegrationTests` — were therefore **not run** and should be covered by CI.
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). trimmable-type-map

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants