Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions src/Microsoft.Android.Sdk.TrimmableTypeMap/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Trimmable typemap build pipeline

This document describes how the **trimmable** typemap implementation
(`_AndroidTypeMapImplementation=trimmable`) is produced during an Android app
build, and how the MSBuild targets are kept incremental. It is aimed at
contributors working on the targets in
`src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.TypeMap.Trimmable*.targets`
and the `GenerateTrimmableTypeMap` MSBuild task.

## Background

The legacy typemap implementations (`llvm-ir`, `managed`) embed the
managed ↔ Java type mapping into native binaries. The **trimmable**
implementation instead generates a set of small managed *TypeMap assemblies*
(one per input assembly, plus a `_Microsoft.Android.TypeMaps` root) and the Java
Callable Wrapper (JCW) `*.java` sources from the same scan. This keeps the
mapping trimmer-friendly: unused entries are removed by the IL linker.

The work happens in the `GenerateTrimmableTypeMap` task, invoked from the
`_GenerateTrimmableTypeMap` target. The runtime is selected by `_AndroidRuntime`
(`CoreCLR` or `NativeAOT`); the runtime-specific imports
(`*.Trimmable.CoreCLR.targets`, `*.Trimmable.NativeAOT.targets`) extend the
shared pipeline.

## Target pipeline (CoreCLR, non-trimmed Debug build)

```
CoreCompile
└─► _GenerateTrimmableTypeMap (AfterTargets="CoreCompile")
• scans @(ReferencePath) + framework/SDK assemblies + the app .dll
• writes typemap/_*.TypeMap.dll + _Microsoft.Android.TypeMaps.dll
• writes typemap/java/**/*.java (JCWs) and acw-map.txt
• writes the merged AndroidManifest.xml
• touches typemap/_GenerateTrimmableTypeMap.stamp
...
_ReadGeneratedTrimmableTypeMapAssemblies (reads typemap-assemblies.txt)
_PrepareTrimmableNativeConfigAssemblies (feeds _GeneratePackageManagerJava)
_PrepareTrimmableTypeMapAssemblies (feeds packaging / assembly store)
_CollectTrimmableTypeMapJavaFiles (globs the JCW *.java)
_GenerateJavaStubs (copies JCWs into android/src, manifest, acw-map)
└─► _CompileJava ─► _CompileToDalvik ─► packaging
```

`_GenerateJavaStubs` **overrides** the legacy target of the same name from
`BuildOrder.targets`; in the trimmable path the JCWs already exist, so this
target only copies them into `$(IntermediateOutputPath)android/src` and wires up
the manifest, `acw-map.txt`, and native config.

For `CoreCLR` + `PublishTrimmed=true`, a second pass
(`_GeneratePostTrimTrimmableTypeMapJavaSources`, in the CoreCLR targets)
regenerates the JCWs from the **linked** assemblies into a `linked-java`
directory, which then becomes the source for `_GenerateJavaStubs`.

## Incrementality design

The pipeline follows the repository's
[MSBuild best practices](../../Documentation/guides/MSBuildBestPractices.md):
every expensive target declares `Inputs`/`Outputs`, re-emits its dynamic
`FileWrites`, and uses stamp files where a real output cannot serve as a
reliable timestamp sentinel.

### 1. A stamp file is the generator's incremental sentinel

`_GenerateTrimmableTypeMap` declares:

```xml
Inputs="@(ReferencePath);@(PrivateSdkAssemblies);@(FrameworkAssemblies);$(IntermediateOutputPath)$(TargetFileName);$(_AndroidManifestAbs);$(_AndroidBuildPropertiesCache)"
Outputs="$(_TypeMapOutputDirectory)$(_TypeMapAssemblyName).dll;$(_TypeMapAssembliesListFile);$(_TrimmableTypeMapOutputStamp)"
```

The generated TypeMap DLLs are written with `Files.CopyIfStreamChanged`, so an
assembly whose **content** is unchanged keeps its old timestamp. If those DLLs
were the only `Outputs`, MSBuild would consider the target perpetually
out-of-date (its inputs are always newer than the untouched outputs) and re-run
it on every build. To avoid this, the target unconditionally `Touch`es a
dedicated stamp:

```xml
<Touch Files="@(_GeneratedTypeMapAssemblies);$(_TypeMapAssembliesListFile);$(_TrimmableTypeMapOutputStamp)" AlwaysCreate="true" />
```

so the stamp is always newer than the inputs after a run, and the target is
correctly **skipped** when none of the inputs changed.

> All assemblies that can contribute managed&nbsp;↔&nbsp;Java mappings must be
> inputs — including `@(PrivateSdkAssemblies)` and `@(FrameworkAssemblies)` —
> otherwise a change in one of them would not trigger regeneration.

### 2. `_GenerateJavaStubs` keys off the stamp

```xml
Inputs="$(_TrimmableTypeMapOutputStamp);@(_EnvironmentFiles)"
Outputs="$(_AndroidStampDirectory)_GenerateJavaStubs.stamp"
```

The stamp captures "the generator ran because its inputs changed" and is left
stable when the generator is skipped, so the JCW copy into `android/src` only
re-runs when something relevant actually changed. The copy uses
`SkipUnchangedFiles="true"` so unchanged JCWs do not churn downstream Java
compilation. For `CoreCLR` + `PublishTrimmed`, the JCWs are sourced from the
`linked-java` directory produced by `_GeneratePostTrimTrimmableTypeMapJavaSources`,
which is itself incremental; the stamp remains the sentinel so a no-op build
still skips `_GenerateJavaStubs`.

### 3. Stale generated Java sources are pruned

When a managed type is removed, its JCW must not linger. The task's
`DeleteStaleJavaSources` enumerates the JCW output directory and deletes any
`*.java` the current pass did not produce, returning them as `DeletedJavaFiles`
(with `RelativePath` metadata). The target mirrors each deletion into the
`android/src` copy and, if anything was deleted, deletes
`$(_AndroidCompileJavaStampFile)` so `_CompileJava` re-runs and drops the stale
`.class` outputs:

```xml
<Delete Files="@(_DeletedCopiedJavaFiles)" />
<Delete Files="$(_AndroidCompileJavaStampFile)" Condition=" '@(_DeletedCopiedJavaFiles->Count())' != '0' " />
```

### 4. Dynamic `FileWrites` are re-emitted on no-op builds

The set of generated assemblies and JCWs is data-dependent, so a build that
*skips* `_GenerateTrimmableTypeMap` never executes the `ItemGroup` that registers
those files in `@(FileWrites)`. `_RecordTrimmableTypeMapFileWrites` re-reads the
generated outputs from `typemap-assemblies.txt` (and globs the JCWs) and
re-emits them — plus the stamp — into `@(FileWrites)` *before* MSBuild's
`IncrementalClean`, so the outputs are not seen as orphaned and deleted between
incremental builds.

### 5. The generator does not run in design-time builds, and runs once

`_GenerateTrimmableTypeMap` is gated on `'$(DesignTimeBuild)' != 'true'`: in a
design-time build, project references may resolve to target paths that are not
produced when `SkipCompilerExecution=true`, and the generator output is not
needed to provide IDE information. Combined with the
`'$(_OuterIntermediateOutputPath)' == ''` guard (which skips inner per-RID
builds), the generator runs exactly once per outer build.

## Files

| File | Role |
| ---- | ---- |
| `Microsoft.Android.Sdk.TypeMap.Trimmable.targets` | Shared pipeline: generation, Java stubs, packaging hookup, incremental `FileWrites`. |
| `Microsoft.Android.Sdk.TypeMap.Trimmable.CoreCLR.targets` | CoreCLR specifics, incl. the post-trim `linked-java` regeneration. |
| `Microsoft.Android.Sdk.TypeMap.Trimmable.NativeAOT.targets` | NativeAOT specifics (ILC inputs, proguard). |
| `Tasks/GenerateTrimmableTypeMap.cs` | The MSBuild task front-end for the generator. |
| `Microsoft.Android.Sdk.TrimmableTypeMap/**` | The generator/scanner library invoked by the task. |
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@
<_PostTrimTrimmableTypeMapJavaStamp>$(_PostTrimTypeMapJavaBaseOutputDir)stamp/_GeneratePostTrimTrimmableTypeMapJavaSources.stamp</_PostTrimTrimmableTypeMapJavaStamp>
<_TrimmableJavaSourceStamp Condition=" '$(_TrimmableJavaSourceStamp)' == '' and '$(_AndroidRuntime)' == 'CoreCLR' and '$(PublishTrimmed)' == 'true' ">$(_PostTrimTrimmableTypeMapJavaStamp)</_TrimmableJavaSourceStamp>
<_TrimmableJavaSourceStamp Condition=" '$(_TrimmableJavaSourceStamp)' == '' ">$(_TypeMapOutputDirectory)$(_TypeMapAssemblyName).dll</_TrimmableJavaSourceStamp>
<!-- Stamp file touched on every _GenerateTrimmableTypeMap run. It is a stable
incremental sentinel: the generated TypeMap DLLs are only rewritten when their
content changes (CopyIfStreamChanged), so their timestamps cannot be used as
Outputs without making the target run on every build. The stamp is always
touched, so the target is correctly skipped when none of its Inputs changed. -->
<_TrimmableTypeMapOutputStamp>$(_TypeMapOutputDirectory)_GenerateTrimmableTypeMap.stamp</_TrimmableTypeMapOutputStamp>
Comment thread
simonrozsival marked this conversation as resolved.
Resolved
<!-- Max array rank for __ArrayMapRank{N} sentinel emission. Defaults to 3 when
dynamic code is unavailable, so array creation uses the typemap path;
defaults to 0 otherwise, where dynamic code can use Array.CreateInstance directly. -->
Expand Down Expand Up @@ -78,28 +84,36 @@
Generate TypeMap assemblies and JCW files.
AfterTargets="CoreCompile" so it runs after compilation.
Uses @(ReferencePath) as the primary input (available after compilation).
Skipped in design-time builds because project references may resolve to target
paths that are not produced when SkipCompilerExecution=true, and because the
generator output is not needed to provide IDE design-time information.
Skipped in inner per-RID builds (_OuterIntermediateOutputPath is set) because
those builds lack the manifest template and full assembly set needed for correct
deferred-registration propagation.
-->
<Target Name="_GenerateTrimmableTypeMap"
Condition=" '$(_AndroidTypeMapImplementation)' == 'trimmable' and '@(ReferencePath->Count())' != '0' and '$(_OuterIntermediateOutputPath)' == '' "
Condition=" '$(_AndroidTypeMapImplementation)' == 'trimmable' and '$(DesignTimeBuild)' != 'true' and '@(ReferencePath->Count())' != '0' and '$(_OuterIntermediateOutputPath)' == '' "
AfterTargets="CoreCompile"
Inputs="@(ReferencePath);$(IntermediateOutputPath)$(TargetFileName);$(_AndroidManifestAbs);$(_AndroidBuildPropertiesCache)"
Outputs="$(_TypeMapOutputDirectory)$(_TypeMapAssemblyName).dll;$(_TypeMapAssembliesListFile)">
Inputs="@(ReferencePath);@(PrivateSdkAssemblies);@(FrameworkAssemblies);$(IntermediateOutputPath)$(TargetFileName);$(_AndroidManifestAbs);$(_AndroidBuildPropertiesCache)"
Outputs="$(_TypeMapOutputDirectory)$(_TypeMapAssemblyName).dll;$(_TypeMapAssembliesListFile);$(_TrimmableTypeMapOutputStamp)">

<ItemGroup>
<_TypeMapInputAssemblies Include="@(ReferencePath)" />
<_TypeMapInputAssemblies Include="@(ResolvedAssemblies)" />
<_TypeMapInputAssemblies Include="@(ResolvedFrameworkAssemblies)" />
<_TypeMapInputAssemblies Include="@(PrivateSdkAssemblies)" />
<_TypeMapInputAssemblies Include="@(FrameworkAssemblies)" />
<_TypeMapFrameworkAssemblies Include="@(ResolvedFrameworkAssemblies)" />
<_TypeMapFrameworkAssemblies Include="@(PrivateSdkAssemblies)" />
<_TypeMapFrameworkAssemblies Include="@(FrameworkAssemblies)" />
<_TypeMapInputAssemblies Include="$(IntermediateOutputPath)$(TargetFileName)"
Condition="Exists('$(IntermediateOutputPath)$(TargetFileName)')" />
</ItemGroup>

<GenerateTrimmableTypeMap
ResolvedAssemblies="@(_TypeMapInputAssemblies)"
ResolvedFrameworkAssemblies="@(ResolvedFrameworkAssemblies)"
FrameworkAssemblyNames="@(ResolvedFrameworkAssemblies->'%(Filename)')"
ResolvedFrameworkAssemblies="@(_TypeMapFrameworkAssemblies)"
FrameworkAssemblyNames="@(_TypeMapFrameworkAssemblies->'%(Filename)')"
OutputDirectory="$(_TypeMapOutputDirectory)"
JavaSourceOutputDirectory="$(_TypeMapJavaOutputDirectory)"
TargetFrameworkVersion="$(TargetFrameworkVersion)"
Expand All @@ -125,13 +139,32 @@
ApplicationRegistrationOutputFile="$(IntermediateOutputPath)android/src/net/dot/android/ApplicationRegistration.java">
<Output TaskParameter="GeneratedAssemblies" ItemName="_GeneratedTypeMapAssemblies" />
<Output TaskParameter="GeneratedJavaFiles" ItemName="_GeneratedJavaFiles" />
<Output TaskParameter="DeletedJavaFiles" ItemName="_DeletedJavaFiles" />
<Output TaskParameter="AdditionalProviderSources" ItemName="_AdditionalProviderSources" />
</GenerateTrimmableTypeMap>

<!-- Mirror any Java sources the generator deleted into the android/src copies so a
removed type does not leave a stale .java (and stale .class) behind. Busting the
Java compile stamp forces _CompileJava to drop the corresponding class output. -->
<ItemGroup>
<_DeletedCopiedJavaFiles Remove="@(_DeletedCopiedJavaFiles)" />
<_DeletedCopiedJavaFiles Include="@(_DeletedJavaFiles->'$(IntermediateOutputPath)android/src/%(RelativePath)')" />
</ItemGroup>
<Delete Files="@(_DeletedCopiedJavaFiles)" />
<Delete Files="$(_AndroidCompileJavaStampFile)" Condition=" '@(_DeletedCopiedJavaFiles->Count())' != '0' " />

<!-- Touch the generated assemblies, the list file, and the stamp so the target's
Outputs are always newer than its Inputs even when CopyIfStreamChanged left an
unchanged DLL untouched. This is what makes the target skip on no-op builds. -->
<Touch Files="@(_GeneratedTypeMapAssemblies);$(_TypeMapAssembliesListFile);$(_TrimmableTypeMapOutputStamp)" AlwaysCreate="true" />

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to only touch the $(_TrimmableTypeMapOutputStamp) file and make it the only Output?

Or are @(_GeneratedTypeMapAssemblies);$(_TypeMapAssembliesListFile) Inputs to another target? -- that would be the only reason to touch so many files.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call — done in 8a70372. _GenerateTrimmableTypeMap now declares only $(_TrimmableTypeMapOutputStamp) as its Outputs and touches only that stamp. The generated TypeMap DLLs and typemap-assemblies.txt are written via CopyIfStreamChanged (stable timestamps when unchanged) and are not consumed as timestamp Inputs by any other target, so they no longer need to be touched.


<ItemGroup>
<FileWrites Remove="@(_DeletedJavaFiles)" />
<FileWrites Remove="@(_DeletedCopiedJavaFiles)" />
<FileWrites Include="@(_GeneratedTypeMapAssemblies)" />
<FileWrites Include="@(_GeneratedJavaFiles)" />
<FileWrites Include="$(_TypeMapAssembliesListFile)" />
<FileWrites Include="$(_TrimmableTypeMapOutputStamp)" />
<FileWrites Include="$(_TypeMapBaseOutputDir)AndroidManifest.xml" />
<FileWrites Include="$(IntermediateOutputPath)acw-map.txt" />
<FileWrites Include="$(IntermediateOutputPath)android/src/net/dot/android/ApplicationRegistration.java" />
Expand Down Expand Up @@ -184,6 +217,7 @@
<FileWrites Include="@(_TypeMapJavaFilesForFileWrites)" />
<FileWrites Include="@(_TypeMapJavaFilesForFileWrites->'$(IntermediateOutputPath)android/src/%(RecursiveDir)%(Filename)%(Extension)')" />
<FileWrites Include="$(_TypeMapAssembliesListFile)" />
<FileWrites Include="$(_TrimmableTypeMapOutputStamp)" Condition="Exists('$(_TrimmableTypeMapOutputStamp)')" />
<FileWrites Include="$(_TypeMapBaseOutputDir)AndroidManifest.xml" Condition="Exists('$(_TypeMapBaseOutputDir)AndroidManifest.xml')" />
<FileWrites Include="$(IntermediateOutputPath)AndroidManifest.xml" Condition="Exists('$(IntermediateOutputPath)AndroidManifest.xml')" />
<FileWrites Include="$(IntermediateOutputPath)acw-map.txt" Condition="Exists('$(IntermediateOutputPath)acw-map.txt')" />
Expand Down Expand Up @@ -249,21 +283,22 @@
so this target only handles JCW file copying, manifest, assembly store setup, and native config.
We keep the name _GenerateJavaStubs because BuildOrder.targets references it.

Inputs uses the TypeMap DLL as a focused sentinel — _GenerateTrimmableTypeMap regenerates
the DLL whenever any of its own inputs (assemblies, manifest, etc.) change, so the DLL
timestamp is a reliable proxy for "something changed that requires re-copying".
Inputs uses the TypeMap output stamp as a focused sentinel — _GenerateTrimmableTypeMap
touches it whenever any of its own inputs (assemblies, manifest, etc.) change, and is
skipped (leaving the stamp stable) on no-op builds. The Copy below then updates
android/src only for Java sources whose content changed.
We keep _GetGenerateJavaStubsInputs in DependsOnTargets so that downstream targets
(_GetGeneratePackageManagerJavaInputs) can still read @(_GenerateJavaStubsInputs).
-->
<Target Name="_GenerateJavaStubs"
DependsOnTargets="_SetLatestTargetFrameworkVersion;_PrepareAssemblies;_GetGenerateJavaStubsInputs;_DefineBuildTargetAbis;_PrepareNativeAssemblySources"
Inputs="$(_TypeMapOutputDirectory)$(_TypeMapAssemblyName).dll;@(_EnvironmentFiles)"
Inputs="$(_TrimmableTypeMapOutputStamp);@(_EnvironmentFiles)"
Outputs="$(_AndroidStampDirectory)_GenerateJavaStubs.stamp">
Comment thread
simonrozsival marked this conversation as resolved.
Resolved

<ItemGroup>
<_TypeMapJavaFiles Include="$(_TypeMapJavaStubsSourceDirectory)/**/*.java" />
</ItemGroup>
<Copy SourceFiles="@(_TypeMapJavaFiles)" DestinationFolder="$(IntermediateOutputPath)android/src/%(RecursiveDir)" />
<Copy SourceFiles="@(_TypeMapJavaFiles)" DestinationFolder="$(IntermediateOutputPath)android/src/%(RecursiveDir)" SkipUnchangedFiles="true" />

<ItemGroup>
<FileWrites Include="@(_TypeMapJavaFiles->'$(IntermediateOutputPath)android/src/%(RecursiveDir)%(Filename)%(Extension)')" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
using Microsoft.Android.Sdk.TrimmableTypeMap;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;
using Xamarin.Android.Tools;

namespace Xamarin.Android.Tasks;

Expand Down Expand Up @@ -101,6 +102,8 @@ public void LogJniAddNativeMethodRegistrationAttributeError (string managedTypeN
[Output]
public ITaskItem [] GeneratedJavaFiles { get; set; } = [];
[Output]
public ITaskItem [] DeletedJavaFiles { get; set; } = [];
[Output]
public string[]? AdditionalProviderSources { get; set; }

public override bool RunTask ()
Expand Down Expand Up @@ -192,6 +195,7 @@ public override bool RunTask ()
GeneratedJavaFiles = JavaSourceInputDirectory.IsNullOrEmpty ()
? WriteJavaSourcesToDisk (result.GeneratedJavaSources)
: CopyJavaSourcesFromInputDirectory (result.GeneratedJavaSources);
DeletedJavaFiles = DeleteStaleJavaSources (GeneratedJavaFiles);

// Write manifest to disk if generated
if (result.Manifest is not null && !MergedAndroidManifestOutput.IsNullOrEmpty ()) {
Expand Down Expand Up @@ -370,6 +374,34 @@ ITaskItem [] WriteJavaSourcesToDisk (IReadOnlyList<GeneratedJavaSource> javaSour
return items.ToArray ();
}

// Removes generated Java sources from a previous build that the current generation pass
// no longer produces (for example when a managed type is removed). Returns the deleted
// files (with a RelativePath metadata) so the targets can mirror the deletion into the
// android/src copies and force a Java recompilation.
ITaskItem [] DeleteStaleJavaSources (IReadOnlyCollection<ITaskItem> generatedJavaFiles)
{
var expectedFiles = new HashSet<string> (
generatedJavaFiles.Select (i => Path.GetFullPath (i.ItemSpec)),
Path.DirectorySeparatorChar == '\\' ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal);
var deleted = new List<ITaskItem> ();

foreach (var path in Directory.EnumerateFiles (JavaSourceOutputDirectory, "*.java", SearchOption.AllDirectories)) {
var fullPath = Path.GetFullPath (path);
if (expectedFiles.Contains (fullPath)) {
continue;
}

File.Delete (fullPath);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 💡 Error handling — The Log.HasLoggedErrors early-out that protects the snapshot/clean branch above is missing from this directory-scan branch, which performs an irreversible File.Delete. If GeneratedJavaFiles were ever incomplete here — e.g. a future caller using CopyJavaSourcesFromInputDirectory (which logs XA4255 and continues past a missing source) without CleanJavaSourceOutputDirectory — a still-valid generated source would be physically deleted while the build is already failing. Today this is unreachable (JavaSourceInputDirectory is only ever set together with CleanJavaSourceOutputDirectory=true, so this branch always pairs with WriteJavaSourcesToDisk, which writes every source and logs nothing), so it's purely latent. Consider hoisting if (Log.HasLoggedErrors) return []; to the top of the method so both branches share the guard before any deletion.

Rule: Assert boundary invariants (Postmortem #44)

Log.LogDebugMessage ($"Deleted stale generated Java source '{fullPath}'.");

var item = new TaskItem (fullPath);
item.SetMetadata ("RelativePath", PathUtil.GetRelativePath (JavaSourceOutputDirectory, fullPath));
deleted.Add (item);
}

return deleted.ToArray ();
}

static Version ParseTargetFrameworkVersion (string tfv)
{
if (tfv.Length > 0 && (tfv [0] == 'v' || tfv [0] == 'V')) {
Expand Down
Loading