Skip to content

Serialize concurrent tool package installations with file locks - #51834

Merged
marcpopMSFT merged 12 commits into
release/11.0.1xxfrom
copilot/fix-concurrent-tool-installation
Oct 19, 2026
Merged

marcpopMSFT merged 12 commits into
release/11.0.1xxfrom
copilot/fix-concurrent-tool-installation

Conversation

Copilot AI commented Nov 20, 2025 •

Copy link
Copy Markdown
Contributor

Concurrent installs, updates, uninstalls, and shared-cache publication for the same .NET tool package could race, causing sharing violations, partial package visibility, or rollback deleting state another operation had already accepted.

Changes

  • Serialize installation of the same package and version with persistent lock files opened using FileShare.None.
  • Scope locks to the package installation root so unrelated stores and package versions can proceed independently.
  • Keep lock files on disk; ownership is represented only by the open handle, so stale files are immediately reusable after a process exits.
  • Acquire shared-cache locks only after a cache miss, then check the cache again after locking, so complete packages remain reusable from read-only package roots.
  • Coordinate package publication with NuGet's canonical target-.nupkg lock so tool installation and ordinary NuGet writers cannot mutate the same package concurrently.
  • Validate the downloaded archive identity against the requested package ID and version before extraction, and always derive extraction and cleanup paths from the requested identity.
  • Publish the whole-package .sha512 and NuGet .nupkg.metadata completion markers atomically after extraction succeeds, including NuGet-compatible content hashes for signed packages.
  • Add a package-wide mutation lock, independent of version, that global install, update, downgrade, and uninstall hold through shim changes and transaction commit or rollback.
  • Read installed state only after acquiring the package-wide lock so uninstall cannot observe an update's transient remove/install gap.
  • Identify the process holding a contended lock through Windows Restart Manager or Linux /proc/locks plus the matching descriptor's /proc/<pid>/fdinfo lock record.
  • Use the generic waiting message on macOS because libproc open-file data and XNU's sticky FWASLOCKED flag do not reliably identify the current BSD flock owner.
  • Treat owner discovery as best-effort: permission, process-lifetime, platform API, and parsing failures retain the generic waiting message without affecting lock correctness.
  • Report prolonged contention, observe caller cancellation during lock polling, and retain a five-minute safety timeout for legitimately long-running tool operations. Command-wide Ctrl+C/SIGTERM token plumbing remains tracked by Plumb CancellationToken through all CLI commands for consistent Ctrl+C/SIGTERM behavior #56332.
  • Exclude the .locks metadata directory from global tool package enumeration.

Fixes #51831

Tactics

Summary

Concurrent dotnet tool install, dotnet tool update, dotnet tool uninstall, dotnet tool exec, and dnx operations could race while downloading, extracting, publishing, replacing, or rolling back the same tool package. This PR adds per-version package locks, interoperates with NuGet's canonical package lock and completion markers, and serializes package-wide global-tool mutations through transaction completion. Contention diagnostics include a verified owner where the platform exposes one and otherwise safely fall back to a generic message. Lock polling is cancellation-aware for callers that provide a token while retaining a timeout as the final bound for a hung operation.

Customer Impact

Users and tools such as Aspire that run tool operations in parallel can hit sharing violations, incomplete package caches, inconsistent package-plus-shim state, or a package disappearing after another process accepted it. The reported Windows failure is tracked by #51831. Avoiding parallel operations or installing the tool once first remains a workaround for affected SDKs.

Regression?

No. This fixes concurrency failures in existing tool operations without changing command syntax or successful single-process behavior.

Testing

  • Full repository build with build.cmd.
  • ToolPackageDownloaderTests: 61 total, 59 passed, 2 Windows-inapplicable tests skipped.
  • Global install/update/uninstall command tests: 65 passed.
  • Added deterministic coverage for canonical NuGet-lock interoperability, atomic hash and metadata publication, archive identity mismatch rejection, rollback visibility, install/uninstall serialization, transient uninstall reads, stale lock reuse, owner diagnostics, read-only cache reuse, lock timeouts, and cancellation during lock contention.

Risk

Medium. The change touches the core tool package download and global mutation paths and introduces cross-process coordination. Risk is mitigated by preserving read-only cache fast paths, using NuGet's canonical lock and marker formats, keeping owner discovery best-effort, supporting cancellation during lock polling, timing out prolonged contention, and covering publication and transaction boundaries with focused concurrent tests.

Copilot AI changed the title [WIP] Fix concurrent tool installation issue with named mutex Fix concurrent tool installation race conditions with named mutex Nov 20, 2025
Copilot AI requested a review from baronfel November 20, 2025 21:17
Copilot AI requested a review from baronfel November 20, 2025 21:51
@baronfel
baronfel marked this pull request as ready for review November 20, 2025 23:47
Copilot AI review requested due to automatic review settings November 20, 2025 23:47

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@github-actions github-actions Bot added the stale label Apr 23, 2026
@github-actions github-actions Bot closed this Apr 30, 2026
@baronfel baronfel reopened this Aug 26, 2026
@baronfel
baronfel requested a review from a team as a code owner August 26, 2026 21:09
@azure-pipelines

This comment was marked as outdated.

@baronfel
baronfel changed the base branch from main to release/11.0.1xx August 26, 2026 21:12
@baronfel
baronfel force-pushed the copilot/fix-concurrent-tool-installation branch from e779d23 to 137d0f7 Compare August 26, 2026 21:12
@baronfel
baronfel requested review from a team as code owners August 26, 2026 21:12
@github-actions

This comment has been minimized.

@baronfel baronfel changed the title Fix concurrent tool installation race conditions with named mutex Serialize concurrent .NET tool operations with file locks Aug 27, 2026
@github-actions

This comment was marked as outdated.

Add succinct XML documentation describing why each lock and owner-discovery method exists, with platform and lifetime details in remarks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@baronfel

baronfel commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Ok @marcpopMSFT I'm happy with this PR now and its ready for review.

@marcpopMSFT marcpopMSFT left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Posted three review findings: incomplete coordination of cached tool reads, a read-only package-cache regression, and potentially incorrect lock-owner attribution.

Note

These comments were generated by AI. Findings were verified through source inspection; the Windows locking primitive was checked locally, but the reported scenarios were not reproduced end-to-end. No source files were changed.

baronfel and others added 3 commits October 13, 2026 16:49
Delay the NuGet completion marker until extraction succeeds and acquire shared-cache install locks only after a cache miss. This preserves read-only reuse of complete package caches while preventing readers from observing partial extraction.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Require Linux fdinfo evidence that the target descriptor owns the exclusive flock, and fall back to the generic contention message on macOS where open-file APIs cannot prove lock ownership.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@baronfel
baronfel requested a review from marcpopMSFT October 13, 2026 22:52
The XNU FWASLOCKED flag is sticky and produced false owner attribution when a process held an unrelated lock while merely opening the target. Keep the generic waiting message because macOS does not expose a reliable current BSD flock owner query.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@baronfel

Copy link
Copy Markdown
Member

/tactics

@marcpopMSFT marcpopMSFT left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Two additional findings from the latest review: non-atomic completion-marker publication and deletion of published shared-cache content during later transaction rollback.

Note

These comments were generated by AI. The Windows marker-write race was reproduced in isolation; the rollback issue was verified through source inspection. No repository files were changed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
Coordinate tool publication with NuGet's canonical package lock, validate downloaded identities, publish canonical completion metadata, and serialize package-wide install, update, and uninstall transactions through rollback.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@baronfel
baronfel requested a review from marcpopMSFT October 15, 2026 17:53
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 815bde8e-0ac4-435d-8ca5-b0d0159ab079
@marcpopMSFT
marcpopMSFT merged commit a77934a into release/11.0.1xx Oct 19, 2026
22 checks passed
@marcpopMSFT
marcpopMSFT deleted the copilot/fix-concurrent-tool-installation branch October 19, 2026 15:37
@rbhanda rbhanda added this to the 11.0.0 milestone Oct 19, 2026
@baronfel

Copy link
Copy Markdown
Member

/backport to release/10.0.4xx

(I think this will not succeed, but worth trying)

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/10.0.4xx (link to workflow run)

@github-actions

Copy link
Copy Markdown
Contributor

@baronfel backporting to release/10.0.4xx failed, the patch most likely resulted in conflicts. Please backport manually!

git am output
$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: Serialize concurrent tool package installation
Using index info to reconstruct a base tree...
M	src/Cli/dotnet/CliStrings.resx
M	src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs
M	src/Cli/dotnet/xlf/CliStrings.cs.xlf
M	src/Cli/dotnet/xlf/CliStrings.de.xlf
M	src/Cli/dotnet/xlf/CliStrings.es.xlf
M	src/Cli/dotnet/xlf/CliStrings.fr.xlf
M	src/Cli/dotnet/xlf/CliStrings.it.xlf
M	src/Cli/dotnet/xlf/CliStrings.ja.xlf
M	src/Cli/dotnet/xlf/CliStrings.ko.xlf
M	src/Cli/dotnet/xlf/CliStrings.pl.xlf
M	src/Cli/dotnet/xlf/CliStrings.pt-BR.xlf
M	src/Cli/dotnet/xlf/CliStrings.ru.xlf
M	src/Cli/dotnet/xlf/CliStrings.tr.xlf
M	src/Cli/dotnet/xlf/CliStrings.zh-Hans.xlf
M	src/Cli/dotnet/xlf/CliStrings.zh-Hant.xlf
Falling back to patching base and 3-way merge...
Auto-merging src/Cli/dotnet/CliStrings.resx
Auto-merging src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs
CONFLICT (content): Merge conflict in src/Cli/dotnet/ToolPackage/ToolPackageDownloaderBase.cs
Auto-merging src/Cli/dotnet/xlf/CliStrings.cs.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.de.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.es.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.fr.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.it.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.ja.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.ko.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.pl.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.pt-BR.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.ru.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.tr.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.zh-Hans.xlf
Auto-merging src/Cli/dotnet/xlf/CliStrings.zh-Hant.xlf
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Serialize concurrent tool package installation
Error: The process '/usr/bin/git' failed with exit code 128

Link to workflow output

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent Tool Installation can fail

6 participants