Skip to content

Fix concurrent BuildCheck callback mutation - #14590

Merged
YuliiaKovalova merged 4 commits into
dotnet:mainfrom
YuliiaKovalova:fix/buildcheck-callback-race
Aug 3, 2026
Merged

YuliiaKovalova merged 4 commits into
dotnet:mainfrom
YuliiaKovalova:fix/buildcheck-callback-race

Conversation

@YuliiaKovalova

@YuliiaKovalova YuliiaKovalova commented Jul 31, 2026 •

Copy link
Copy Markdown
Member

Summary

  • replace mutable callback lists with immutable copy-on-write snapshots published through Volatile
  • keep Has…Actions checks and callback dispatch lock-free and allocation-free
  • deregister checks from all callback registries while preserving cleanup for tasks already in flight
  • skip only the disabled callback instead of suppressing all later checks
  • document snapshot and advisory-presence semantics

Motivation

Graph builds can register or remove BuildCheck callbacks while another thread dispatches the same registry. Enumerating the mutable List caused InvalidOperationException, which the logging pipeline surfaced as InternalLoggerException / MSB4166.

The previous deregistration path also left task, environment-variable, and project-import callbacks active after a check was removed. Task callback removal now keeps task-finished cleanup unconditional so in-flight tracking entries are not orphaned.

Tests

  • concurrent registration uses the next dispatch snapshot
  • concurrent deregistration safely completes the current snapshot
  • deregistration clears all ten callback registries
  • task completion drains tracking after the last task callback is removed
  • a disabled check does not suppress later enabled checks
  • full Microsoft.Build.BuildCheck.UnitTests suite: 235 passed, 0 failed

Follow-up

Snapshot registered callbacks under their synchronization lock before dispatch and synchronize every deregistration path. This prevents graph-build logging from failing when checks are registered or removed concurrently.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3b1601d6-eb67-4320-8084-deba81695daa
Copilot AI review requested due to automatic review settings July 31, 2026 12:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a race in BuildCheck callback dispatch where callback lists could be mutated (register/deregister) while being enumerated, leading to InvalidOperationException and MSB4166 failures in graph builds.

Changes:

  • Snapshot callback registries under the existing per-registry lock before dispatching callbacks to avoid concurrent mutation during enumeration.
  • Synchronize callback deregistration and ensure all callback registries are covered.
  • Add a regression unit test validating that registering a callback during dispatch is deferred to the next dispatch.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/Build/BuildCheck/Infrastructure/BuildCheckCentralContext.cs Locks and snapshots callback lists before dispatch; locks deregistration and covers all registries to prevent concurrent List<T> mutation during enumeration.
src/BuildCheck.UnitTests/BuildCheckCentralContext_Tests.cs Adds regression test for registering an evaluated-properties action during dispatch and ensuring it runs on the next dispatch.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MSBuild Expert Review — PR #14590

# Dimension Verdict
13 Concurrency & Thread Safety 🟡 MODERATE
22 Correctness & Edge Cases 🟠 MAJOR (pre-existing)
4 Test Coverage & Completeness 🟠 MAJOR

✅ 21/24 dimensions clean.


Summary

The PR correctly fixes the core race condition: RunRegisteredActions now takes a snapshot under the list's own lock before iterating, and DeregisterCheck / RegisterAction both hold the same lock during mutation. The approach is sound and idiomatic.

Three findings worth addressing:

  • Concurrency (MODERATE) — HasXxxActions properties read .Count without holding the lock. This is benign today (atomic int read used as a hint), but inconsistent with the newly-added locking discipline. Either add the lock or add a comment that these are single-threaded reads. (Inline comment at line 79)

  • Correctness (MAJOR, pre-existing) — return inside RunRegisteredActions exits the whole dispatch loop rather than skipping the single disabled check. If two checks are registered and the first has IsEnabled = false, the second check is never invoked. Consider changing return → continue. (Inline comment at line 248)

  • Test Coverage (MAJOR) — The new test covers register-during-dispatch but not the original bug scenario (deregister-during-dispatch causing InvalidOperationException). A regression test for the original failure mode is missing. (Inline comment on test file)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Expert Code Review (on open) for #14590 · sonnet46 · 66.3 AIC · ⌖ 5.54 AIC · ⊞ 4.9K

Comments that could not be inline-anchored

src/Build/BuildCheck/Infrastructure/BuildCheckCentralContext.cs:79

[MODERATE] Concurrency: HasXxxActions properties read Count without holding the lock

The PR adds lock(list) guards to all mutations (RegisterAction, DeregisterCheck, snapshot in RunRegisteredActions), but the HasXxxActions convenience properties — used by callers to decide whether to collect data before calling RunXxxActions — read .Count without any synchronization:

internal bool HasEvaluatedPropertiesActions =&gt; _globalCallbacks.EvaluatedPropertiesActions.Coun…

</details>

<details><summary>src/Build/BuildCheck/Infrastructure/BuildCheckCentralContext.cs:248</summary>

**[MAJOR] Correctness: Early `return` exits entire dispatch loop, not just one callback**

Lines 247-250 (unchanged from the original, but the snapshot approach makes this more visible):

```csharp
if (!commonConfig.IsEnabled)
{
    return;   // ← exits RunRegisteredActions entirely!
}

With the old code this would have stopped iterating the original list. With the new snapshot approach the behaviour is identical, but the intent is probably continue (skip this one callback) rather than `r…

src/BuildCheck.UnitTests/BuildCheckCentralContext_Tests.cs:17

[MAJOR] Test Coverage: No test for the concurrent deregister-during-dispatch scenario

The PR fixes two races: (1) register-during-dispatch (covered by the new test) and (2) concurrent deregister-while-dispatch-is-iterating (the original bug). The existing test only validates the snapshot semantics for case (1) — a callback registered mid-dispatch isn't called until the next dispatch — but there is no test that exercises case (2) concurrently.

A regression test for the original bug would l…

Synchronize callback presence checks, skip only disabled callbacks, and add coverage for concurrent deregistration and disabled callback ordering.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3b1601d6-eb67-4320-8084-deba81695daa
@YuliiaKovalova

Copy link
Copy Markdown
Member Author

Addressed the automated review feedback in ba59b19:

  • synchronized all Has…Actions reads with callback registry mutations
  • changed disabled-check handling from return to continue, with coverage proving later enabled checks still run
  • added a deterministic concurrent deregistration regression test that verifies dispatch uses the captured snapshot
  • removed the unused import

Focused result: 3 passed, 0 failed in BuildCheckCentralContext_Tests.

@JanProvaznik JanProvaznik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down — the diagnosis is right and the core design is good: RunRegisteredActions snapshots under the lock and releases it before invoking any callback, so check code can re-entrantly register/deregister without deadlocking. That's the part people usually get wrong.

I verified the change locally before reviewing:

Result
Build of PR head 0 errors, 0 warnings
New tests, with fix 3/3 pass
New tests, with BuildCheckCentralContext.cs reverted to base 3/3 fail — exact InvalidOperationException: Collection was modified at List'1.Enumerator.MoveNext()
Full BuildCheck.UnitTests suite 233/233 pass

So these are genuine regression tests that reproduce the reported MSB4166. I also grepped every access to the ten registries — all four paths (register / deregister / count / dispatch) are now synchronized, the ten locks in DeregisterCheck are sequential rather than nested (no lock-ordering risk), and the record-generated members don't enumerate the lists.

Requesting changes on two items below; the rest are notes.


1. Blocking: the Has* lock lands on the property-read hot path

HasRegisteredActions is consulted on every $(Property) expansion in the build:

Expander property expansion
  -> PropertiesUseTracker.TrackRead                  (Evaluation/PropertiesUseTracker.cs:48)
  -> LoggingContext.ProcessPropertyRead
  -> BuildCheckManager.ProcessPropertyRead           (BuildCheckManagerProvider.cs:717)
       if (!_buildCheckCentralContext.HasPropertyReadActions) return;   <-- now Monitor.Enter/Exit
  -> RunPropertyReadActions -> RunRegisteredActions                     <-- lock + ToArray()

This turns a lock-free int field read into a monitor acquire/release on an object shared by every evaluating thread, whenever -check is on.

The lock buys nothing here: List<T>.Count is a single aligned int read and cannot tear, and it gives callers no atomicity either — the value can change the instant the lock is released, so every consumer is TOCTOU regardless. Correctness is carried entirely by the snapshot in RunRegisteredActions.

Cheapest fix — keep a count updated inside the locks you already take in RegisterAction and DeregisterCheck, and read it without locking:

private volatile int _evaluatedPropertiesActionsCount;
internal bool HasEvaluatedPropertiesActions => _evaluatedPropertiesActionsCount > 0;

The alternative that also removes the per-dispatch ToArray() allocation is copy-on-write: store the callbacks as an array/ImmutableArray<T> swapped under a write lock, and let readers do a plain Volatile.Read + Length > 0. That makes the read path lock-free and allocation-free, and matches the existing CopyOnWritePropertyDictionary precedent in this repo. Worth noting RunEvaluatedItemsActions, RunEnvironmentVariableActions, RunProjectImportedActions and RunProjectProcessingDoneActions have no Has* guard at all, so they take the lock unconditionally per event.

If you'd rather keep the lock as-is, could you post a before/after measurement on a large solution with -check?

2. Blocking: adding TaskInvocationActions deregistration orphans _tasksBeingExecuted

Before this PR TaskInvocationActions was never deregistered, so HasTaskInvocationActions was monotonic once true. Now it can flip true -> false mid-build — CheckWrapper.MaxReportsNumberPerRule = 20 sets IsThrottled, and RemoveChecksAfterExecutedActions sweeps throttled checks into RemoveCheck -> DeregisterCheck. ExecCliBuildCheck hitting 20 reports on a build with many Exec tasks is realistic.

ProcessTaskStartedEventArgs adds to _tasksBeingExecuted under the flag, but ProcessTaskFinishedEventArgs (BuildEventsProcessor.cs:174) re-checks the flag and returns before _tasksBeingExecuted.Remove(taskKey). Entries for tasks in flight at the moment of the flip are never removed from that plain Dictionary, and there's no other cleanup path. Bounded by in-flight tasks per node, so small — but it's new, and previously unreachable.

Suggest doing the Remove unconditionally and gating only the RunTaskInvocationActions call on the flag.


3. The three added deregistrations are a separate bug fix, and are untested

DeregisterCheck(TaskInvocationActions, check);               // new
DeregisterCheck(EnvironmentVariableCheckDataActions, check); // new
DeregisterCheck(ProjectImportedCheckDataActions, check);     // new

This isn't just "cover every registry" — previously a check dismounted for throwing, throttling, or invalid config kept receiving those three callback kinds for the rest of the build, against an instance RemoveCheck had already Dispose()d. That's a real leak worth calling out in the description, and it's a behavior change in its own right (those checks will now correctly stop firing).

All three new tests only exercise EvaluatedPropertiesActions. A table-style test asserting DeregisterCheck clears all ten registries would be cheap. Note EnvironmentVariableCheckDataActions, ProjectImportedCheckDataActions, EvaluatedItemsActions and ProjectRequestProcessingDoneActions have no Has* accessor, so they aren't directly assertable today — adding them is trivial and would also let BuildEventsProcessor skip work for those events.

4. return -> continue is correct, but worth one line in the description

To be clear, this is a straightforward bug fix — the old return aborted dispatch for every subsequent check in the registry, and the abort point depended on ConcurrentBag enumeration order, so it can't have been intentional. continue is obviously right.

The only reason to mention it: its user-visible effect is that diagnostics which were previously suppressed will now fire, and this repo treats "new warnings appear" carefully because of /WarnAsError. The blast radius is genuinely small — SetupSingleCheck never materializes a check that's disabled everywhere, so reaching the disabled branch at dispatch requires per-project config variation (enabled for project A, disabled for project B via a different .editorconfig). Combined with -check being opt-in and experimental, I don't think this needs a ChangeWave. It just shouldn't be a surprise to whoever reads this PR later, since the description currently presents it as a mechanical refactor.

Same for the trailing _removeChecks(checksToRemove, checkContext) that return was skipping: its throttled-check sweep runs unconditionally and logs "Dismounting check ..." at MessageImportance.Normal, which is visible at default verbosity.

5. Follow-up: the rest of the concurrently-reached state is still unsynchronized

Not asking you to fix this here, but a tracking issue linked from this PR would be good. Per BuildCheck-Architecture.md a single manager instance serves both the logging thread (EventArgs) and RequestBuilder/evaluation threads (BuildExecution) in the in-proc node. If that's concurrent enough to corrupt a List<T> enumerator, it's concurrent enough to corrupt these:

  • _deferredEvalDiagnostics — plain Dictionary, mutated from inside check callbacks via IResultReporter.ReportResult. This is the one I'd prioritise; a Dictionary race is silent corruption rather than a clean exception. Making it a ConcurrentDictionary is a small change.
  • _tasksBeingExecuted, _deferredProjectEvalIdToImportedProjects — plain Dictionary.
  • CheckWrapper._stopwatch (a Stopwatch, documented not thread-safe, started/stopped by every WrappedHandler), _reportsCount++, IsThrottled, CommonConfig.
  • RemoveCheck's drain-and-refill of the ConcurrentBag isn't atomic — a concurrent SetupChecksForNewProject can observe an empty registry and set up a project with no checks.

Related: since the snapshot is taken before dispatch, "deregistered" no longer means "will not be invoked" — RemoveCheck calls Check.Dispose() immediately after DeregisterCheck, so a stale snapshot can invoke a disposed check. Benign for inbox checks (Dispose is a virtual no-op), not necessarily for acquired custom ones. A short <remarks> on BuildCheckCentralContext documenting the threading contract this PR establishes — snapshot semantics, Has* being advisory only, deregistration being best-effort for in-flight dispatches — would save the next reader a lot of time.

6. Nit: test hygiene

DeregisteringCheckDuringDispatchUsesCurrentSnapshot can't hang CI — continueAction.Set() is in a finally, so the worker is always released. But on the assertion-failure path await dispatch is skipped, so using var continueAction can dispose an event with a waiter still on it (ObjectDisposedException on an unobserved task). Moving the drain into the finally fixes it:

finally
{
    continueAction.Set();
    await dispatch.WaitAsync(TimeSpan.FromSeconds(30));
}

Also, new CheckWrapper(checkRule, null!) and the null! check contexts only work because checkContext is never dereferenced on the success path — it's used solely in the catch block. Tiny no-op ICheckContext / IResultReporter fakes would stop that from breaking the moment anyone adds logging there.

Use lock-free copy-on-write callback snapshots, preserve task cleanup when callbacks are removed, document dispatch semantics, and expand regression coverage across every registry.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 3b1601d6-eb67-4320-8084-deba81695daa
@YuliiaKovalova

Copy link
Copy Markdown
Member Author

@JanProvaznik Addressed the requested changes in 6bed7ab:

  • switched to copy-on-write arrays with lock-free Volatile reads, removing both the hot-path monitor and per-dispatch ToArray() allocation
  • made task-finished cleanup unconditional before checking whether task callbacks remain
  • added coverage for all ten deregistration paths and the in-flight task lifecycle
  • drained the worker task in the concurrency test and replaced
    ull! dependencies with no-op fakes
  • documented snapshot/advisory semantics and expanded the PR description for the additional behavior changes
  • opened Audit BuildCheck shared state for concurrent access #14598 for the broader shared-state concurrency audit

Full BuildCheck unit suite: 235 passed, 0 failed.

This was referenced Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants