Repository navigation
Fix concurrent BuildCheck callback mutation - #14590
Conversation
Snapshot registered callbacks under their synchronization lock before dispatch and synchronize every deregistration path. This prevents graph-build logging from failing when checks are registered or removed concurrently. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b1601d6-eb67-4320-8084-deba81695daa
There was a problem hiding this comment.
Pull request overview
Fixes a race in BuildCheck callback dispatch where callback lists could be mutated (register/deregister) while being enumerated, leading to InvalidOperationException and MSB4166 failures in graph builds.
Changes:
- Snapshot callback registries under the existing per-registry lock before dispatching callbacks to avoid concurrent mutation during enumeration.
- Synchronize callback deregistration and ensure all callback registries are covered.
- Add a regression unit test validating that registering a callback during dispatch is deferred to the next dispatch.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/Build/BuildCheck/Infrastructure/BuildCheckCentralContext.cs | Locks and snapshots callback lists before dispatch; locks deregistration and covers all registries to prevent concurrent List<T> mutation during enumeration. |
| src/BuildCheck.UnitTests/BuildCheckCentralContext_Tests.cs | Adds regression test for registering an evaluated-properties action during dispatch and ensuring it runs on the next dispatch. |
There was a problem hiding this comment.
MSBuild Expert Review — PR #14590
| # | Dimension | Verdict |
|---|---|---|
| 13 | Concurrency & Thread Safety | 🟡 MODERATE |
| 22 | Correctness & Edge Cases | 🟠 MAJOR (pre-existing) |
| 4 | Test Coverage & Completeness | 🟠 MAJOR |
✅ 21/24 dimensions clean.
Summary
The PR correctly fixes the core race condition: RunRegisteredActions now takes a snapshot under the list's own lock before iterating, and DeregisterCheck / RegisterAction both hold the same lock during mutation. The approach is sound and idiomatic.
Three findings worth addressing:
-
Concurrency (MODERATE) —
HasXxxActionsproperties read.Countwithout holding the lock. This is benign today (atomicintread used as a hint), but inconsistent with the newly-added locking discipline. Either add the lock or add a comment that these are single-threaded reads. (Inline comment at line 79) -
Correctness (MAJOR, pre-existing) —
returninsideRunRegisteredActionsexits the whole dispatch loop rather than skipping the single disabled check. If two checks are registered and the first hasIsEnabled = false, the second check is never invoked. Consider changingreturn→continue. (Inline comment at line 248) -
Test Coverage (MAJOR) — The new test covers register-during-dispatch but not the original bug scenario (deregister-during-dispatch causing
InvalidOperationException). A regression test for the original failure mode is missing. (Inline comment on test file)
Warning
Firewall blocked 1 domain
The following domain was blocked by the firewall during workflow execution:
awmgmcpg
To allow these domains, add them to the
network.allowedlist in your workflow frontmatter:
network:
allowed:
- defaults
- "awmgmcpg"See Network Configuration for more information.
Generated by Expert Code Review (on open) for #14590 · sonnet46 · 66.3 AIC · ⌖ 5.54 AIC · ⊞ 4.9K
Comments that could not be inline-anchored
src/Build/BuildCheck/Infrastructure/BuildCheckCentralContext.cs:79
[MODERATE] Concurrency: HasXxxActions properties read Count without holding the lock
The PR adds lock(list) guards to all mutations (RegisterAction, DeregisterCheck, snapshot in RunRegisteredActions), but the HasXxxActions convenience properties — used by callers to decide whether to collect data before calling RunXxxActions — read .Count without any synchronization:
internal bool HasEvaluatedPropertiesActions => _globalCallbacks.EvaluatedPropertiesActions.Coun…
</details>
<details><summary>src/Build/BuildCheck/Infrastructure/BuildCheckCentralContext.cs:248</summary>
**[MAJOR] Correctness: Early `return` exits entire dispatch loop, not just one callback**
Lines 247-250 (unchanged from the original, but the snapshot approach makes this more visible):
```csharp
if (!commonConfig.IsEnabled)
{
return; // ← exits RunRegisteredActions entirely!
}With the old code this would have stopped iterating the original list. With the new snapshot approach the behaviour is identical, but the intent is probably continue (skip this one callback) rather than `r…
src/BuildCheck.UnitTests/BuildCheckCentralContext_Tests.cs:17
[MAJOR] Test Coverage: No test for the concurrent deregister-during-dispatch scenario
The PR fixes two races: (1) register-during-dispatch (covered by the new test) and (2) concurrent deregister-while-dispatch-is-iterating (the original bug). The existing test only validates the snapshot semantics for case (1) — a callback registered mid-dispatch isn't called until the next dispatch — but there is no test that exercises case (2) concurrently.
A regression test for the original bug would l…
Synchronize callback presence checks, skip only disabled callbacks, and add coverage for concurrent deregistration and disabled callback ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b1601d6-eb67-4320-8084-deba81695daa
|
Addressed the automated review feedback in ba59b19:
Focused result: 3 passed, 0 failed in |
JanProvaznik
left a comment
There was a problem hiding this comment.
Thanks for tracking this down — the diagnosis is right and the core design is good: RunRegisteredActions snapshots under the lock and releases it before invoking any callback, so check code can re-entrantly register/deregister without deadlocking. That's the part people usually get wrong.
I verified the change locally before reviewing:
| Result | |
|---|---|
| Build of PR head | 0 errors, 0 warnings |
| New tests, with fix | 3/3 pass |
New tests, with BuildCheckCentralContext.cs reverted to base |
3/3 fail — exact InvalidOperationException: Collection was modified at List'1.Enumerator.MoveNext() |
Full BuildCheck.UnitTests suite |
233/233 pass |
So these are genuine regression tests that reproduce the reported MSB4166. I also grepped every access to the ten registries — all four paths (register / deregister / count / dispatch) are now synchronized, the ten locks in DeregisterCheck are sequential rather than nested (no lock-ordering risk), and the record-generated members don't enumerate the lists.
Requesting changes on two items below; the rest are notes.
1. Blocking: the Has* lock lands on the property-read hot path
HasRegisteredActions is consulted on every $(Property) expansion in the build:
Expander property expansion
-> PropertiesUseTracker.TrackRead (Evaluation/PropertiesUseTracker.cs:48)
-> LoggingContext.ProcessPropertyRead
-> BuildCheckManager.ProcessPropertyRead (BuildCheckManagerProvider.cs:717)
if (!_buildCheckCentralContext.HasPropertyReadActions) return; <-- now Monitor.Enter/Exit
-> RunPropertyReadActions -> RunRegisteredActions <-- lock + ToArray()
This turns a lock-free int field read into a monitor acquire/release on an object shared by every evaluating thread, whenever -check is on.
The lock buys nothing here: List<T>.Count is a single aligned int read and cannot tear, and it gives callers no atomicity either — the value can change the instant the lock is released, so every consumer is TOCTOU regardless. Correctness is carried entirely by the snapshot in RunRegisteredActions.
Cheapest fix — keep a count updated inside the locks you already take in RegisterAction and DeregisterCheck, and read it without locking:
private volatile int _evaluatedPropertiesActionsCount;
internal bool HasEvaluatedPropertiesActions => _evaluatedPropertiesActionsCount > 0;The alternative that also removes the per-dispatch ToArray() allocation is copy-on-write: store the callbacks as an array/ImmutableArray<T> swapped under a write lock, and let readers do a plain Volatile.Read + Length > 0. That makes the read path lock-free and allocation-free, and matches the existing CopyOnWritePropertyDictionary precedent in this repo. Worth noting RunEvaluatedItemsActions, RunEnvironmentVariableActions, RunProjectImportedActions and RunProjectProcessingDoneActions have no Has* guard at all, so they take the lock unconditionally per event.
If you'd rather keep the lock as-is, could you post a before/after measurement on a large solution with -check?
2. Blocking: adding TaskInvocationActions deregistration orphans _tasksBeingExecuted
Before this PR TaskInvocationActions was never deregistered, so HasTaskInvocationActions was monotonic once true. Now it can flip true -> false mid-build — CheckWrapper.MaxReportsNumberPerRule = 20 sets IsThrottled, and RemoveChecksAfterExecutedActions sweeps throttled checks into RemoveCheck -> DeregisterCheck. ExecCliBuildCheck hitting 20 reports on a build with many Exec tasks is realistic.
ProcessTaskStartedEventArgs adds to _tasksBeingExecuted under the flag, but ProcessTaskFinishedEventArgs (BuildEventsProcessor.cs:174) re-checks the flag and returns before _tasksBeingExecuted.Remove(taskKey). Entries for tasks in flight at the moment of the flip are never removed from that plain Dictionary, and there's no other cleanup path. Bounded by in-flight tasks per node, so small — but it's new, and previously unreachable.
Suggest doing the Remove unconditionally and gating only the RunTaskInvocationActions call on the flag.
3. The three added deregistrations are a separate bug fix, and are untested
DeregisterCheck(TaskInvocationActions, check); // new
DeregisterCheck(EnvironmentVariableCheckDataActions, check); // new
DeregisterCheck(ProjectImportedCheckDataActions, check); // newThis isn't just "cover every registry" — previously a check dismounted for throwing, throttling, or invalid config kept receiving those three callback kinds for the rest of the build, against an instance RemoveCheck had already Dispose()d. That's a real leak worth calling out in the description, and it's a behavior change in its own right (those checks will now correctly stop firing).
All three new tests only exercise EvaluatedPropertiesActions. A table-style test asserting DeregisterCheck clears all ten registries would be cheap. Note EnvironmentVariableCheckDataActions, ProjectImportedCheckDataActions, EvaluatedItemsActions and ProjectRequestProcessingDoneActions have no Has* accessor, so they aren't directly assertable today — adding them is trivial and would also let BuildEventsProcessor skip work for those events.
4. return -> continue is correct, but worth one line in the description
To be clear, this is a straightforward bug fix — the old return aborted dispatch for every subsequent check in the registry, and the abort point depended on ConcurrentBag enumeration order, so it can't have been intentional. continue is obviously right.
The only reason to mention it: its user-visible effect is that diagnostics which were previously suppressed will now fire, and this repo treats "new warnings appear" carefully because of /WarnAsError. The blast radius is genuinely small — SetupSingleCheck never materializes a check that's disabled everywhere, so reaching the disabled branch at dispatch requires per-project config variation (enabled for project A, disabled for project B via a different .editorconfig). Combined with -check being opt-in and experimental, I don't think this needs a ChangeWave. It just shouldn't be a surprise to whoever reads this PR later, since the description currently presents it as a mechanical refactor.
Same for the trailing _removeChecks(checksToRemove, checkContext) that return was skipping: its throttled-check sweep runs unconditionally and logs "Dismounting check ..." at MessageImportance.Normal, which is visible at default verbosity.
5. Follow-up: the rest of the concurrently-reached state is still unsynchronized
Not asking you to fix this here, but a tracking issue linked from this PR would be good. Per BuildCheck-Architecture.md a single manager instance serves both the logging thread (EventArgs) and RequestBuilder/evaluation threads (BuildExecution) in the in-proc node. If that's concurrent enough to corrupt a List<T> enumerator, it's concurrent enough to corrupt these:
_deferredEvalDiagnostics— plainDictionary, mutated from inside check callbacks viaIResultReporter.ReportResult. This is the one I'd prioritise; aDictionaryrace is silent corruption rather than a clean exception. Making it aConcurrentDictionaryis a small change._tasksBeingExecuted,_deferredProjectEvalIdToImportedProjects— plainDictionary.CheckWrapper._stopwatch(aStopwatch, documented not thread-safe, started/stopped by everyWrappedHandler),_reportsCount++,IsThrottled,CommonConfig.RemoveCheck's drain-and-refill of theConcurrentBagisn't atomic — a concurrentSetupChecksForNewProjectcan observe an empty registry and set up a project with no checks.
Related: since the snapshot is taken before dispatch, "deregistered" no longer means "will not be invoked" — RemoveCheck calls Check.Dispose() immediately after DeregisterCheck, so a stale snapshot can invoke a disposed check. Benign for inbox checks (Dispose is a virtual no-op), not necessarily for acquired custom ones. A short <remarks> on BuildCheckCentralContext documenting the threading contract this PR establishes — snapshot semantics, Has* being advisory only, deregistration being best-effort for in-flight dispatches — would save the next reader a lot of time.
6. Nit: test hygiene
DeregisteringCheckDuringDispatchUsesCurrentSnapshot can't hang CI — continueAction.Set() is in a finally, so the worker is always released. But on the assertion-failure path await dispatch is skipped, so using var continueAction can dispose an event with a waiter still on it (ObjectDisposedException on an unobserved task). Moving the drain into the finally fixes it:
finally
{
continueAction.Set();
await dispatch.WaitAsync(TimeSpan.FromSeconds(30));
}Also, new CheckWrapper(checkRule, null!) and the null! check contexts only work because checkContext is never dereferenced on the success path — it's used solely in the catch block. Tiny no-op ICheckContext / IResultReporter fakes would stop that from breaking the moment anyone adds logging there.
Use lock-free copy-on-write callback snapshots, preserve task cleanup when callbacks are removed, document dispatch semantics, and expand regression coverage across every registry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3b1601d6-eb67-4320-8084-deba81695daa
|
@JanProvaznik Addressed the requested changes in 6bed7ab:
Full BuildCheck unit suite: 235 passed, 0 failed. |
Summary
Motivation
Graph builds can register or remove BuildCheck callbacks while another thread dispatches the same registry. Enumerating the mutable List caused InvalidOperationException, which the logging pipeline surfaced as InternalLoggerException / MSB4166.
The previous deregistration path also left task, environment-variable, and project-import callbacks active after a check was removed. Task callback removal now keeps task-finished cleanup unconditional so in-flight tracking entries are not orphaned.
Tests
Follow-up