Add locks on concurrent alias following checker accesses under incremental mode#2051
Conversation
…e checker access tracked by checker pools
There was a problem hiding this comment.
Pull Request Overview
This PR fixes a race condition in incremental compilation mode where concurrent access to the same checker for alias following operations could cause data races. The fix introduces thread-safe exclusive checker access via locks and enables incremental mode testing through compiler tests with @incremental: true.
- Adds
GetCheckerForFileExclusivemethods to checker pool interfaces, which lock a mutex per checker to prevent concurrent access - Updates
affectedfileshandler.goto use exclusive checker access for alias following operations - Extends the
ProgramLikeinterface with additional methods needed for incremental program compatibility - Enables incremental mode in test infrastructure when
@incremental: trueis specified
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
| testdata/tests/cases/compiler/incrementalConcurrentSafeAliasFollowing.ts | New regression test with multiple files and export aliases to reproduce the race condition |
| testdata/baselines/reference/compiler/incrementalConcurrentSafeAliasFollowing.* | Expected baseline outputs for the new test |
| internal/project/checkerpool.go | Adds per-checker mutex locks and GetCheckerForFileExclusive implementation for the project-level checker pool |
| internal/compiler/checkerpool.go | Adds per-checker mutex locks and GetCheckerForFileExclusive implementation for the compiler-level checker pool |
| internal/compiler/program.go | Exposes GetCheckerForFileExclusive and extends ProgramLike interface |
| internal/execute/incremental/affectedfileshandler.go | Uses exclusive checker access to prevent race condition during alias following |
| internal/execute/incremental/program.go | Implements additional ProgramLike interface methods for compatibility |
| internal/testutil/harnessutil/harnessutil.go | Enables incremental program construction in tests, changes Program field to ProgramLike |
| internal/testutil/tsbaseline/*.go | Updates to work with ProgramLike interface instead of concrete Program type |
| internal/testrunner/compiler_runner.go | Adds type switching logic to extract underlying Program from ProgramLike |
|
|
||
| func (p *checkerPool) GetCheckerForFileExclusive(ctx context.Context, file *ast.SourceFile) (*checker.Checker, func()) { | ||
| c, done := p.GetCheckerForFile(ctx, file) | ||
| idx := slices.Index(p.checkers, c) |
There was a problem hiding this comment.
Missing error handling for when slices.Index returns -1 (checker not found). If the checker is not in the slice, this will cause a panic when accessing p.locks[idx]. Although this should never happen in normal operation, defensive programming suggests adding a check or assertion.
| idx := slices.Index(p.checkers, c) | |
| idx := slices.Index(p.checkers, c) | |
| if idx == -1 { | |
| panic("checker not found in checker pool") | |
| } |
|
|
||
| //// [a.tsbuildinfo] | ||
| { | ||
| "version": "TEST", |
There was a problem hiding this comment.
Error: Unexpected token
This reverts commit b42a8b5.
* Port 'go to type definition' tests (microsoft#1883) * Fix panic in `getTokenAtPosition` for JSDoc type assertions (microsoft#1846) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> Co-authored-by: andrewbranch <3277153+andrewbranch@users.noreply.github.com> Co-authored-by: Andrew Branch <andrew@wheream.io> * Don’t look in JSExportAssignment and CommonJSExport for nodes (microsoft#1886) * Fix link in native preview platform packages (microsoft#1838) * fix(1880): No error message for JSDoc type parsing (microsoft#1881) * Add vscode editor issue template (microsoft#1893) Co-authored-by: Ryan Cavanaugh <RyanCavanaugh@users.noreply.github.com> * Add "Report Issue" button to TSGO status bar commands (microsoft#1889) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: DanielRosenwasser <972891+DanielRosenwasser@users.noreply.github.com> Co-authored-by: Daniel Rosenwasser <DanielRosenwasser@users.noreply.github.com> * fix(1898): adjust location handling in find-refs (microsoft#1901) * Fix panic of empty string in type reference directive (microsoft#1908) * Consistently error on full circle of circular import aliases (microsoft#1904) * Fix panic in textDocument/onTypeFormatting when tokenAtPosition is nil (microsoft#1845) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Update submodule (microsoft#1913) * Disable create-cache.yml in forks (microsoft#1912) * Forbid platform specific package uses in agnostic files (microsoft#1911) * Fix JSDoc comment formatting with tab indentation (microsoft#1900) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Clear local baseline dir in hereby test (microsoft#1921) * Unskip passing fourslash test (microsoft#1922) * Support auto-import completion fourslash tests, fix bugs (microsoft#1917) * Fix JSX indentation in JavaScript output (microsoft#1792) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Implement printAllHelp to fix `tsgo --all` producing no output (microsoft#1843) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Bump the github-actions group across 1 directory with 2 updates (microsoft#1909) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com> * Ensure os package is forbidden in lint (microsoft#1924) * Speed up levenshteinWithMax by reusing buffers (microsoft#1823) * Fix incorrect formatting for comments inside multi-line argument lists and method chains (microsoft#1929) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Handle nil end position in getMappedLocation (microsoft#1920) * Fix formatter adding extra space at end of line without trailing newline (microsoft#1933) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Fix vscode issue template (microsoft#1934) * userpreferences parsing/ls config handing (microsoft#1729) * Plumb through TokenFlagsSingleQuote; use for auto import quote detection (microsoft#1937) * Invalidate caches on batches of 1000+ watch changes (microsoft#1869) * Create clickable links in quick info from @link JSDoc tags (microsoft#1935) * Don't report errors on `{@link foo.bar}` references (microsoft#1941) * Fix crash in `invocationErrorRecovery` function (microsoft#1944) * Fix leading source file comment emit bugs (microsoft#1945) * Implement selection ranges (microsoft#1939) * Fix porting bug in isArgumentAndStartLineOverlapsExpressionBeingCalled (microsoft#1948) * Add Range to Hover (microsoft#1489) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: DanielRosenwasser <972891+DanielRosenwasser@users.noreply.github.com> Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com> * Properly handle hovering on `this` (microsoft#1953) * Bump the github-actions group across 1 directory with 2 updates (microsoft#1959) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fixed document highlight for reference directive (microsoft#1951) * Several fixes to JS typing of functions and methods (microsoft#1960) * Update submodule, port 6.0 options defaults (microsoft#1961) * Reapply microsoft#1951 and microsoft#1960 after bad merge (microsoft#1964) Co-authored-by: Anders Hejlsberg <andersh@microsoft.com> Co-authored-by: John Favret <64748847+johnfav03@users.noreply.github.com> * Update submodule with ES5 removals (microsoft#1963) * Actually transform KindCommonJSExport in declaration emit (microsoft#1962) * Quick Info fixes (microsoft#1971) * Fix various named enum types (microsoft#1973) * Move change tracker, converters, utils to separate packages (microsoft#1977) * Consistent rules for mixing `@type`, `@param`, `@return`, `@template` (microsoft#1979) * Check for identifier before obtaining text of name (microsoft#1984) * Respect client capabilities for diagnostics (microsoft#1980) * Store explicitly declared members ahead of inherited members (microsoft#1987) * Add --checkers to control number of checkers per Program (microsoft#1985) * Export all types referenced through other exported APIs, enforce (microsoft#1978) * Switch custom runners from mariner-2.0 to azure-linux-3 (microsoft#1989) * Use `LocationLink` in go to definition (microsoft#1884) * Use a different set of commands to detect fourslash test updates (microsoft#1923) * Skip erasableSyntaxOnly checks for JavaScript files (microsoft#1956) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Update submodule for new cherry-picks (microsoft#1996) * Port TypeScript PR #62604: Propagate variance reliability (microsoft#1916) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com> Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com> * Only export `@typedef` type aliases in modules (microsoft#1999) * Bump github/codeql-action from 4.31.0 to 4.31.2 in the github-actions group across 1 directory (microsoft#2005) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Hoist @typedef and @import tags to containing scopes that permit them (microsoft#2003) * Remove concept of "unsupported extensions", clean up test skips (microsoft#2004) * Update golangci-lint, fix issues, modernize (microsoft#1981) * Implement more handling of client capabilities (microsoft#1998) * Add docs to signature help (microsoft#2009) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix unused identifier diags, LSP tag diags (microsoft#2007) * fix(2015): abstract property created, overshadowing override (microsoft#2016) * Always check refCount after acquiring lock (microsoft#1986) * Delete resolver unit tests (microsoft#2008) * Fix missing parent for `Expression` in `TypeParameterDeclaration` (microsoft#2017) * Add missing nil check in `getCompletionItemActions` (microsoft#2018) * Fix crash in find-all-refs on `exports.xxx` in .js file (microsoft#2023) * Properly include JSX attributes in find-all-references (microsoft#2025) * Fix crash by removing `getNameFromImportDeclaration` in favor of `Node.Name()` (microsoft#2027) * Fix losing options from command line in watch mode (microsoft#2024) * Add issue investigator agent (microsoft#2030) * Switch 1ESPT pipelines to 1ESPT-AzureLinux3 (microsoft#2031) * Port inlay hints (microsoft#1705) * Split "use strict" into separate transformer, fix bugs with prologues (microsoft#2028) Co-authored-by: Sheetal Nandi <shkamat@microsoft.com> * Use a more cross-architecture-friendly devcontainer image. (microsoft#2034) * Fix nil pointer dereference in range formatting (microsoft#1993) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: DanielRosenwasser <972891+DanielRosenwasser@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> Co-authored-by: Daniel Rosenwasser <DanielRosenwasser@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Port missing `checkJs` logic (microsoft#2046) * Ignore reparsed nodes when determining external module indicator (microsoft#2044) * Fix various fuzzer-caught crashes in the parser (microsoft#2038) * Fix moduleDetection for node18, fix __esModule in detect=force (microsoft#2045) * Fix panic in syncmap on loading nil (microsoft#2056) * Use accessors on `Node` instead of casts and field accesses (microsoft#2052) * Add locks on concurrent alias following checker accesses under incremental mode (microsoft#2051) * Don't add `export` modifier to `JSTypeAliasDeclaration` from `@callback` (microsoft#2063) * Introduce GetECMALineOfPosition to avoid unused rune counting (microsoft#2065) * Don't add `export` modifier to `KindCommonJSExport` reparsed nodes (microsoft#2066) * Fix panic in inlay hints for tuple types (microsoft#2040) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Accurately recognize fourslash test as submodule (microsoft#2068) * Implement auto-import code actions, port tests and fix some bugs (microsoft#2053) * Port tsc --init (microsoft#2033) * Make CheckerPool iteration concurrent by default (microsoft#2070) * Use Microsoft build of Go in CI (microsoft#2069) * Detect Windows junctions with GetFileAttributesEx (microsoft#2013) * Fix CI cache workflow (microsoft#2071) * Use information from contextual type in hovers/quick info (microsoft#2073) * fix(2074): No quick info on function and other similar tokens (microsoft#2078) * Unify locks used on checkers between exclusive pool borrows and EmitResolver scopes (microsoft#2080) * Port non-baseline diagnostics tests (microsoft#2079) * Use SkipTrivia instead of GetRangeOfTokenAtPosition where possible (microsoft#2089) * Move unreachable checks to checker, allowing more AST reuse (microsoft#2067) * Fix scanning of valid surrogate pairs (microsoft#2032) * Fix misplaced parentheses in `Checker.isIndirectCall` (microsoft#2093) * Various agent mode updates (microsoft#2094) * Handle configuration changes in LSP for 'typescript.*' options. (microsoft#2088) * Fix nil pointer dereference in getAdjustedLocation for type-only exports (microsoft#2090) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Fix nil pointer dereference in code actions when diagnostic code is nil (microsoft#2091) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> * Fully resolve LSP client caps to non-pointers, pass by context (microsoft#2095) * Fix hover on `module.exports` (microsoft#2098) * Accept and document jsdoc diffs, round 1 (microsoft#1426) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Jake Bailey <5341706+jakebailey@users.noreply.github.com> * Port baseline diagnostics tests (microsoft#2097) * Clean up disk space in CI before running (microsoft#2103) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add GOBIN to PATH in CI (microsoft#2105) * Make client requests type safe, unmarshal (microsoft#2099) * Display inherited JSDoc documentation in quick info (microsoft#2111) * Sort failingTests and manualTests in en-US (microsoft#2113) * fix(2047): Incomplete declaration emit of callback tag with no return tag (microsoft#2100) * Fix canHaveSyntheticDefault (microsoft#2101) * Misc fixes (microsoft#2112) * chore: fix incorrect function name in comment (microsoft#2109) Signed-off-by: weifangc <weifangcheng@outlook.jp> * Fix typedef binding with CJS `exports=` (microsoft#826) * Provide Program diagnostics as push diags in tsconfig.json (microsoft#2118) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update dependencies (microsoft#2116) * Remove copilot-setup-steps env var (microsoft#2124) * Fix panic on negative parameterIndex in type predicate flow analysis (microsoft#2122) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com> Co-authored-by: Ryan Cavanaugh <ryanca@microsoft.com> * Port tests for go to implementation and diff definitions tests (microsoft#2130) * Partially fix multi-checker diagnostics consistency (microsoft#2134) * Implement reportStyleChecksAsWarnings (microsoft#2132) * Include docs on resolved client caps (microsoft#2135) * Fix dynamic import grammar check (microsoft#2138) * Refine LSP with our own types, generate more stuff (microsoft#2141) * Display all symbol meanings in quick info (microsoft#2144) * Multiproject requests like find all refs, rename and workspace symbols (microsoft#1991) * Add stringer-alike String methods to non-string LSP enums (microsoft#2148) * Enable localization (microsoft#2123) * Port workspace symbols tests (microsoft#2146) * Update readme, issue template (microsoft#2140) * Ignore config port (microsoft#1755) * fix(2157): jsdocfunction param is inferred as implicit any when directly assigned to module.exports (microsoft#2158) * fixes after merge * some more fixes * more errors * builds * fmt * fix nil pointer deref * fix error messages --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: weifangc <weifangcheng@outlook.jp> Co-authored-by: Gabriela Araujo Britto <garaujobritto@gmail.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: jakebailey <5341706+jakebailey@users.noreply.github.com> Co-authored-by: andrewbranch <3277153+andrewbranch@users.noreply.github.com> Co-authored-by: Andrew Branch <andrew@wheream.io> Co-authored-by: Andrew Branch <andrewbranch@users.noreply.github.com> Co-authored-by: Oleksandr T. <oleksandr.tarasiuk@outlook.com> Co-authored-by: Matt Bierner <12821956+mjbvz@users.noreply.github.com> Co-authored-by: Ryan Cavanaugh <RyanCavanaugh@users.noreply.github.com> Co-authored-by: DanielRosenwasser <972891+DanielRosenwasser@users.noreply.github.com> Co-authored-by: Daniel Rosenwasser <DanielRosenwasser@users.noreply.github.com> Co-authored-by: Twacqwq <69360546+Twacqwq@users.noreply.github.com> Co-authored-by: Anders Hejlsberg <andersh@microsoft.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Isabel Duan <isabelduan@microsoft.com> Co-authored-by: John Favret <64748847+johnfav03@users.noreply.github.com> Co-authored-by: Wesley Wigham <wwigham@gmail.com> Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: xu0o0 <hq.xu0o0@gmail.com> Co-authored-by: Sheetal Nandi <shkamat@microsoft.com> Co-authored-by: Nathan Shively-Sanders <293473+sandersn@users.noreply.github.com> Co-authored-by: weifangc <weifangcheng@outlook.jp> Co-authored-by: Ryan Cavanaugh <ryanca@microsoft.com> Co-authored-by: Nathan Whitaker <nathan@deno.com>
And, so we can test for these races more simply, enable the incremental mode program construction+emit codepath for compiler tests with
@incremental: truespecified.There are other
GetCheckerForFilecalls ininternal/execute/incrementalthat probably need to be swapped toGetCheckerForFileExclusive, as using basically any checker functions in a concurrent context without a lock on it could lead to a data race (thanks to lazy caching everywhere inside the checker), but the single call I changed here is definitely the root cause of the crash in the issue. I could just go swap them all, but we don't have regression tests for those sites to prove that's actually needed. The LS has request-scoped checker associations and assumes one thread per request, so is unlikely to need the per-checker lock functionality (though it is here, since I made it part of the pool interface), unless the number of in-flight requests exceeds the maximum number of available checkers (which can only happen if there's a disconnect between request workgroup max paralleism and max checkers afaik).Fixes #1470