Repository navigation
perf: cache pnpm's lockfile verification results #287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 3 commits
c0a6b0f
f141ddd
544072d
b543421
e6cb65a
34f0a19
987541b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| import { restoreCache, saveCache } from '@actions/cache' | ||
| import { debug, getState, info, saveState, warning } from '@actions/core' | ||
| import { getExecOutput } from '@actions/exec' | ||
| import { existsSync } from 'fs' | ||
| import os from 'os' | ||
| import path from 'path' | ||
| import { removeWindowsExtendedPathPrefix } from '../windows-path' | ||
|
|
||
| /** | ||
| * Where pnpm v11+ memoizes which lockfile passed which supply-chain policies. | ||
| * A job without it re-checks every lockfile entry against the registry, which | ||
| * on a large repository costs more than the install. | ||
| */ | ||
| const VERIFICATION_CACHE_FILE = 'lockfile-verified.jsonl' | ||
|
|
||
| const PATH_STATE = 'lockfile_verification_cache_path' | ||
| const KEY_STATE = 'lockfile_verification_cache_key' | ||
| const RESTORED_STATE = 'lockfile_verification_cache_restored' | ||
|
|
||
| /** | ||
| * The verdict is only valid for the exact lockfile content it was recorded | ||
| * for, so this cache is keyed on the same lockfile hash as the store cache | ||
| * but restored without prefix fallback: an older entry could never be used. | ||
| */ | ||
| export async function restoreVerificationCache(lockfileHash: string): Promise<void> { | ||
| try { | ||
| const cacheFilePath = path.join(await getPnpmCacheDirectory(), VERIFICATION_CACHE_FILE) | ||
| const key = `pnpm-lockfile-verified-${process.env.RUNNER_OS}-${os.arch()}-${lockfileHash}` | ||
|
Comment on lines
+35
to
+42
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Inspect verification-cache key material and all policy inputs.
rg -n -C 3 'lockfile-verified|lockfile_verification_cache|saveVerificationCache' src
rg -n -C 3 'minimumReleaseAge|trustPolicy|minimumReleaseAgeExclude|trustPolicyExclude' \
src .github action.yml README.mdRepository: pnpm/action-setup Length of output: 4902 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- lockfile verification cache ---'
cat -n src/lockfile-verification-cache/index.ts
printf '%s\n' '--- input and install flow ---'
ast-grep outline src/inputs.ts src/index.ts src/pnpm-install.ts 2>/dev/null || true
rg -n -C 5 'restoreVerificationCache|minimumReleaseAge|trustPolicy|minimumReleaseAgeExclude|trustPolicyExclude|pnpmInstall|lockfileHash|hashFiles' \
src action.yml README.md .github/workflows
printf '%s\n' '--- package metadata and tests ---'
rg -n -C 4 'lockfile-verification|verification cache|minimumReleaseAge|trustPolicy|saveState|restoreCache|saveCache' \
package.json __tests__ test tests 2>/dev/null || trueRepository: pnpm/action-setup Length of output: 11152 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- cache inputs and dependency paths ---'
cat -n src/cache-restore/run.ts
cat -n src/inputs.ts
cat -n action.yml | sed -n '1,180p'
printf '%s\n' '--- policy/configuration references ---'
rg -n -C 6 'cacheDependencyPath|pnpm-workspace|\\.npmrc|minimumReleaseAge|trustPolicy|config' \
src action.yml README.md .github/workflows package.json
printf '%s\n' '--- tracked files relevant to policy resolution ---'
git ls-files | rg '(^|/)(pnpm-workspace\\.yaml|\\.npmrc|package\\.json|action\\.yml|README\\.md)$|lockfile-verification'Repository: pnpm/action-setup Length of output: 2664 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- repository paths ---'
fd -t f -a | rg '(^|/)(inputs|action\\.yml|README\\.md|package\\.json)$|lockfile-verification|cache-restore'
printf '%s\n' '--- action inputs ---'
cat -n action.yml | sed -n '1,220p'
printf '%s\n' '--- input implementation ---'
fd -t f -a src | rg 'input'
for f in $(fd -t f -a src | rg 'input'); do
echo "--- $f"
cat -n "$f"
done
printf '%s\n' '--- policy/configuration references ---'
rg -n -C 6 'cacheDependencyPath|pnpm-workspace|\\.npmrc|minimumReleaseAge|trustPolicy|config' \
src action.yml README.md .github/workflows package.json 2>/dev/null || trueRepository: pnpm/action-setup Length of output: 2210 🌐 Web query:
💡 Result: In pnpm v11, lockfile verification was significantly hardened to prevent supply-chain attacks, primarily by ensuring that existing lockfile entries are re-validated against security policies even when they were already present in the lockfile [1][2][3]. Previously, pnpm might skip security checks for dependencies already resolved in the lockfile, which created a vulnerability where malicious or immature packages could persist undetected [3]. Key components and changes in pnpm v11 regarding lockfile verification: 1. Mandatory Lockfile Re-validation: pnpm v11 introduced a post-load, pre-fetch gate that re-applies policies like minimumReleaseAge and trustPolicy ('no-downgrade') to every dependency entry in the lockfile [4][1][2]. If a locked version violates these policies, the install aborts (e.g., ERR_PNPM_MINIMUM_RELEASE_AGE_VIOLATION) [1]. 2. trustPolicy and minimumReleaseAge: - trustPolicy: 'no-downgrade' ensures that a lockfile entry does not use a version with weaker security evidence (like missing provenance) than an earlier published version [5]. - minimumReleaseAge: Enforces a time-based cutoff for packages to prevent installing very recent (potentially malicious) releases [1]. 3. trustLockfile Setting: Introduced in v11.3, this setting allows users to explicitly opt out of the re-validation pass for trusted environments, such as closed-source projects where every commit is authored by a trusted team member [4][6][7]. When set to true in pnpm-workspace.yaml (or via --trust-lockfile), pnpm skips the verification of existing lockfile entries to improve performance and avoid potential memory bottlenecks [6][7]. 4. Performance and Memory Optimizations: Early iterations of the v11 re-validation pass caused significant memory usage and performance regressions on large workspaces [8][7]. Subsequent updates (v11.3 and later) optimized this by: - Storing only essential metadata fields (time, trustedPublisher, attestations) in the cache rather than full packuments [4][6]. - Enabling metadata cache reuse for the verification pass, resolving issues where re-validation would incorrectly attempt to fetch full metadata for every entry [8]. - Running lockfile verification concurrently with fetching and linking instead of blocking the entire installation process [9]. 5. Verification Security: Regardless of policies, pnpm v11 also enforces an always-on integrity check for tarballs and rejects malformed or malicious lockfile aliases (e.g., those attempting path traversal) before any filesystem operations occur [10][9]. Regarding "lockfile-verified.jsonl," this file is not a standard pnpm-maintained configuration file; based on current documentation, it does not exist as a formal part of the pnpm v11 specification. Verification results and policy enforcements are handled internally by pnpm's resolution and installation engines [1][5]. Citations:
🌐 Web query:
💡 Result: lockfile-verified.jsonl is a cache file used by the pnpm package manager (and its associated Rust-based implementation, pacquet) to store and memoize the results of lockfile verification checks [1][2][3]. Its primary purpose is to enable a fast-path performance optimization during the installation process. When pnpm performs lockfile verification—such as checking policies like minimumReleaseAge—it caches the outcome in this file [2][3]. On subsequent runs, if the lockfile remains unchanged, pnpm can use this cached record to skip redundant verification steps [4][5][2]. Key technical aspects include: - Format: It uses the JSON Lines (JSONL) format [1][3]. - Optimization logic: It employs a "stat shortcut" to quickly identify unchanged lockfiles by comparing metadata such as file size, modification time (mtime), and inode [1][3]. If the metadata matches, it avoids expensive re-hashing of the lockfile [1]. - Reliability: If metadata differs (for example, in CI environments where file timestamps may be reset), it falls back to a content-based hash (sha256) to verify if the lockfile is effectively the same [1][3]. - Location: The file is stored within pnpm's on-disk cache directory [4][5]. - Maintenance: It includes a compaction mechanism that dedupes entries by (path, hash) and prunes old entries to maintain a reasonable file size [2]. By memoizing these verification verdicts, pnpm significantly improves performance for repeated installs in scenarios where lockfile integrity policies are enforced [6][3]. Citations:
🌐 Web query:
💡 Result: In pnpm v11, the Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
base='https://raw.githubusercontent.com/pnpm/pnpm/main'
for file in \
'pnpm11/resolving/resolver-base/src/lockfileVerificationCache.ts' \
'pnpm11/resolving/resolver-base/src/verifyLockfile.ts' \
'pnpm11/resolving/resolver-base/CHANGELOG.md' \
'installing/deps-installer/src/install/verifyLockfileResolutions.ts'
do
echo "--- $file"
curl -fsSL "$base/$file" | rg -n -C 5 \
'lockfile-verified|verification.*cache|cache.*verif|minimumReleaseAge|trustPolicy|policy|hash' || true
done
printf '%s\n' '--- repository tree candidates ---'
curl -fsSL 'https://api.github.com/repos/pnpm/pnpm/git/trees/main?recursive=1' |
jq -r '.tree[]?.path' |
rg 'lockfile.*verif|verif.*cache|verifyLockfileResolutions|resolver-base'Repository: pnpm/action-setup Length of output: 679 🌐 Web query:
💡 Result: The file lockfile-verified.jsonl serves as a persistent, on-disk cache for lockfile verification results in pnpm's Pacquet implementation [1][2]. Its primary purpose is to avoid redundant, expensive registry lookups and lockfile hashing on repeated installs when the lockfile and verification policies remain unchanged [2][3][4]. Key technical aspects include: 1. Functionality: It acts as a memoization layer for the lockfile verification process (which enforces policies like minimumReleaseAge or trustPolicy) [2][3]. When a user runs an install, the system checks this cache first. If a valid, compatible record exists, the verification gate is bypassed [5][6][4]. 2. Cache Logic: The system uses a "stat shortcut" to quickly identify if the lockfile has changed [3]. It compares current file metadata (size, mtime, inode) against the cached record [2][3]. If the metadata matches, the system trusts the previous verification result [3]. If it differs, it falls back to computing a hash of the lockfile to confirm if the content is still valid under the currently active policies [2][3]. 3. Policy Hash/Snapshot: To ensure correctness, the cache record includes a representation of the policies in effect at the time of verification [1][5]. This prevents "downgrade" issues where a user might tighten their security policies (e.g., increasing a minimum release age) between two installations; if the policies are different, the cache entry is invalidated, and re-verification occurs [5][3][4]. 4. Compatibility: This file is designed to mirror the behavior of the upstream TypeScript implementation in pnpm, ensuring that caches populated by one version or implementation can be read by others [1]. 5. Maintenance: The cache includes built-in compaction logic to manage its size, typically triggered when it reaches a certain number of entries or size limit (e.g., 1000 entries or ~1.5MB) [2]. This mechanism effectively enables "fast-path" installs that complete in milliseconds when the environment is warm and the project state is unchanged [5][4]. Citations:
🌐 Web query:
💡 Result: In the pnpm codebase, specifically within the pacquet project's lockfile verification logic, the policy cache key is derived from a combination of the lockfile's content (often hashed) and the active security or supply-chain policy snapshots [1][2][3]. The system uses a cache to short-circuit the lockfile verification process, which ensures that dependencies meet defined security and integrity policies [1][4]. The caching mechanism functions as follows: 1. Cache Identity: Verification is memoized based on an identity that includes the lockfile's state (typically its hash) and the merged policy snapshot [2][3]. Every active ResolutionVerifier contributes its policy to this merged map [2]. 2. Cache Lookup: Before running full verification, the system performs a lookup using this identity [2][4]. If a previous, successful verification run matches the current lockfile and satisfies a policy at least as strict as the current one, the system can skip the full verification pass [2][4]. 3. Implementation: In the Rust implementation ( Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
for url in \
'https://raw.githubusercontent.com/pnpm/pnpm/a751c7f2/pacquet/crates/lockfile-verification/src/cache.rs' \
'https://raw.githubusercontent.com/pnpm/pnpm/a751c7f2/pacquet/crates/lockfile-verification/src/verify_lockfile_resolutions.rs' \
'https://raw.githubusercontent.com/pnpm/pnpm/a751c7f2/installing/deps-installer/src/install/verifyLockfileResolutions.ts'
do
echo "--- $url"
curl -fsSL "$url" | sed -n '1,260p'
doneRepository: pnpm/action-setup Length of output: 34144 🏁 Script executed: #!/bin/bash
set -euo pipefail
url='https://raw.githubusercontent.com/pnpm/pnpm/a751c7f2/pacquet/crates/lockfile-verification/src/cache.rs'
curl -fsSL "$url" | rg -n -C 12 \
'every_verifier_trusts_cached_run|can_trust_past_check|merge_policies|policy|strict|minimumReleaseAge|trustPolicy'Repository: pnpm/action-setup Length of output: 8457 Include the effective verification-policy configuration in the cache key. pnpm stores the policy snapshot in Add a stable fingerprint of the effective policy configuration to the key, and update 🤖 Prompt for AI Agents |
||
| saveState(PATH_STATE, cacheFilePath) | ||
| saveState(KEY_STATE, key) | ||
| debug(`Lockfile verification cache path is ${cacheFilePath}, key is ${key}`) | ||
|
|
||
| const restoredKey = await restoreCache([cacheFilePath], key) | ||
| if (!restoredKey) { | ||
| info('Lockfile verification cache is not found') | ||
| return | ||
| } | ||
|
|
||
| saveState(RESTORED_STATE, 'true') | ||
| info(`Lockfile verification cache restored from key: ${restoredKey}`) | ||
| } catch (error) { | ||
| // The gate only costs time, never correctness — a job that cannot reuse | ||
| // a past verdict re-verifies and moves on. | ||
| warning(`Failed to restore the lockfile verification cache: ${(error as Error).message}`) | ||
| } | ||
| } | ||
|
|
||
| export async function saveVerificationCache(): Promise<void> { | ||
| if (getState(RESTORED_STATE) === 'true') return | ||
|
greptile-apps[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| const cacheFilePath = getState(PATH_STATE) | ||
| const key = getState(KEY_STATE) | ||
| if (!cacheFilePath || !key || !existsSync(cacheFilePath)) return | ||
|
|
||
| try { | ||
| const cacheId = await saveCache([cacheFilePath], key) | ||
| if (cacheId === -1) return | ||
| info(`Lockfile verification cache saved with the key: ${key}`) | ||
| } catch (error) { | ||
| warning(`Failed to save the lockfile verification cache: ${(error as Error).message}`) | ||
| } | ||
| } | ||
|
|
||
| async function getPnpmCacheDirectory(): Promise<string> { | ||
| const { stdout } = await getExecOutput('pnpm config get cacheDir', undefined, { | ||
| silent: true, | ||
| ignoreReturnCode: true, | ||
| }) | ||
| const configured = stdout.trim() | ||
| // `pnpm config get` reports settings, not defaults: an unset `cacheDir` | ||
| // prints `undefined` and the default has to be derived here. | ||
| if (configured && configured !== 'undefined') { | ||
| return removeWindowsExtendedPathPrefix(configured) | ||
| } | ||
| return defaultPnpmCacheDirectory() | ||
| } | ||
|
|
||
| /** Mirrors pnpm's own `cacheDir` default. */ | ||
| function defaultPnpmCacheDirectory(): string { | ||
| const { XDG_CACHE_HOME, LOCALAPPDATA } = process.env | ||
| if (XDG_CACHE_HOME) return path.join(XDG_CACHE_HOME, 'pnpm') | ||
|
|
||
| const homeDir = os.homedir() | ||
| switch (process.platform) { | ||
| case 'darwin': | ||
| return path.join(homeDir, 'Library', 'Caches', 'pnpm') | ||
| case 'win32': | ||
| return LOCALAPPDATA ? path.join(LOCALAPPDATA, 'pnpm-cache') : path.join(homeDir, '.pnpm-cache') | ||
| default: | ||
| return path.join(homeDir, '.cache', 'pnpm') | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| /** | ||
| * pnpm may report an extended-length path on Windows. The `?` in that prefix | ||
| * is interpreted as a wildcard by `@actions/cache`, which rejects it as a glob | ||
| * in the root segment. Cache APIs do not need the extended-length form, so | ||
| * convert it back to a regular drive or UNC path. | ||
| */ | ||
| export function removeWindowsExtendedPathPrefix(cachePath: string): string { | ||
| const extendedPathPrefix = '\\\\?\\' | ||
| if (!cachePath.startsWith(extendedPathPrefix)) return cachePath | ||
|
|
||
| const pathWithoutPrefix = cachePath.slice(extendedPathPrefix.length) | ||
| const uncPrefix = 'UNC\\' | ||
| if (pathWithoutPrefix.toUpperCase().startsWith(uncPrefix)) { | ||
| return `\\\\${pathWithoutPrefix.slice(uncPrefix.length)}` | ||
| } | ||
| return pathWithoutPrefix | ||
| } | ||
|
|
||
| export default removeWindowsExtendedPathPrefix |
Uh oh!
There was an error while loading. Please reload this page.