Skip to content

Run commands with a cleared environment, a time limit and a bounded output tail - #33

Merged
tomkaczocha merged 9 commits into
mainfrom
t3/factory-issue-8
Oct 15, 2026
Merged

tomkaczocha merged 9 commits into
mainfrom
t3/factory-issue-8

Conversation

@rawphp

@rawphp rawphp commented Oct 15, 2026

Copy link
Copy Markdown

Closes #8

Contract

  • done: apps/server/src/factory/process.ts exports run({ command, args, shell, cwd, env, timeoutMs, tailBytes }) returning an Effect of { exitCode, timedOut, tail } over node:child_process; the child sees only PATH, HOME, USER, LOGNAME, SHELL, LANG, LC_*, TMPDIR, TERM from the server plus the passed env; it runs in its own process group; on timeout the group gets SIGTERM, then SIGKILL once the group is empty or 10 s have passed, with timedOut: true; stdout and stderr share one pipe and only the last tailBytes (default 16 KiB) are kept; design.md open question 2 is answered.
  • out: gate.ts, the gh runner, the ledger, a cancel flag, the leftover-group file, Windows.
  • risk: none
  • test: vp test run apps/server/src/factory → pass (22 tests, 4 files)

Acceptance criteria

  • run(...) returns { exitCode, timedOut, tail }, in apps/server/src/factory/process.ts because it uses node:child_process — process.ts:117, every test in process.test.ts
  • Child env is the allowlist plus env — tests "does not pass variables outside the allowlist", "passes PATH, LC_* and the env argument through" (includes a passed HOME overriding the inherited one)
  • Own process group; timeout → SIGTERM, SIGKILL 10 s later, timedOut: true — detached: true and kill(-pid) in process.ts; tests "a timeout kills the whole process group, grandchild included", "escalates to SIGKILL when SIGTERM is ignored", "gives a TERM-trapping grandchild its grace before SIGKILL" (grace is termGraceMs, default 10 000, exposed so the test runs in under a second)
  • stdout and stderr interleaved, last tailBytes only (default 16 KiB) — one pipe via a /bin/sh wrapper with exec 2>&1; Tail keeps at most about twice the cap; tests "interleaves stdout and stderr in order", "merges stderr into the tail without a shell", "keeps only the last tailBytes of a 1 MB output"
  • Tests: outside var unseen; timeout kills a grandchild; 1 MB tail; exit code passes through — the tests above plus "passes the exit code through"
  • design.md open question 2 answered — factory/docs/design.md item 2 and the code-layout tree

Decisions the issue left open

  • ProcessRunner was checked against apps/server/src/processRunner.ts and Effect's Node spawner: it does detach the child into its own group and SIGTERMs the group on interrupt, but it always extends the parent environment (extendEnv is true whenever env is set), reads stdout and stderr on separate pipes, drops partial output on timeout and never escalates to SIGKILL. So process.ts is outside factory/t3/ and no seam was added. factory/scripts/check-seams.sh exits 0.
  • exitCode is number | null (null when the process died by a signal, including every timeout), Node's own shape, rather than softfac's -1. The gate module that writes the ledger can map it.
  • shell: true runs command as a /bin/sh -c script and passes args as its positional parameters ($0, $1, …). Argv mode goes through the same wrapper with exec "$0" "$@", so both modes share one pipe.
  • Default timeoutMs is 30 minutes (softfac's gate default). POSIX only, as softfac is.
  • After the leader exits normally, the group is SIGKILLed at once (softfac's "nothing the command started may outlive it"); the SIGTERM grace applies to the timeout path only, as the issue says.

Gate

  • vp test run apps/server/src/factory → exit 0 (.factory/gate-1.log)
  • vp run --filter t3 typecheck → exit 0 (.factory/gate-2.log; only pre-existing suggestion lines in other files)

Models

  • maker: claude-sonnet-5-5 (via host; .dw/config.yml has no models block, so Tom's roster applied)
  • checker: claude-opus-5-5 (via host)

Review

  • cycles: 2
  • fixed: correctness-1 (in shell mode the wrapper sh died on SIGTERM and the group was SIGKILLed at once, so the real command got no grace; now the grace waits for the group to be empty), reliability-1 (the leader pid is signalled alongside the group), reliability-2 and -3 (spawn and kill now sit in acquireUseRelease, so an interrupt always kills the group and destroys the pipe), testing-1, -2, -4 (tests added for env override, post-exit stray cleanup timing, argv-mode stderr merge; the two surviving mutants were re-applied by hand after the fix and both are killed)
  • leftovers: testing-3 (nit): the "at most twice the cap in memory" bound of Tail is not observable through run(); testing it would mean exporting Tail. Left untested. reliability-1 and the interrupt path have no deterministic test (a detached session leader cannot leave its group without a cooperating descendant; an interrupt race cannot be forced).
  • mutation review: 12 mutants, 9 killed on the first pass, 3 survivors, 2 of them now killed, 1 accepted (above).

Lessons

  • After a test-only fix cycle driven by mutation survivors, re-apply the survivors yourself. Symptom: the mutating review runs once per issue, so the tests added to kill its survivors are never shown to kill them. Cause: the testing budget forbids a second mutating pass, and the maker only runs the suite green. Default: the orchestrator applies each surviving mutant in one command (edit, run the filtered test, git checkout the file) on the fixed tip and records killed or survived on the receipt; no second review needed.

Notes

  • The worktree was cut one merge behind origin/main (issue Factory skeleton: an empty FactoryLayer wired in through one seam #1 had just landed); fast-forwarded with git merge --ff-only origin/main before provisioning (field lesson 18).
  • Labels created this run: none (all seven factory:* labels already existed).
  • Red/green: the test: commits 8b4567307 (module absent, import fails) and ed6472569 (reported red with expected '' to include 'caught') precede their fixes. dd6c4467c has no red phase by design: it pins behaviour that already worked.

Commits

  • 8b4567307 test(factory): pin run() env allowlist, process-group kill and output tail — Make slice (red)
  • 5a1d95e79 feat(factory): run commands with a cleared env, a time limit and an output tail — Make slice
  • b60ba307c fix(factory): process tests read time through Effect's Clock — Make slice (typecheck diagnostics)
  • f49f63b9b docs(factory): answer open question 2, process.ts uses node:child_process — Make slice
  • ed6472569 test(factory): a TERM-trapping grandchild gets its grace before SIGKILL — Check cycle 1 (red)
  • cbfc11ced fix(factory): give the whole group its SIGTERM grace, signal the leader too, kill on interrupt — Check cycle 1
  • dd6c4467c test(factory): pin env override, post-exit stray cleanup and argv stderr merge — Check cycle 2

Files

  • apps/server/src/factory/process.ts — run(), env allowlist, Tail, process-group lifecycle
  • apps/server/src/factory/process.test.ts — the tests above
  • factory/docs/design.md — open question 2 answered; process.ts moved out of t3/ in the layout tree

… tail

Red: apps/server/src/factory/process.ts does not exist yet.

Issue: #8
Red: the wrapper sh dies on SIGTERM and the group is SIGKILLed at once,
so the grandchild's marker never reaches the tail.

Issue: #8
…er too, kill on interrupt

After SIGTERM, wait up to termGraceMs for the leader to exit and the
group to be empty before SIGKILL; in shell mode the wrapper sh died
first and the real command got no grace. Also signal the leader pid
alongside the group (softfac kill_all), and spawn inside
acquireUseRelease so an interrupt always kills the group and destroys
the pipe. The leader-left-group and interrupt cases have no
deterministic test.

Issue: #8
…err merge

Three mutation survivors from review. No red phase: the behaviour already
works; these tests pin it.

Issue: #8
@rawphp
rawphp marked this pull request as ready for review October 15, 2026 11:02
@rawphp

rawphp commented Oct 15, 2026

Copy link
Copy Markdown
Author

Factory gate: passed (attempt 1, run by the factory in a fresh checkout of the pull request's head dd6c446 merged with the base branch tip f1dba9a)

Command Exit Time
vp test run apps/server/src/factory 0 2.6 s
vp run --filter t3 typecheck 0 6.1 s

…rrupt and drain tests

- spawn's synchronous throws (a NUL byte in env or args) now fail with
  RunSpawnError instead of dying
- RunSpawnError keeps the command only in cause; its attributes are shell and cwd
- the env allowlist and defaults are no longer exported (knip)
- RunResult.exitCode doc: a timed-out command can still exit 0, check timedOut first
- the post-run kill comment says a process that left the group escapes
- tests: an interrupted run kills the group, a stray outside the group does not
  hold run past the drain wait, a tail cut inside a UTF-8 character
- design.md Shape diagram: gate commands go through node:child_process in
  factory/process.ts, not ProcessRunner
@rawphp

rawphp commented Oct 15, 2026

Copy link
Copy Markdown
Author

Factory gate: passed again (attempt 2, run by the factory in a fresh checkout of the pull request's head 4d016bb merged with the base branch tip b679ddd)

Command Exit Time
factory/scripts/check.sh 0 15.1 s

Callers depend on the FactoryProcess tag, so their tests can swap in a
fake layer instead of starting real processes. The two process-group
tests now use withTempDir and waitForDeath, so a failed assertion no
longer leaks the temp dir, and their wall-clock bound is 5 s instead of
1 s; the grandchild death check still proves the kill.
@tomkaczocha
tomkaczocha merged commit 8a65faf into main Oct 15, 2026
@rawphp
rawphp deleted the t3/factory-issue-8 branch October 28, 2026 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run commands with a cleared environment, a time limit and a bounded output tail

2 participants