Skip to content

Recovery: roll back an install that does not come up, and a hand-back runbook - #66

Merged
tomkaczocha merged 5 commits into
mainfrom
t3/factory-issue-59
Oct 16, 2026
Merged

tomkaczocha merged 5 commits into
mainfrom
t3/factory-issue-59

Conversation

@rawphp

@rawphp rawphp commented Oct 16, 2026 •

Copy link
Copy Markdown

Closes #59

Contract

  • done: after restart_job, install.sh waits up to T3FAC_HEALTH_WAIT s (90) for a TCP connect on the port and a factory started or no factory config line appended to the log after the restart (size noted before; a smaller file is read from offset 0). Either missing: last 50 log lines, rollback.sh, exit 1 with install of <version> failed its health check; rolled back to <previous>; no previous: exit 1 and says so. rollback.sh runs the same check on the restored version and exits 1 naming both versions. README gains a Recovery section.
  • out: log rotation (Rotate the t3fac log so it stops growing #60), handover.md (Handover: the fork's factory is ready to build this repo #21), plist or port changes.
  • risk: none
  • test: vp test run apps/server/src/factory/installScripts.test.ts → pass (31 tests)

Acceptance criteria

  • install.sh waits up to T3FAC_HEALTH_WAIT (default 90) for the port and a fresh log line — wait_healthy in factory/scripts/lib/t3fac.sh; tests keeps a healthy install, rolls back an install whose port does not accept connections, rolls back an install whose server never logs that it started, keeps an install whose server logs that it has no factory config
  • Missing: last 50 log lines, rollback.sh, exit 1 with the message — rolls back an install whose server never logs that it started (asserts line 11 shown, line 10 cut)
  • No previous version: exit 1 and says so — exits 1 and says so when an unhealthy first install has nothing to roll back to
  • rollback.sh checks the restored version and names both versions — rollback.sh exits 1 naming both versions when the restored version is unhealthy too, install.sh reports when the rollback fails its health check as well
  • Old factory started line does not count; truncated log reads from offset 0 — does not count a factory started line that was there before the restart, counts a line written after the log was truncated to a size below the noted one
  • Four cases with a stub launchctl, a stub listener and a written log — apps/server/src/factory/installScripts.test.ts, describe install.sh and rollback.sh health check
  • README "Recovery" section: rolling back by hand, hand-back to the Rust factory, moving back, telling the fork is down — verified factory/README.md section ## Recovery

Assumption: the issue says tests in factory/scripts/; the existing script tests live in apps/server/src/factory/installScripts.test.ts, which is what the gate runs, so the new cases are there. T3FAC_PORT is a test-only override for the health check port; the plist still serves 3876.

Gate

  • factory/scripts/check.sh → exit 0 (seams, 33 test files / 1138 tests, server and web typecheck)

Models

  • maker: claude-sonnet-5-5 (host subagent); the Check-fix commit (tests only) was made by the orchestrating session
  • checker: claude-opus-5-5

Review

  • cycles: 1
  • roster: correctness (approve), reliability (approve), testing (12/12 named mutants killed; one extra survivor: no test for the no factory config line)
  • fixed: test for the no factory config line; the log-tail assertion now proves the 50-line bound (both mutants re-run and killed by the orchestrator)
  • leftovers: none

Lessons

  • Push the branch by HEAD, not by the name in the launch snapshot. Symptom: git push -u origin <branch> fails with "src refspec does not match any". Cause: the git status snapshot at session start named a different branch (t3/8c8fa6d2) from the worktree's real one (t3/factory-issue-59). Default: read git branch --show-current first, or push HEAD, and never pass the snapshot's name.

  • Read skill references with absolute paths, never cd into the skill directory. Symptom: after cd <skill>/references && cat ... the harness reports the primary working directory moved off the worktree. Cause: the session's cwd follows the shell. Default: cat /abs/path/file from the worktree; every command keeps starting with cd <worktree> &&.

  • Pass --repo <owner/repo> to gh pr create in a fork. Symptom: the draft PR opens on the fork's parent repository (here pingdotgg/t3code#20179, closed at once). Cause: gh resolves a fork checkout's default repository to the parent unless gh repo set-default was run. Default: always pass --repo (the repo the issue lives in) and --head <branch> with the worktree's own branch name, then confirm the URL's owner before linking.

Commits

  • 520aaab0b test: install and rollback wait for the server to come up — Make slice (red: 6 failed)
  • b33d8dfcd feat(factory): roll back an install that fails its health check — Make slice (green)
  • 626a4971b docs(factory): recovery section — Make slice (docs, no red phase)
  • 51fe118b3 test: a server that logs no factory config is healthy, and the log tail is 50 lines — Check cycle 1 (tests for behavior that already worked, no red phase)

Files

  • factory/scripts/lib/t3fac.sh — log, port (T3FAC_PORT), log_before noted in restart_job, new wait_healthy
  • factory/scripts/install.sh — health check after the restart; rollback or say there is nothing to roll back to; exit 1
  • factory/scripts/rollback.sh — health check on the restored version; exit 1 naming both versions
  • apps/server/src/factory/installScripts.test.ts — shared stub listener, LAUNCHCTL_HEALTHY / LAUNCHCTL_HEALTHY_LINE / LAUNCHCTL_TRUNCATE_LOG in the launchctl stub, nine health-check cases
  • factory/README.md — Recovery section; install and rollback one-liners mention the health check

Isolate: the factory's worktree was behind origin/main (ancestor, clean tree); fast-forwarded with git merge --ff-only origin/main before provisioning. Repo labels: all seven existed; none created.

Fails until the scripts check health: the unhealthy install, closed port, old log line, no-previous, and both rollback-check cases. The healthy and truncated-log cases already pass.

Issue: #59
After the restart, install.sh and rollback.sh wait up to T3FAC_HEALTH_WAIT seconds (90) for the port and a fresh factory started or no factory config log line. A failed install runs rollback.sh; a failed rollback exits 1 naming both versions.

Issue: #59
…il is 50 lines

Review findings: nothing exercised the "no factory config" line, and the
tail assertion could not tell 50 lines from the whole log. Both already pass.

Issue: #59
@rawphp
rawphp marked this pull request as ready for review October 16, 2026 06:45
@rawphp

rawphp commented Oct 16, 2026

Copy link
Copy Markdown
Author

Factory gate: passed (attempt 1, run by the factory in a fresh checkout of the pull request's head 51fe118 merged with the base branch tip ff23fa2)

Command Exit Time
factory/scripts/check.sh 0 50.0 s

…nfig notes

- rollback.sh restores previous when current is missing, naming only the restored version on failure
- tests: a server whose log line or port comes up after the first check is kept
- README: a factory config error fails the health check; hand-back moves config.yml aside when the repo is the only one
- plist and lib/t3fac.sh point at each other for the port and log path
@rawphp

rawphp commented Oct 16, 2026

Copy link
Copy Markdown
Author

Factory gate: passed again (attempt 2, run by the factory in a fresh checkout of the pull request's head 9ed810d merged with the base branch tip ff23fa2)

Command Exit Time
factory/scripts/check.sh 0 94.1 s

@tomkaczocha
tomkaczocha merged commit eb8d983 into main Oct 16, 2026
@rawphp
rawphp deleted the t3/factory-issue-59 branch October 28, 2026 02:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recovery: roll back an install that does not come up, and a hand-back runbook

2 participants