Repository navigation
Recovery: roll back an install that does not come up, and a hand-back runbook - #66
Merged
Merged
Conversation
Fails until the scripts check health: the unhealthy install, closed port, old log line, no-previous, and both rollback-check cases. The healthy and truncated-log cases already pass. Issue: #59
After the restart, install.sh and rollback.sh wait up to T3FAC_HEALTH_WAIT seconds (90) for the port and a fresh factory started or no factory config log line. A failed install runs rollback.sh; a failed rollback exits 1 naming both versions. Issue: #59
…il is 50 lines Review findings: nothing exercised the "no factory config" line, and the tail assertion could not tell 50 lines from the whole log. Both already pass. Issue: #59
rawphp
marked this pull request as ready for review
October 16, 2026 06:45
Author
|
Factory gate: passed (attempt 1, run by the factory in a fresh checkout of the pull request's head
|
…nfig notes - rollback.sh restores previous when current is missing, naming only the restored version on failure - tests: a server whose log line or port comes up after the first check is kept - README: a factory config error fails the health check; hand-back moves config.yml aside when the repo is the only one - plist and lib/t3fac.sh point at each other for the port and log path
Author
|
Factory gate: passed again (attempt 2, run by the factory in a fresh checkout of the pull request's head
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #59
Contract
restart_job,install.shwaits up toT3FAC_HEALTH_WAITs (90) for a TCP connect on the port and afactory startedorno factory configline appended to the log after the restart (size noted before; a smaller file is read from offset 0). Either missing: last 50 log lines,rollback.sh, exit 1 withinstall of <version> failed its health check; rolled back to <previous>; no previous: exit 1 and says so.rollback.shruns the same check on the restored version and exits 1 naming both versions. README gains a Recovery section.vp test run apps/server/src/factory/installScripts.test.ts→ pass (31 tests)Acceptance criteria
install.shwaits up toT3FAC_HEALTH_WAIT(default 90) for the port and a fresh log line —wait_healthyinfactory/scripts/lib/t3fac.sh; testskeeps a healthy install,rolls back an install whose port does not accept connections,rolls back an install whose server never logs that it started,keeps an install whose server logs that it has no factory configrollback.sh, exit 1 with the message —rolls back an install whose server never logs that it started(asserts line 11 shown, line 10 cut)exits 1 and says so when an unhealthy first install has nothing to roll back torollback.shchecks the restored version and names both versions —rollback.sh exits 1 naming both versions when the restored version is unhealthy too,install.sh reports when the rollback fails its health check as wellfactory startedline does not count; truncated log reads from offset 0 —does not count a factory started line that was there before the restart,counts a line written after the log was truncated to a size below the noted onelaunchctl, a stub listener and a written log —apps/server/src/factory/installScripts.test.ts, describeinstall.sh and rollback.sh health checkfactory/README.mdsection## RecoveryAssumption: the issue says tests in
factory/scripts/; the existing script tests live inapps/server/src/factory/installScripts.test.ts, which is what the gate runs, so the new cases are there.T3FAC_PORTis a test-only override for the health check port; the plist still serves 3876.Gate
factory/scripts/check.sh→ exit 0 (seams, 33 test files / 1138 tests, server and web typecheck)Models
Review
no factory configline)no factory configline; the log-tail assertion now proves the 50-line bound (both mutants re-run and killed by the orchestrator)Lessons
Push the branch by
HEAD, not by the name in the launch snapshot. Symptom:git push -u origin <branch>fails with "src refspec does not match any". Cause: the git status snapshot at session start named a different branch (t3/8c8fa6d2) from the worktree's real one (t3/factory-issue-59). Default: readgit branch --show-currentfirst, or pushHEAD, and never pass the snapshot's name.Read skill references with absolute paths, never
cdinto the skill directory. Symptom: aftercd <skill>/references && cat ...the harness reports the primary working directory moved off the worktree. Cause: the session's cwd follows the shell. Default:cat /abs/path/filefrom the worktree; every command keeps starting withcd <worktree> &&.Pass
--repo <owner/repo>togh pr createin a fork. Symptom: the draft PR opens on the fork's parent repository (herepingdotgg/t3code#20179, closed at once). Cause:ghresolves a fork checkout's default repository to the parent unlessgh repo set-defaultwas run. Default: always pass--repo(the repo the issue lives in) and--head <branch>with the worktree's own branch name, then confirm the URL's owner before linking.Commits
520aaab0btest: install and rollback wait for the server to come up— Make slice (red: 6 failed)b33d8dfcdfeat(factory): roll back an install that fails its health check— Make slice (green)626a4971bdocs(factory): recovery section— Make slice (docs, no red phase)51fe118b3test: a server that logs no factory config is healthy, and the log tail is 50 lines— Check cycle 1 (tests for behavior that already worked, no red phase)Files
log,port(T3FAC_PORT),log_beforenoted inrestart_job, newwait_healthyLAUNCHCTL_HEALTHY/LAUNCHCTL_HEALTHY_LINE/LAUNCHCTL_TRUNCATE_LOGin the launchctl stub, nine health-check casesIsolate: the factory's worktree was behind
origin/main(ancestor, clean tree); fast-forwarded withgit merge --ff-only origin/mainbefore provisioning. Repo labels: all seven existed; none created.