Problem
The bounded production image canary can boot its blue replacement successfully and then lose that replacement before cutover. The generic replacement-cleanup sweep may select a pre-cutover blue fence while its lifecycle job is still active, and database pool/advisory-lock waits are not bounded at the PostgreSQL boundary. A JavaScript watchdog can therefore abandon its awaiter while pooled database work continues, leaving the job failed and blocking safe retries.
The latest production canary reproduced this as a healthy blue boot followed by cleanup of the blue VPN node and a failed lifecycle job. The serving generation remained intact, but Notes and Calendar could not reach the new immutable image.
Acceptance criteria
- A background cleanup sweep must not retire replacement resources owned by a pending or in-progress exclusive lifecycle job.
- Pre-cutover cleanup candidates must observe a database-time recovery grace; post-cutover old resources remain immediately eligible.
- The inner lifecycle/row-locked cleanup path must recheck active jobs and grace after outer selection.
- Replacement workloads count toward node capacity and remain protected from orphan cleanup until durable retirement.
- Every Eliza lifecycle/admin advisory-lock transaction must have bounded lock and statement waits, and PostgreSQL pool checkout must be finite.
- Authoritative delete semantics and existing idempotency/CAS guarantees must remain intact.
- Prove the deadline with two real PostgreSQL sessions and prove the cleanup race, capacity ownership, recovery, and retry paths with deterministic tests.
- Deploy the exact reviewed fix, run one bounded canary, and verify Notes/Calendar/Cerebras/restart persistence before widening.
Evidence plan
- Backend logs: sanitized canary/reconciler results, exact deployed SHA, and bounded cleanup stage/elapsed logs.
- Database/domain artifacts: lifecycle job result, cleanup fence/capacity state, and post-restart Simple Views state.
- Tests: real PostgreSQL contention plus PGlite lifecycle, cleanup TOCTOU, capacity/orphan, enqueue/delete, and recovery suites.
- UI screenshots/video: N/A — backend control-plane fix; physical LP3 proof is captured after the canary is healthy.
- Real model trajectory: N/A for the fix itself; the final device acceptance includes a real Cerebras reply.
Related launch command center: #16895. Related timestamp-precision repair: #17149.
Problem
The bounded production image canary can boot its blue replacement successfully and then lose that replacement before cutover. The generic replacement-cleanup sweep may select a pre-cutover blue fence while its lifecycle job is still active, and database pool/advisory-lock waits are not bounded at the PostgreSQL boundary. A JavaScript watchdog can therefore abandon its awaiter while pooled database work continues, leaving the job failed and blocking safe retries.
The latest production canary reproduced this as a healthy blue boot followed by cleanup of the blue VPN node and a failed lifecycle job. The serving generation remained intact, but Notes and Calendar could not reach the new immutable image.
Acceptance criteria
Evidence plan
Related launch command center: #16895. Related timestamp-precision repair: #17149.