Workflow Reliability

Workflow Reliability

Making the TDD pipeline trustworthy through structured output, path elimination, and postmortem-driven fixes

---

The Reliability Stack

Each fix feeds the next — instrumentation reveals issues, structured output eliminates parsing failures, WorkspaceContext eliminates path bugs, and postmortems close remaining gaps.

graph LR
    I["#774: LLM Instrumentation<br/>See every call"] --> S["#775: Structured Output<br/>Eliminate regex parsing"]
    S --> W["#838: WorkspaceContext<br/>Eliminate path prop-drilling"]
    W --> P["#842: TDD Postmortem<br/>Identify remaining gaps"]
    P --> F["#840, #789, #826<br/>Fix specific failures"]

Loading

---

Structured Output Migration (#775)

The Problem

LLM output was parsed with regex. Regex is fragile — one extra newline, one wrong heading format, and the parser fails silently or extracts garbage.

The Solution

Migrate all LLM calls to use response_schema (Anthropic API) or --json-schema (Claude CLI). The LLM returns structured JSON conforming to a schema. No regex.

Before	After
re.search(r'## Implementation\n(.*?)##', output)	response["implementation"]
Silent failure on format change	Schema validation error with clear message
Regex maintained per output format	Schema defined once, used everywhere

Impact: Eliminated an entire class of "LLM output parsing" failures from the pipeline.

---

WorkspaceContext Refactor (#838)

The Problem

Every function in the implementation pipeline needed 4–6 path parameters:

def generate_file(repo_root, worktree_path, issue_dir,
                  lld_path, spec_path, main_repo_path):

This "prop drilling" made the code fragile — wrong paths passed to wrong functions, missing parameters at call sites, inconsistent path formats.

The Solution

WorkspaceContext dataclass bundles all path-related state:

@dataclass
class WorkspaceContext:
    repo_root: Path
    worktree_path: Path
    issue_dir: Path
    lld_path: Path
    spec_path: Path
    main_repo_path: Path

Functions receive one ctx parameter instead of 6:

def generate_file(ctx: WorkspaceContext, file_spec: FileSpec):

Impact: Eliminated path-related bugs where wrong paths were passed to wrong functions.

---

TDD Postmortem (#842)

After the structured output migration (#775), the TDD workflow required massive manual fixup. The postmortem identified root causes:

Root Cause	Issue	Fix
Impl spec hallucinations — references nonexistent files	#840	Validate file paths exist before TDD
scaffold_tests doesn't set generated_tests flag	#789	Fix scaffold state initialization
run_pytest passes --cov without pytest-cov installed	#789	Check for pytest-cov before adding flag
LangChain Pydantic warnings crash stderr parsing	#826	Filter warnings before error checking

---

Pydantic V1 Deprecation (#824, #826)

The Problem

LangChain uses Pydantic V1 internally. Pydantic V2 emits deprecation warnings to stderr. The TDD workflow checks stderr for errors — Pydantic warnings were misinterpreted as failures.

The Fix

Filter LangChain Pydantic warnings from stderr before error checking (#826). The filter is surgical — only removes known Pydantic deprecation patterns, preserving real errors.

---

LLM Call Instrumentation (#774)

Every LLM call now logs:

Input	Output
Provider (claude/anthropic/gemini)	Success/failure
Model requested	Model actually used
System prompt length	Input tokens
Content length	Output tokens
JSON schema (if any)	Cache read/creation tokens
	Cost in USD
	Duration in ms

This enables postmortem analysis: "Which call was expensive? Which prompt was too large? Where did caching fail?"

---

Related

• Prompt Economics — Cost optimization that depends on reliable execution
• Implementation Workflow — The pipeline these fixes stabilize
• The WinError 206 Hack — Platform reliability
• Worktree Isolation — Execution sandboxing