diff --git a/.opencode/opencode.jsonc b/.opencode/opencode.jsonc index 53d0b09ed..3be78d098 100644 --- a/.opencode/opencode.jsonc +++ b/.opencode/opencode.jsonc @@ -35,6 +35,8 @@ // "context_history": false, // "thread_park": false, // "thread_list": false, + // "classifier_threads": false, + // "distill_threads": false, }, "agent": { // classifier is enabled by default — used by classifier_threads and distill_threads tools diff --git a/AGENTS.md b/AGENTS.md index 3e89b87fd..67b3758c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -125,7 +125,13 @@ Use `context_edit` to keep the conversation context clean and accurate: - **Replace** incorrect statements with corrections. - **Externalize** verbose tool output into CAS, leaving a compact summary inline. Use `context_deref` to retrieve the full content later if needed. - **Annotate** parts with notes for future reference. -- Do not hide errors the user should see. Do not edit user messages. Do not edit the last 2 turns. +- **Mark** parts with lifecycle hints for automatic cleanup: + - `discardable` (auto-hide after 3 turns) — for failed commands, dead-end explorations + - `ephemeral` (auto-externalize after 5 turns) — for verbose output where only the conclusion matters + - `side-thread` — candidate for parking when `/focus` runs + - `pinned` — never auto-discard +- Do not hide errors the user should see. Do not edit the last 2 turns. +- Target parts using `query` (content search), `toolName`, or `nthFromEnd`. Avoid guessing raw part/message IDs. Use `thread_park` to defer off-topic findings: @@ -133,6 +139,8 @@ Use `thread_park` to defer off-topic findings: - Include a clear title, description, priority, category, and related files. - Use `thread_list` to check existing threads before parking duplicates. +Use `classifier_threads` to analyze the conversation by topic, and `distill_threads` to classify + park side threads in one step. + Use `context_history` to navigate the edit DAG: - `log` to review what was edited and when. diff --git a/DO_NEXT.md b/DO_NEXT.md index 06704e37d..acda40096 100644 --- a/DO_NEXT.md +++ b/DO_NEXT.md @@ -1,34 +1,25 @@ # Frankencode — Do Next -## All 4 Phases Complete +## Implemented -### Implementation done: -- [x] Phase 1: CAS (SQLite) + Part Editing (EditMeta, filterEdited, context_edit, context_deref) -- [x] Phase 2: Conversation Graph (edit_graph DAG, context_history with log/tree/checkout/fork) -- [x] Phase 3: Focus Agent + Side Threads (side_thread table, thread_park, thread_list, focus agent, objective tracker) -- [x] Phase 4: Integration (system prompt injection, plugin hooks) +- [x] CAS (SQLite) + Part Editing (EditMeta, LifecycleMeta, filterEdited, context_edit, context_deref) +- [x] Conversation Graph (edit_graph DAG, context_history with log/tree/checkout/fork) +- [x] Focus Agent + Side Threads (side_thread table, thread_park, thread_list, classifier, focus agents) +- [x] Integration (system prompt injection, plugin hooks, lifecycle sweeper) +- [x] v2: query/toolName targeting, classifier_threads, distill_threads, /btw, /focus, /reset-context +- [x] Config-based control (no feature toggles) +- [x] Documentation (README, docs/context-editing, docs/schema, docs/agents, AGENTS.md) -### Remaining (polish & testing): -- [ ] Write unit tests for CAS (store, get, dedup via ON CONFLICT) -- [ ] Write unit tests for filterEdited (hidden parts stripped, superseded parts stripped, empty messages dropped) -- [ ] Write unit tests for EditGraph (commit chain, log walk, checkout restore, fork branch) -- [ ] Write unit tests for SideThread CRUD -- [ ] Write unit tests for ContextEdit validation (ownership, budget, recency) -- [ ] Manual end-to-end test: enable `OPENCODE_EXPERIMENTAL_FOCUS_AGENT=1`, run a session, verify: - - context_edit(hide) removes part from next LLM call - - context_edit(externalize) replaces with summary, context_deref retrieves original - - context_history(log) shows edit chain - - thread_park creates a project-level thread - - Focus agent runs post-turn and parks side threads - - System prompt includes focus status + thread summary +## Next -### Future enhancements (from design docs): -- [ ] Threshold refocus mode (configurable %, proactive context rewrite) -- [ ] Background curator mode (between-turn cleanup) -- [ ] Pin & decay scoring (relevance decay per turn) -- [ ] Handoff artifacts (cross-session persistence) -- [ ] thread_investigate (spawn subagent with pre-loaded CAS context) -- [ ] thread_promote (swap side thread into main objective) -- [ ] TUI rendering (toggle edit indicators, sidebar thread panel) -- [ ] Focus intensity levels (relaxed/moderate/strict) -- [ ] CAS garbage collection (orphan cleanup) +- [ ] Unit tests for CAS (store, get, dedup via ON CONFLICT) +- [ ] Unit tests for filterEdited (hidden parts stripped, empty messages dropped) +- [ ] Unit tests for EditGraph (commit chain, log walk, checkout restore) +- [ ] Unit tests for SideThread CRUD +- [ ] Unit tests for ContextEdit validation (ownership, budget, recency, privileged agents) +- [ ] Unit tests for lifecycle sweeper (discardable auto-hide, ephemeral auto-externalize) +- [ ] Test classifier_threads + distill_threads with a real session +- [ ] Test /btw command (verify it forks, doesn't pollute main thread) +- [ ] Explore: make /btw use Session.fork() for true message-level isolation +- [ ] Explore: CAS garbage collection (orphan cleanup, size limits) +- [ ] Explore: TUI rendering of edit indicators (hidden/replaced/annotated parts) diff --git a/PLAN.md b/PLAN.md index 892cb7e16..e019f43b8 100644 --- a/PLAN.md +++ b/PLAN.md @@ -25,29 +25,34 @@ OpenCode agents accumulate stale tool output, wrong assumptions, and off-topic e **New file:** `packages/opencode/src/cas/index.ts` The CAS lives in SQLite (same DB as everything else). This gives us: + - Atomic transactions with part updates (CAS write + part edit in one tx) - Queryable (find all CAS entries for a session, GC orphans) - No filesystem overhead - Conversation content is text, well within SQLite's comfort zone **Schema:** + ```typescript // cas.sql.ts -export const CASObjectTable = sqliteTable("cas_object", { - hash: text().primaryKey(), // SHA-256 of content - content: text().notNull(), // Original content (JSON-serialized) - content_type: text().notNull(), // "part" | "text" | "tool-output" | "reasoning" - tokens: integer().notNull(), // Token estimate - session_id: text(), // Source session - message_id: text(), // Source message - part_id: text(), // Source part - ...Timestamps, -}, (table) => [ - index("cas_object_session_idx").on(table.session_id), -]) +export const CASObjectTable = sqliteTable( + "cas_object", + { + hash: text().primaryKey(), // SHA-256 of content + content: text().notNull(), // Original content (JSON-serialized) + content_type: text().notNull(), // "part" | "text" | "tool-output" | "reasoning" + tokens: integer().notNull(), // Token estimate + session_id: text(), // Source session + message_id: text(), // Source message + part_id: text(), // Source part + ...Timestamps, + }, + (table) => [index("cas_object_session_idx").on(table.session_id)], +) ``` **Migration:** `packages/opencode/migration/YYYYMMDDHHMMSS_cas/migration.sql` + ```sql CREATE TABLE `cas_object` ( `hash` text PRIMARY KEY NOT NULL, @@ -64,17 +69,25 @@ CREATE INDEX `cas_object_session_idx` ON `cas_object`(`session_id`); ``` **Export from schema registry:** Add to `packages/opencode/src/storage/schema.ts`: + ```typescript export { CASObjectTable } from "../cas/cas.sql" ``` **Module (~80 lines):** + ```typescript // cas/index.ts export namespace CAS { - export async function store(content: string, meta: { - contentType: string, sessionID?: string, messageID?: string, partID?: string - }): Promise { + export async function store( + content: string, + meta: { + contentType: string + sessionID?: string + messageID?: string + partID?: string + }, + ): Promise { const hash = createHash("sha256").update(content).digest("hex") Database.use((db) => { db.insert(CASObjectTable) @@ -87,21 +100,20 @@ export namespace CAS { message_id: meta.messageID, part_id: meta.partID, }) - .onConflictDoNothing() // idempotent — same content = same hash + .onConflictDoNothing() // idempotent — same content = same hash .run() }) return hash } export function get(hash: string): CASObject | null { - return Database.use((db) => - db.select().from(CASObjectTable).where(eq(CASObjectTable.hash, hash)).get() ?? null - ) + return Database.use((db) => db.select().from(CASObjectTable).where(eq(CASObjectTable.hash, hash)).get() ?? null) } export function exists(hash: string): boolean { - return Database.use((db) => - !!db.select({ hash: CASObjectTable.hash }).from(CASObjectTable).where(eq(CASObjectTable.hash, hash)).get() + return Database.use( + (db) => + !!db.select({ hash: CASObjectTable.hash }).from(CASObjectTable).where(eq(CASObjectTable.hash, hash)).get(), ) } } @@ -115,23 +127,25 @@ Note: SQLite ops are synchronous (Bun SQLite), matching the pattern in `todo.ts` ```typescript // NEW: Insert before PartBase (line 81) -export const EditMeta = z.object({ - hidden: z.boolean(), - casHash: z.string().optional(), // hash into CAS for original content - supersededBy: PartID.zod.optional(), // points to replacement part - replacementOf: PartID.zod.optional(), // on replacement: points to original - annotation: z.string().optional(), - editedAt: z.number(), - editedBy: z.string(), // agent name - version: z.string().optional(), // graph node ID -}).optional() +export const EditMeta = z + .object({ + hidden: z.boolean(), + casHash: z.string().optional(), // hash into CAS for original content + supersededBy: PartID.zod.optional(), // points to replacement part + replacementOf: PartID.zod.optional(), // on replacement: points to original + annotation: z.string().optional(), + editedAt: z.number(), + editedBy: z.string(), // agent name + version: z.string().optional(), // graph node ID + }) + .optional() // MODIFY: PartBase (line 81-85) — add edit field const PartBase = z.object({ id: PartID.zod, sessionID: SessionID.zod, messageID: MessageID.zod, - edit: EditMeta, // NEW — all 12 part types inherit this + edit: EditMeta, // NEW — all 12 part types inherit this }) ``` @@ -144,16 +158,16 @@ Safe: `.optional()` means existing parts parse as `edit: undefined`. No SQL migr ```typescript export function filterEdited(messages: WithParts[]): WithParts[] { return messages - .map(msg => ({ + .map((msg) => ({ ...msg, - parts: msg.parts.filter(part => { + parts: msg.parts.filter((part) => { if (!part.edit) return true if (part.edit.hidden) return false if (part.edit.supersededBy) return false return true - }) + }), })) - .filter(msg => msg.parts.length > 0) + .filter((msg) => msg.parts.length > 0) } ``` @@ -170,9 +184,10 @@ export function filterEdited(messages: WithParts[]): WithParts[] { **New file:** `packages/opencode/src/context-edit/index.ts` (~300 lines) -Operations: `hide`, `unhide`, `replace`, `annotate`, `externalize` +Operations: `hide`, `unhide`, `replace`, `annotate`, `externalize`, `mark` Each operation: + 1. Validates ownership (`msg.role !== "user"`, `msg.agent === caller.agent`) 2. Validates budget (max 10/turn, max 70% hidden) 3. Validates recency (cannot edit last 2 turns) @@ -182,12 +197,23 @@ Each operation: 7. Publishes bus event via `Database.effect()` Key: `replace` uses `Database.transaction()` for atomicity: + ```typescript Database.transaction(() => { const hash = CAS.store(JSON.stringify(part), { contentType: "part", sessionID, partID }) const newPartID = Identifier.ascending("part") - Session.updatePart({...part, edit: {hidden:true, casHash:hash, supersededBy:newPartID, editedAt:Date.now(), editedBy:agent}}) - Session.updatePart({id:newPartID, sessionID, messageID, type:"text", text:replacement, edit:{hidden:false, replacementOf:partID, editedAt:Date.now(), editedBy:agent}}) + Session.updatePart({ + ...part, + edit: { hidden: true, casHash: hash, supersededBy: newPartID, editedAt: Date.now(), editedBy: agent }, + }) + Session.updatePart({ + id: newPartID, + sessionID, + messageID, + type: "text", + text: replacement, + edit: { hidden: false, replacementOf: partID, editedAt: Date.now(), editedBy: agent }, + }) }) ``` @@ -214,7 +240,9 @@ Constraints: own messages only, not last 2 turns, max 10 edits/turn.`, annotation: z.string().optional(), summary: z.string().optional(), }), - async execute(args, ctx) { /* dispatch to ContextEdit.* */ } + async execute(args, ctx) { + /* dispatch to ContextEdit.* */ + }, })) ``` @@ -228,7 +256,7 @@ export const ContextDerefTool = Tool.define("context_deref", async () => ({ const entry = CAS.get(args.hash) if (!entry) return { title: "Not found", output: `No content for hash ${args.hash}`, metadata: {} } return { title: "Retrieved", output: entry.content, metadata: { hash: args.hash, tokens: entry.tokens } } - } + }, })) ``` @@ -255,28 +283,30 @@ The conversation graph models edits as a DAG with parent pointers — like git c **New file:** `packages/opencode/src/cas/graph.sql.ts` ```typescript -export const EditGraphNodeTable = sqliteTable("edit_graph_node", { - id: text().primaryKey(), // Node ID - parent_id: text(), // Parent node (forms DAG) - session_id: text().notNull(), // Session scope - part_id: text().notNull(), // Part that was edited - operation: text().notNull(), // hide | unhide | replace | annotate | externalize - cas_hash: text(), // CAS hash of content BEFORE this edit - agent: text().notNull(), // Who made the edit - ...Timestamps, -}, (table) => [ - index("edit_graph_session_idx").on(table.session_id), - index("edit_graph_parent_idx").on(table.parent_id), -]) +export const EditGraphNodeTable = sqliteTable( + "edit_graph_node", + { + id: text().primaryKey(), // Node ID + parent_id: text(), // Parent node (forms DAG) + session_id: text().notNull(), // Session scope + part_id: text().notNull(), // Part that was edited + operation: text().notNull(), // hide | unhide | replace | annotate | externalize + cas_hash: text(), // CAS hash of content BEFORE this edit + agent: text().notNull(), // Who made the edit + ...Timestamps, + }, + (table) => [index("edit_graph_session_idx").on(table.session_id), index("edit_graph_parent_idx").on(table.parent_id)], +) export const EditGraphHeadTable = sqliteTable("edit_graph_head", { - session_id: text().primaryKey(), // One head per session - node_id: text().notNull(), // Current tip + session_id: text().primaryKey(), // One head per session + node_id: text().notNull(), // Current tip branches: text({ mode: "json" }).$type>(), // name → node ID }) ``` **Migration:** Same migration directory as CAS (or separate): + ```sql CREATE TABLE `edit_graph_node` ( `id` text PRIMARY KEY NOT NULL, @@ -308,14 +338,18 @@ CREATE TABLE `edit_graph_head` ( ```typescript export namespace EditGraph { export function commit(input: { - sessionID: string, partID: string, operation: string, - casHash?: string, agent: string, parentID?: string - }): string // returns node ID + sessionID: string + partID: string + operation: string + casHash?: string + agent: string + parentID?: string + }): string // returns node ID export function log(sessionID: string): GraphNode[] // Walk parent pointers from head to root - export function tree(sessionID: string): { nodes: GraphNode[], head: string, branches: Record } + export function tree(sessionID: string): { nodes: GraphNode[]; head: string; branches: Record } // Full DAG for the session export function checkout(sessionID: string, nodeID: string): void @@ -344,7 +378,9 @@ export const ContextHistoryTool = Tool.define("context_history", async () => ({ nodeID: z.string().optional(), branch: z.string().optional(), }), - async execute(args, ctx) { /* dispatch to EditGraph.* */ } + async execute(args, ctx) { + /* dispatch to EditGraph.* */ + }, })) ``` @@ -376,23 +412,33 @@ The session index for the graph is the `edit_graph_head` table — one row per s **New file:** `packages/opencode/src/session/side-thread.sql.ts` ```typescript -export const SideThreadTable = sqliteTable("side_thread", { - id: text().primaryKey(), - project_id: text().notNull().references(() => ProjectTable.id, { onDelete: "cascade" }), - title: text().notNull(), - description: text().notNull(), - status: text().notNull().$default(() => "parked"), - priority: text().notNull().$default(() => "medium"), - category: text().notNull().$default(() => "other"), - source_session_id: text(), - source_part_ids: text({ mode: "json" }).$type(), - cas_refs: text({ mode: "json" }).$type(), - related_files: text({ mode: "json" }).$type(), - created_by: text().notNull(), - ...Timestamps, -}, (table) => [ - index("side_thread_project_idx").on(table.project_id, table.status), -]) +export const SideThreadTable = sqliteTable( + "side_thread", + { + id: text().primaryKey(), + project_id: text() + .notNull() + .references(() => ProjectTable.id, { onDelete: "cascade" }), + title: text().notNull(), + description: text().notNull(), + status: text() + .notNull() + .$default(() => "parked"), + priority: text() + .notNull() + .$default(() => "medium"), + category: text() + .notNull() + .$default(() => "other"), + source_session_id: text(), + source_part_ids: text({ mode: "json" }).$type(), + cas_refs: text({ mode: "json" }).$type(), + related_files: text({ mode: "json" }).$type(), + created_by: text().notNull(), + ...Timestamps, + }, + (table) => [index("side_thread_project_idx").on(table.project_id, table.status)], +) ``` Add to same migration. Export from `storage/schema.ts`. @@ -435,17 +481,9 @@ focus: { **New file:** `packages/opencode/src/tool/thread-park.ts` (~60 lines) **New file:** `packages/opencode/src/tool/thread-list.ts` (~40 lines) -### 3.5 Post-turn focus hook - -**Modify:** `packages/opencode/src/session/prompt.ts` (~line 686) - -```typescript -if (config.experimental?.focus_agent && step >= 2 && result === "continue") { - await runFocusAgent(sessionID, model, abort, msgs) -} -``` +### 3.5 Focus agent invocation (on-demand) -Follows `SessionCompaction.process()` pattern. +**Note:** The automatic post-turn focus hook was removed in v2. The focus agent is now invoked on-demand via the `/focus` command. The system prompt injects focus status (objective + parked threads) when `context_edit` is in the resolved tool set, so build/plan agents self-manage context. ### 3.6 Objective tracker @@ -467,32 +505,32 @@ Follows `SessionCompaction.process()` pattern. ## Files Summary -| Phase | File | Action | ~LOC | -|:-----:|------|--------|:----:| -| 1 | `src/cas/cas.sql.ts` | New | 20 | -| 1 | `src/cas/index.ts` | New | 80 | -| 1 | `src/storage/schema.ts` | Modify | +3 | -| 1 | `migration/.../migration.sql` | New | 30 | -| 1 | `src/session/message-v2.ts` | Modify | +25 | -| 1 | `src/session/prompt.ts` | Modify | +1 | -| 1 | `src/context-edit/index.ts` | New | 300 | -| 1 | `src/tool/context-edit.ts` | New | 80 | -| 1 | `src/tool/context-deref.ts` | New | 40 | -| 1 | `src/tool/registry.ts` | Modify | +5 | -| 2 | `src/cas/graph.sql.ts` | New | 30 | -| 2 | `src/cas/graph.ts` | New | 200 | -| 2 | `src/tool/context-history.ts` | New | 60 | -| 3 | `src/session/side-thread.sql.ts` | New | 25 | -| 3 | `src/session/side-thread.ts` | New | 120 | -| 3 | `src/agent/agent.ts` | Modify | +20 | -| 3 | `src/agent/prompt/focus.txt` | New | 50 | -| 3 | `src/tool/thread-park.ts` | New | 60 | -| 3 | `src/tool/thread-list.ts` | New | 40 | -| 3 | `src/session/prompt.ts` | Modify | +15 | -| 3 | `src/session/objective.ts` | New | 80 | -| 4 | `src/session/prompt.ts` | Modify | +10 | -| 4 | `packages/plugin/src/index.ts` | Modify | +12 | -| | | **Total** | **~1,380** | +| Phase | File | Action | ~LOC | +| :---: | -------------------------------- | --------- | :--------: | +| 1 | `src/cas/cas.sql.ts` | New | 20 | +| 1 | `src/cas/index.ts` | New | 80 | +| 1 | `src/storage/schema.ts` | Modify | +3 | +| 1 | `migration/.../migration.sql` | New | 30 | +| 1 | `src/session/message-v2.ts` | Modify | +25 | +| 1 | `src/session/prompt.ts` | Modify | +1 | +| 1 | `src/context-edit/index.ts` | New | 300 | +| 1 | `src/tool/context-edit.ts` | New | 80 | +| 1 | `src/tool/context-deref.ts` | New | 40 | +| 1 | `src/tool/registry.ts` | Modify | +5 | +| 2 | `src/cas/graph.sql.ts` | New | 30 | +| 2 | `src/cas/graph.ts` | New | 200 | +| 2 | `src/tool/context-history.ts` | New | 60 | +| 3 | `src/session/side-thread.sql.ts` | New | 25 | +| 3 | `src/session/side-thread.ts` | New | 120 | +| 3 | `src/agent/agent.ts` | Modify | +20 | +| 3 | `src/agent/prompt/focus.txt` | New | 50 | +| 3 | `src/tool/thread-park.ts` | New | 60 | +| 3 | `src/tool/thread-list.ts` | New | 40 | +| 3 | `src/session/prompt.ts` | Modify | +15 | +| 3 | `src/session/objective.ts` | New | 80 | +| 4 | `src/session/prompt.ts` | Modify | +10 | +| 4 | `packages/plugin/src/index.ts` | Modify | +12 | +| | | **Total** | **~1,380** | All paths relative to `packages/opencode/`. @@ -502,19 +540,20 @@ All paths relative to `packages/opencode/`. ### Why SQLite for CAS (not files) -| Concern | SQLite | File-based | -|---------|--------|------------| -| Atomicity with part updates | Same transaction | Separate write, can drift | -| Queryable (GC, session lookup) | Yes (SQL) | Must scan filesystem | -| Deduplication | `ON CONFLICT DO NOTHING` | Check before write | -| Performance | Fast for text blobs <1MB | File-per-blob overhead | -| DB size growth | Only concern | Not an issue | +| Concern | SQLite | File-based | +| ------------------------------ | ------------------------ | ------------------------- | +| Atomicity with part updates | Same transaction | Separate write, can drift | +| Queryable (GC, session lookup) | Yes (SQL) | Must scan filesystem | +| Deduplication | `ON CONFLICT DO NOTHING` | Check before write | +| Performance | Fast for text blobs <1MB | File-per-blob overhead | +| DB size growth | Only concern | Not an issue | Mitigation for DB growth: add `VACUUM` to the existing hourly `Snapshot.cleanup()` scheduler. Content is text, compresses well in WAL mode. ### Why conversation graph in SQLite (not file-based Storage) The graph needs: + - Parent pointer traversal (walk DAG) — `WHERE parent_id = ?` is fast with index - Session-scoped queries — `WHERE session_id = ?` - Atomic commits (graph node + CAS entry + part update in one tx) @@ -542,25 +581,26 @@ Session.fork() copies messages. Edit graph.fork() creates a branch within the sa ## Key Reuse Points -| Existing Code | Reuse For | -|--------------|-----------| -| `Database.transaction()` + `Database.use()` | Atomic CAS + part + graph writes | -| `Database.effect()` | Bus events after DB commit | -| `Session.updatePart()` | All part mutations | -| `BusEvent.define()` + `Bus.publish()` | Edit events | -| `SessionCompaction.process()` pattern | Focus agent post-turn invocation | -| `Todo` module pattern | Side thread CRUD | -| `Identifier.ascending("part")` | New part IDs, graph node IDs | -| `Token.estimate()` | Token counting for CAS | -| `Timestamps` from `storage/schema.ts` | `time_created`/`time_updated` on new tables | -| `index()` from drizzle-orm | Table indexes | -| Schema export pattern in `storage/schema.ts` | Register new tables | +| Existing Code | Reuse For | +| -------------------------------------------- | ------------------------------------------- | +| `Database.transaction()` + `Database.use()` | Atomic CAS + part + graph writes | +| `Database.effect()` | Bus events after DB commit | +| `Session.updatePart()` | All part mutations | +| `BusEvent.define()` + `Bus.publish()` | Edit events | +| `SessionCompaction.process()` pattern | Focus agent post-turn invocation | +| `Todo` module pattern | Side thread CRUD | +| `Identifier.ascending("part")` | New part IDs, graph node IDs | +| `Token.estimate()` | Token counting for CAS | +| `Timestamps` from `storage/schema.ts` | `time_created`/`time_updated` on new tables | +| `index()` from drizzle-orm | Table indexes | +| Schema export pattern in `storage/schema.ts` | Register new tables | --- ## Verification ### Phase 1 + 1. Create session, get assistant response with tool calls 2. `context_edit(operation:"hide", partID:"prt_...", messageID:"msg_...")` 3. Verify: hidden part absent from next LLM call; CAS entry exists in `cas_object` table @@ -571,20 +611,24 @@ Session.fork() copies messages. Edit graph.fork() creates a branch within the sa 8. Verify: original in CAS, new TextPart created, old part has `supersededBy` ### Phase 2 + 9. After edits, `context_history(operation:"log")` — verify chain n1→n2→n3 10. `context_history(operation:"fork", nodeID:"n2", branch:"alt")` — branch created 11. `context_history(operation:"checkout", nodeID:"n1")` — parts restored from CAS ### Phase 3 + 12. Enable focus agent, multi-turn session with divergence 13. Verify focus agent parks a side thread, hides divergent content 14. `thread_list` — parked thread appears with CAS refs ### Phase 4 + 15. Verify system prompt contains focus status + thread summary 16. Verify plugin `context.edit.before` hook fires ### Running Tests + ```bash cd packages/opencode bun test src/cas/ diff --git a/README.md b/README.md index 0348389dd..f12c1b386 100644 --- a/README.md +++ b/README.md @@ -8,16 +8,125 @@ A fork of [OpenCode](https://github.com/anomalyco/opencode) with agent-driven co Agents can surgically edit their own conversation context — hiding stale tool output, replacing incorrect statements, externalizing verbose content to a content-addressable store — while preserving all original content in a git-like versioned DAG. A deterministic sweeper automatically cleans up parts marked as discardable or ephemeral. -## Quick Start +--- + +## Installation + +### Prerequisites + +- [Bun](https://bun.sh) 1.3.10+ (`bun upgrade` if you have an older version) +- [Git](https://git-scm.com/) +- An API key for at least one LLM provider + +### Clone and install ```bash git clone https://github.com/e6qu/frankencode.git cd frankencode bun install +``` + +### Configure a provider + +Frankencode needs at least one LLM provider configured. Run the provider login flow: + +```bash +bun run --cwd packages/opencode dev -- providers login +``` + +Or set an API key directly in your environment: + +```bash +# Anthropic +export ANTHROPIC_API_KEY="sk-ant-..." + +# OpenAI +export OPENAI_API_KEY="sk-..." + +# DeepSeek +export DEEPSEEK_API_KEY="..." + +# Or any other supported provider (see opencode.ai/docs/providers) +``` + +### Run + +```bash +# Start the TUI (terminal UI) bun run --cwd packages/opencode dev + +# Or with a specific model +bun run --cwd packages/opencode dev -- --model anthropic/claude-sonnet-4-6 +``` + +This launches the interactive TUI. Use `Tab` to switch agents, `Ctrl+P` for the command palette, `/` for slash commands. + +### Non-interactive mode (CLI) + +```bash +# Single message +bun run --cwd packages/opencode dev -- run "explain what packages/opencode/src/cas/index.ts does" + +# Continue a session +bun run --cwd packages/opencode dev -- run "now externalize that read result" --continue + +# Continue a specific session +bun run --cwd packages/opencode dev -- run "hide the old grep" --session ses_abc123 + +# JSON output (for scripting) +bun run --cwd packages/opencode dev -- run "list your tools" --format json + +# With a specific model +bun run --cwd packages/opencode dev -- run "hello" --model deepseek/deepseek-chat +``` + +### Global config (optional) + +Create `~/.config/opencode/opencode.jsonc` to set defaults across all projects: + +```jsonc +{ + "$schema": "https://opencode.ai/config.json", + "provider": { + // Set your default provider + "anthropic": {}, + }, + "permission": { + // Auto-allow context editing tools + "context_edit": { "*": "allow" }, + "context_deref": { "*": "allow" }, + "context_history": { "*": "allow" }, + "thread_park": { "*": "allow" }, + "thread_list": { "*": "allow" }, + "classifier_threads": { "*": "allow" }, + "distill_threads": { "*": "allow" }, + }, + "agent": { + // Use a cheap model for the classifier + "classifier": { + // "model": "anthropic/claude-haiku-4-5" + }, + }, +} ``` -Requires [Bun](https://bun.sh) 1.3.10+. +### Project config + +Per-project settings go in `.opencode/opencode.jsonc` at the project root. See [Configuration](#configuration) below. + +### File locations + +| What | Where | +| ------------------ | ------------------------------------- | +| Global config | `~/.config/opencode/opencode.jsonc` | +| Project config | `.opencode/opencode.jsonc` | +| Credentials | `~/.local/share/opencode/auth.json` | +| Database | `~/.local/share/opencode/opencode.db` | +| CAS + edit history | Inside the database (SQLite tables) | +| Session storage | `~/.local/share/opencode/storage/` | +| Logs | `~/.local/share/opencode/log/` | + +--- ## Tools @@ -40,6 +149,73 @@ Requires [Bun](https://bun.sh) 1.3.10+. | `/btw ` | Side conversation — answers without polluting the main thread | | `/reset-context` | Restore all edited parts to originals from CAS | +--- + +## Usage Examples + +### Hiding stale content + +``` +You: That grep result from earlier is stale — I refactored auth since then. Hide it. + +Agent: [context_edit(operation: "hide", toolName: "grep")] + Applied hide on prt_abc123. Original preserved: 7f3a9b2e... +``` + +### Externalizing verbose output + +``` +You: The 200-line read result — externalize it, we only need the summary. + +Agent: [context_edit(operation: "externalize", toolName: "read", + summary: "CAS module with SHA-256 hashing and SQLite CRUD")] + Applied externalize. Original preserved: d41e9086... + +You: Actually, show me that file again. + +Agent: [context_deref(hash: "d41e9086...")] + [full file content retrieved from CAS] +``` + +### Marking parts for auto-cleanup + +``` +You: Run the tests. + +Agent: [bash("bun test")] + Error: 3 tests failed... + [context_edit(operation: "mark", toolName: "bash", + hint: "discardable", reason: "Failed test run, will retry")] + Marked as discardable — will auto-hide after 3 turns. +``` + +### Parking side threads + +``` +You: I noticed the auth middleware has no rate limiting. Park that. + +Agent: [thread_park(title: "Auth middleware missing rate limiting", + priority: "high", category: "security")] + [Side thread thr_abc parked] + +You: What side threads do we have? + +Agent: [thread_list] + thr_abc [parked, high, security] "Auth middleware missing rate limiting" +``` + +### Edit history + +``` +You: Show me what we've edited. + +Agent: [context_history(operation: "log")] + prt_f1a2 (HEAD) externalize on prt_abc1 by build [14:23:01] + prt_e5d6 hide on prt_def4 by build [14:22:45] +``` + +--- + ## Documentation | Document | Contents | @@ -52,7 +228,7 @@ Requires [Bun](https://bun.sh) 1.3.10+. ## Configuration -All features are controlled via `opencode.jsonc`: +All features controlled via `opencode.jsonc` (project-level at `.opencode/opencode.jsonc` or global at `~/.config/opencode/opencode.jsonc`): ```jsonc { @@ -70,6 +246,8 @@ All features are controlled via `opencode.jsonc`: } ``` +Config merges from lowest to highest priority: global → project → runtime. + ## Upstream Frankencode tracks upstream OpenCode (`dev` branch). All original features, providers, tools, and configuration work as documented at [opencode.ai/docs](https://opencode.ai/docs). diff --git a/WHAT_WE_DID.md b/WHAT_WE_DID.md index 00cb7eb1d..1e54dfac1 100644 --- a/WHAT_WE_DID.md +++ b/WHAT_WE_DID.md @@ -1,126 +1,96 @@ # Frankencode — What We Did -## Phase 0: Research & Design (completed) +## Phase 0: Research & Design - Deep research on OpenCode architecture - Designed editable context system (6 modes, Merkle CAS, focus agent, side threads) - Literature review of 40+ papers/tools/frameworks - Narrowed to MVP plan -### Documents produced (now in `docs/research/`): -`REPORT.md`, `EDITABLE_CONTEXT.md`, `EDITABLE_CONTEXT_PLUGIN_PLAN.md`, `EDITABLE_CONTEXT_FORK_PLAN.md`, `EDITABLE_CONTEXT_MODES.md`, `EDITABLE_CONTEXT_MERKLE.md`, `EDITABLE_CONTEXT_FOCUS.md`, `EDITABLE_CONTEXT_PRESS_RELEASE.md`, `DEEP_RESEARCH_POST_FACTUM_CONTEXT_EDITING.md`, `CONTEXT_EDITING_MVP.md`, `UI_CUSTOMIZATION.md` +### Documents produced (in `docs/research/`): + +`DEEP_RESEARCH_POST_FACTUM_CONTEXT_EDITING.md`, `UI_CUSTOMIZATION.md` Root-level: `PLAN.md`, `WHAT_WE_DID.md`, `DO_NEXT.md` --- -## Phase 1: CAS + Part Editing Foundation (completed) - -### Files created: -- `packages/opencode/src/cas/cas.sql.ts` — Drizzle table definitions for `cas_object`, `edit_graph_node`, `edit_graph_head` -- `packages/opencode/src/cas/index.ts` — CAS module: `store()`, `get()`, `exists()`, `listBySession()` -- `packages/opencode/src/context-edit/index.ts` — Core edit logic: `hide`, `unhide`, `replace`, `annotate`, `externalize` with ownership/budget/recency validation, CAS integration, bus events -- `packages/opencode/src/tool/context-edit.ts` — `context_edit` tool (5 operations) -- `packages/opencode/src/tool/context-deref.ts` — `context_deref` tool (retrieve CAS content by hash) -- `packages/opencode/migration/20260315120000_context_editing/migration.sql` — SQL migration for all 3 new tables +## Phase 1: CAS + Part Editing Foundation -### Files modified: -- `packages/opencode/src/session/message-v2.ts` — Added `EditMeta` schema on `PartBase` (all 12 part types inherit `edit` field); added `filterEdited()` function -- `packages/opencode/src/session/prompt.ts` — Inserted `msgs = MessageV2.filterEdited(msgs)` after `filterCompacted` in the main loop (line 302) -- `packages/opencode/src/storage/schema.ts` — Exported new tables from CAS module -- `packages/opencode/src/tool/registry.ts` — Registered `ContextEditTool` and `ContextDerefTool` in BUILTIN array +- `cas_object` SQLite table for content-addressable storage +- `EditMeta` and `LifecycleMeta` schemas on `PartBase` (all 12 part types inherit) +- `filterEdited()` in message pipeline + deterministic sweeper for lifecycle markers +- `context_edit` tool (hide, unhide, replace, annotate, externalize, mark) +- `context_deref` tool (retrieve CAS content by hash) +- Plugin hooks: `context.edit.before` / `context.edit.after` -### Verification: -- All 1310 existing tests pass (0 failures) -- 1 pre-existing error in retry.test.ts (unrelated `test.concurrent` issue) +## Phase 2: Conversation Graph ---- +- `edit_graph_node` + `edit_graph_head` SQLite tables (DAG with parent pointers) +- `EditGraph` module (commit, log, tree, checkout, fork, switchBranch) +- `context_history` tool (log, tree, checkout, fork) +- All edit operations record graph nodes atomically -## Phase 2: Conversation Graph (completed) +## Phase 3: Focus Agent + Side Threads -### Files created: -- `packages/opencode/src/cas/graph.ts` — `EditGraph` namespace: `commit`, `getLog`, `tree`, `checkout`, `fork`, `switchBranch`. DAG of edit nodes with parent pointers, per-session head tracking, named branches. -- `packages/opencode/src/tool/context-history.ts` — `context_history` tool (log, tree, checkout, fork operations) +- `side_thread` SQLite table (project-level, survives sessions) +- `SideThread` CRUD module +- `thread_park` / `thread_list` tools +- Objective tracker (extracts goal from first user message) +- Focus agent (hidden, on-demand via `/focus` command) +- Classifier agent (read-only, labels messages as main/side/mixed with topics) +- Focus-rewrite-history agent (full conversation rewrite with user confirmation) -### Files modified: -- `packages/opencode/src/context-edit/index.ts` — All 4 edit operations (hide, replace, externalize, annotate) now call `EditGraph.commit()` inside the same `Database.transaction()`, setting `edit.version` on parts -- `packages/opencode/src/tool/registry.ts` — Registered `ContextHistoryTool` in BUILTIN array -- `packages/opencode/src/session/message-v2.ts` — Fixed pre-existing `as` casts (line 536, 873) to route through `unknown` for compatibility with new `edit` field on `PartBase` -- `packages/opencode/src/session/prompt.ts` — Fixed pre-existing `as` cast (line 992) same pattern +## Phase 4: Integration + v2 -### Verification: -- All 1310 existing tests pass (0 failures) +- System prompt injection (focus status + side threads when context_edit available) +- `classifier_threads` tool (run classifier, return structured JSON) +- `distill_threads` tool (classify + park side threads + store metadata) +- Config-based control (no feature toggles): tools disabled via config, agents via `disable: true` +- `/btw`, `/focus`, `/focus-rewrite-history`, `/reset-context` commands +- Lifecycle markers (discardable, ephemeral, side-thread, pinned) with deterministic sweeper +- Privileged agents (focus, compaction) can edit any message +- Query and toolName targeting for `context_edit` --- -## Phase 3: Focus Agent + Side Threads (completed) - -### Files created: -- `packages/opencode/src/session/side-thread.sql.ts` — Drizzle table for `side_thread` (project-level, survives sessions) -- `packages/opencode/src/session/side-thread.ts` — CRUD module: `create`, `get`, `list`, `update` with bus events -- `packages/opencode/src/session/objective.ts` — Objective tracker: `get`, `set`, `extract` (from first user message, cached in Storage) -- `packages/opencode/src/tool/thread-park.ts` — `thread_park` tool -- `packages/opencode/src/tool/thread-list.ts` — `thread_list` tool -- `packages/opencode/src/agent/prompt/focus.txt` — Focus agent system prompt - -### Files modified: -- `packages/opencode/migration/20260315120000_context_editing/migration.sql` — Added `side_thread` table + index -- `packages/opencode/src/storage/schema.ts` — Exported `SideThreadTable` -- `packages/opencode/src/agent/agent.ts` — Added `focus` agent (hidden, temp 0, max 8 steps, restricted to context_edit/thread_park/thread_list/question tools) -- `packages/opencode/src/flag/flag.ts` — Added `OPENCODE_EXPERIMENTAL_FOCUS_AGENT` flag -- `packages/opencode/src/session/prompt.ts` — Added post-turn focus agent hook (after processor.process(), guarded by flag, runs on step >= 2) -- `packages/opencode/src/tool/registry.ts` — Registered `ThreadParkTool` and `ThreadListTool` - -### Verification: -- All 1310 existing tests pass (0 failures) - ---- +## Files (all paths relative to `packages/opencode/`) -## Phase 4: Integration (completed) +### New files: -### Files modified: -- `packages/opencode/src/session/prompt.ts` — Injected focus status block + side thread summary into system prompt (when `OPENCODE_EXPERIMENTAL_FOCUS_AGENT` is set) -- `packages/plugin/src/index.ts` — Added `context.edit.before` and `context.edit.after` hook types to Hooks interface -- `packages/opencode/src/context-edit/index.ts` — Added `Plugin.trigger()` calls: `pluginGuard()` before hide/replace/externalize, `pluginNotify()` after successful operations - -### Verification: -- All 1310 existing tests pass (0 failures) - ---- +| File | Purpose | +|------|---------| +| `src/cas/cas.sql.ts` | Drizzle tables: cas_object, edit_graph_node, edit_graph_head | +| `src/cas/index.ts` | CAS module: store, get, exists, listBySession | +| `src/cas/graph.ts` | Edit graph DAG: commit, log, tree, checkout, fork | +| `src/context-edit/index.ts` | Edit operations + validation + sweeper + reset + plugin hooks | +| `src/tool/context-edit.ts` | context_edit tool (query/toolName targeting) | +| `src/tool/context-deref.ts` | context_deref tool | +| `src/tool/context-history.ts` | context_history tool | +| `src/tool/thread-park.ts` | thread_park tool | +| `src/tool/thread-list.ts` | thread_list tool | +| `src/tool/classifier-threads.ts` | classifier_threads tool | +| `src/tool/distill-threads.ts` | distill_threads tool | +| `src/session/side-thread.sql.ts` | Drizzle table: side_thread | +| `src/session/side-thread.ts` | SideThread CRUD module | +| `src/session/objective.ts` | Objective tracker | +| `src/agent/prompt/focus.txt` | Focus agent prompt | +| `src/agent/prompt/classifier.txt` | Classifier agent prompt | +| `src/agent/prompt/rewrite-history.txt` | Rewrite-history agent prompt | +| `src/command/template/btw.txt` | /btw command template | +| `src/command/template/focus.txt` | /focus command template | +| `src/command/template/focus-rewrite-history.txt` | /focus-rewrite-history template | +| `src/command/template/reset-context.txt` | /reset-context template | +| `migration/20260315120000_context_editing/` | SQL migration for 4 new tables | + +### Modified files: -## Summary of All Changes - -### New files (13): -| File | LOC | Purpose | -|------|:---:|---------| -| `src/cas/cas.sql.ts` | 40 | Drizzle tables: cas_object, edit_graph_node, edit_graph_head | -| `src/cas/index.ts` | 85 | CAS module (SQLite): store, get, exists, listBySession | -| `src/cas/graph.ts` | 235 | Edit graph DAG: commit, log, tree, checkout, fork, switchBranch | -| `src/context-edit/index.ts` | 370 | Core edit ops: hide, unhide, replace, annotate, externalize + validation + plugin hooks | -| `src/tool/context-edit.ts` | 90 | context_edit tool | -| `src/tool/context-deref.ts` | 35 | context_deref tool | -| `src/tool/context-history.ts` | 95 | context_history tool | -| `src/tool/thread-park.ts` | 55 | thread_park tool | -| `src/tool/thread-list.ts` | 45 | thread_list tool | -| `src/session/side-thread.sql.ts` | 30 | Drizzle table: side_thread | -| `src/session/side-thread.ts` | 155 | SideThread CRUD module | -| `src/session/objective.ts` | 55 | Objective tracker | -| `src/agent/prompt/focus.txt` | 30 | Focus agent system prompt | -| `migration/.../migration.sql` | 45 | SQL migration for 4 new tables | -| **Total** | **~1,365** | | - -### Modified files (8): | File | Changes | |------|---------| -| `src/session/message-v2.ts` | +EditMeta on PartBase, +filterEdited(), fixed as-casts | -| `src/session/prompt.ts` | +filterEdited in pipeline, +focus agent post-turn hook, +focus status in system prompt | -| `src/storage/schema.ts` | +exports for 4 new tables | -| `src/tool/registry.ts` | +6 new tools in BUILTIN array | -| `src/agent/agent.ts` | +focus agent definition (hidden, temp 0, restricted tools) | -| `src/flag/flag.ts` | +OPENCODE_EXPERIMENTAL_FOCUS_AGENT flag | +| `src/session/message-v2.ts` | +EditMeta +LifecycleMeta on PartBase, +filterEdited() | +| `src/session/prompt.ts` | +filterEdited +sweeper in pipeline, +focus status in system prompt | +| `src/storage/schema.ts` | +exports for new tables | +| `src/tool/registry.ts` | +9 new tools in BUILTIN array | +| `src/agent/agent.ts` | +classifier +focus +focus-rewrite-history agent definitions | +| `src/command/index.ts` | +btw +focus +focus-rewrite-history +reset-context commands | | `packages/plugin/src/index.ts` | +context.edit.before/after hook types | -| `src/context-edit/index.ts` | +Plugin.trigger() guard/notify calls | - ---- - -*Last updated after: Phase 4 (all phases complete)* diff --git a/docs/context-editing.md b/docs/context-editing.md index 51bff6289..a9e7543d2 100644 --- a/docs/context-editing.md +++ b/docs/context-editing.md @@ -48,6 +48,12 @@ Edit conversation parts. Targets parts by content search, tool name, or exact ID - Max 10 edits/turn, max 70% hidden ratio - `skill` tool results are protected +**Deterministic sweeper:** + +The sweeper runs automatically in the prompt loop (after `filterEdited()`, before `toModelMessages()`). No LLM call — it reads lifecycle markers, checks how many turns have elapsed since `turnWhenSet`, and auto-hides or auto-externalizes expired parts. All swept content is preserved in CAS. + +Defaults are set in the `mark` operation: `afterTurns` defaults to 3 for `discardable`, 5 for `ephemeral`. Override per-mark by passing a custom `afterTurns` value. + ## context_deref Retrieve content by CAS hash. Returns the original content before it was externalized or hidden.