refactor(index): finish Stroma extraction#313
Conversation
There was a problem hiding this comment.
Pull request overview
Completes the Stroma/Pituitary indexing boundary by moving Pituitary-only fields into a pituitary_records sidecar table and switching spec search to use stroma/index.Search, while updating update/reuse/analysis codepaths to the Stroma records/chunks layout.
Changes:
- Introduce
pituitary_records(status/domain/adapter) and updateartifactscompatibility view to join Stromarecordswith Pituitary state. - Switch
SearchSpecscandidate retrieval tostroma/index.Search, then apply Pituitary status/domain filtering and scoring on returned hits. - Update update/reuse and analysis queries to use
chunks(record_ref, heading, content)and removeartifact_ref/sectiondependencies.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| internal/index/update.go | Update paths now write/query chunks and record_ref instead of legacy chunk tables/columns. |
| internal/index/update_test.go | Adjust orphan-chunk assertion for record_ref → records. |
| internal/index/search.go | Replace SQL vec query with stroma/index.Search + Pituitary post-filters. |
| internal/index/reuse.go | Reuse-state loading updated to Stroma chunk schema (record_ref, heading). |
| internal/index/rebuild.go | Bump schema version and add pituitary_records + updated artifacts view; remove Pituitary fields from Stroma-owned tables. |
| internal/index/rebuild_test.go | Update schema expectations and chunk heading assertions for new layout. |
| internal/analysis/semantic_terminology.go | Update terminology search query to use heading/record_ref. |
| internal/analysis/repository_similarity.go | Update similarity shortlist query to use heading/record_ref. |
| internal/analysis/overlap.go | Update overlap section loading to use record_ref/heading. |
| internal/analysis/doc_drift.go | Update doc drift section loading to use record_ref/heading. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| hits, err := stindex.Search(ctx, stindex.SearchQuery{ | ||
| Path: indexPath, | ||
| Text: query.Query, | ||
| Limit: searchCandidateLimit(query.Limit), | ||
| Kinds: []string{query.Kind}, | ||
| Embedder: embedder, | ||
| }) |
There was a problem hiding this comment.
Search now applies status/domain filtering after calling stroma/index.Search. Since the stroma query is limited before these filters (only an overfetch via searchCandidateLimit), the final result can return fewer than query.Limit even when enough matching specs exist but rank below the overfetch window. Consider iteratively increasing the stroma search limit until enough candidates survive filtering (or a hard cap), or otherwise adjust overfetch based on active filters to preserve the previous “LIMIT after filtering” behavior.
Summary
pituitary_recordssidecar tablesearch-specstostroma/index.Searchand apply Pituitary status/domain ranking filters on top of Stroma hitsrecords/chunkslayout while keeping a zero-copyartifactscompatibility viewWhy
The previous branch only moved rebuild onto Stroma. Pituitary still duplicated substrate ownership by extending Stroma tables directly and by querying a Pituitary-shaped compatibility chunk surface. This change completes the extraction boundary so Stroma owns corpus storage and Pituitary owns governance/state on top.
Validation
go test ./internal/indexgo test ./internal/analysisgo test ./...