Skip to content

refactor(index): finish Stroma extraction#313

Merged
autholykos merged 1 commit intomainfrom
session/stroma-extraction
Apr 13, 2026
Merged

refactor(index): finish Stroma extraction#313
autholykos merged 1 commit intomainfrom
session/stroma-extraction

Conversation

@autholykos
Copy link
Copy Markdown
Member

Summary

  • keep Stroma as the owner of base index storage and move Pituitary-only fields into a pituitary_records sidecar table
  • switch search-specs to stroma/index.Search and apply Pituitary status/domain ranking filters on top of Stroma hits
  • migrate reuse, update, and section-loading analysis paths onto the Stroma records/chunks layout while keeping a zero-copy artifacts compatibility view

Why

The previous branch only moved rebuild onto Stroma. Pituitary still duplicated substrate ownership by extending Stroma tables directly and by querying a Pituitary-shaped compatibility chunk surface. This change completes the extraction boundary so Stroma owns corpus storage and Pituitary owns governance/state on top.

Validation

  • go test ./internal/index
  • go test ./internal/analysis
  • go test ./...

Copilot AI review requested due to automatic review settings April 13, 2026 09:43
@autholykos autholykos merged commit 47595f5 into main Apr 13, 2026
7 checks passed
@autholykos autholykos deleted the session/stroma-extraction branch April 13, 2026 09:47
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Completes the Stroma/Pituitary indexing boundary by moving Pituitary-only fields into a pituitary_records sidecar table and switching spec search to use stroma/index.Search, while updating update/reuse/analysis codepaths to the Stroma records/chunks layout.

Changes:

  • Introduce pituitary_records (status/domain/adapter) and update artifacts compatibility view to join Stroma records with Pituitary state.
  • Switch SearchSpecs candidate retrieval to stroma/index.Search, then apply Pituitary status/domain filtering and scoring on returned hits.
  • Update update/reuse and analysis queries to use chunks(record_ref, heading, content) and remove artifact_ref/section dependencies.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
internal/index/update.go Update paths now write/query chunks and record_ref instead of legacy chunk tables/columns.
internal/index/update_test.go Adjust orphan-chunk assertion for record_refrecords.
internal/index/search.go Replace SQL vec query with stroma/index.Search + Pituitary post-filters.
internal/index/reuse.go Reuse-state loading updated to Stroma chunk schema (record_ref, heading).
internal/index/rebuild.go Bump schema version and add pituitary_records + updated artifacts view; remove Pituitary fields from Stroma-owned tables.
internal/index/rebuild_test.go Update schema expectations and chunk heading assertions for new layout.
internal/analysis/semantic_terminology.go Update terminology search query to use heading/record_ref.
internal/analysis/repository_similarity.go Update similarity shortlist query to use heading/record_ref.
internal/analysis/overlap.go Update overlap section loading to use record_ref/heading.
internal/analysis/doc_drift.go Update doc drift section loading to use record_ref/heading.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +277 to +283
hits, err := stindex.Search(ctx, stindex.SearchQuery{
Path: indexPath,
Text: query.Query,
Limit: searchCandidateLimit(query.Limit),
Kinds: []string{query.Kind},
Embedder: embedder,
})
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Search now applies status/domain filtering after calling stroma/index.Search. Since the stroma query is limited before these filters (only an overfetch via searchCandidateLimit), the final result can return fewer than query.Limit even when enough matching specs exist but rank below the overfetch window. Consider iteratively increasing the stroma search limit until enough candidates survive filtering (or a hard cap), or otherwise adjust overfetch based on active filters to preserve the previous “LIMIT after filtering” behavior.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants