Repository navigation
[PR] Storage wrap-up, nine items in one batch: GC/archive + reap ledger + media CAS + incremental pointers + dsh generation c | 存储收尾九项一车:GC/archive + reap 台账 + 媒体 CAS + 增量指针 + dsh 代压缩默认开 + 多实例文档 (#3901) - #3911
Merged
Conversation
…pointers + dsh-authored flip + multi-instance docs (#3901) - #3463: storage.archiveAfterDays (default 365) age sweep + POST /__bili/storage/gc|restore; auto_vacuum=INCREMENTAL - #3841: schema v6 sessions.orphans_json reap journal — reaped block rows deleted, orphan contents merged into views, never resurrected - #3762: schema v7 message_media media CAS — imageBase64 content-addressed, {$media:hash} refs, byte-identical hydration, refcount release - #3798: inc:canonical-ptrs row pointers replace the base-seq count slice; resync invalidates pre-response; turn lock reordered (409 in response phase, ack-then-wrap-up chain awaited) - #3723/#3733: dsh.authoredCompaction default ON (env 0 / file false = #1729 refusal) - #3633: README 'Running multiple instances' posture section Fixes #3901, fixes #3463, fixes #3762, fixes #3841, fixes #3798, fixes #3633
ranxianglei
changed the title
Contributor
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4659-pr.3911.205724eThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3911.tgz
npm install -g packageInstall instructions are refreshed on each push. |
Cosmetic follow-up to 8409552 (assertion fixes only): test names and comments still read v4/v5 while asserting v7, and the v4-fixture comment did not say why the newer additive shapes riding in the shared DDL re-land as no-ops.
Owner
Author
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复,但发现一个需要处置的关联项:PR #3466 仍 open 且同样声明 初判状态:
标签: pr + dsh + scope/storage + p1(含「重启后坏块复活」#3841 与「存储只增不减」#3762 两个 p1 类缺陷;混合车含功能/文档故不打 bug) 审核进行中:全量 diff 通读 + 既有行为回归清点 + 定向测试,随后给结构化八问结论。 |
Owner
Author
(ai: vllm-qwen/qwen3.8-27b) 查重: 重复触发——本条与先前通知是同一 head(8a53a0f)、同一组失败 job,按一次处理,不重复修。 结论先行:CI 已全绿可合。 8a53a0f 上的两个红门 + 随后暴露的两个存储迁移测试,都已在分支上修掉并验证。当前 head 结构化八问
CI 状态(head e23d9c7)
自动修复通道用量:2 次以内,无需设计决策。判定:可合(待剩余矩阵分片自然转绿即可,预期无意外)。 |
added 3 commits
October 20, 2026 14:52
…#3901) The full-path authoritative sweep deleted stale message rows and released their content hashes but never touched message_media — the CAS refcount leaked forever and the image bytes were never reclaimable, contradicting the #3762 release contract that deleteSession/archiveSession already keep. Release each referenced media hash and drop the mappings with their row. Regression test pins red→green.
Contributor
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.649-master.9ffb852 # this exact build
npm install -g billion-context@master # always the newest merged build
|
Owner
Author
|
Review follow-up (single-file archive): sessions are small and numerous, so a per-session export file would spray tiny files across the archive dir. Archiving now appends ONE JSON line per session to a single |
This was referenced Oct 20, 2026
ranxianglei
added a commit
that referenced
this pull request
Oct 20, 2026
[PR] storage: GC archive as one append-only JSONL log (follow-up to #3911)
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
…ve/restore + bili storage gc (#3928) Thin command layer over the store APIs #3911 exposed over HTTP (POST /__bili/storage/gc, POST /__bili/storage/restore): no new store semantics, no scheduler. - sessions ls: id, label, decoded size, last-updated, storage status (active/archived); DB rows unioned with the JSON tree like bili export. - sessions rm <id>: full delete via the existing deleteSession discipline (content refcount release incl. media), VACUUM afterwards (--no-vacuum). - sessions archive/restore <id>: wrap archiveSession/restoreArchived on the single append-only archive.jsonl log; unified-store-only commands refuse legacy lanes with the BILI_STORAGE_UNIFIED hint. - storage gc [--days N] [--vacuum full|incremental] [--dry-run]: prints the sweep plan (age-sweep candidates, zero-ref content rows, ~pages) from read-only queries, then runs the same unifiedGc() the HTTP endpoint calls; --dry-run stops after the plan. Additive store surface backing the plan/statistics: GcPlan/gcPlan (read-only prediction), contentZeroRefReport, sessionStorageStats, vacuum(mode), status column in sessionSummaries/webSummaryRows. Tests: tests/storage-maintenance-cli.test.ts demonstrates every command against scratch stores and pins gc parity with direct store.gc(). Docs: README en/zh 'Storage maintenance CLI' + CONFIGURATION en/zh.
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
…ve/restore + bili storage gc (#3928) Thin command layer over the store APIs #3911 exposed over HTTP (POST /__bili/storage/gc, POST /__bili/storage/restore): no new store semantics, no scheduler. - sessions ls: id, label, decoded size, last-updated, storage status (active/archived); DB rows unioned with the JSON tree like bili export. - sessions rm <id>: full delete via the existing deleteSession discipline (content refcount release incl. media), VACUUM afterwards (--no-vacuum). - sessions archive/restore <id>: wrap archiveSession/restoreArchived on the single append-only archive.jsonl log; unified-store-only commands refuse legacy lanes with the BILI_STORAGE_UNIFIED hint. - storage gc [--days N] [--vacuum full|incremental] [--dry-run]: prints the sweep plan (age-sweep candidates, zero-ref content rows, ~pages) from read-only queries, then runs the same unifiedGc() the HTTP endpoint calls; --dry-run stops after the plan. Additive store surface backing the plan/statistics: GcPlan/gcPlan (read-only prediction), contentZeroRefReport, sessionStorageStats, vacuum(mode), status column in sessionSummaries/webSummaryRows. Tests: tests/storage-maintenance-cli.test.ts demonstrates every command against scratch stores and pins gc parity with direct store.gc(). Docs: README en/zh 'Storage maintenance CLI' + CONFIGURATION en/zh.
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
…ve/restore + bili storage gc (#3928) Thin command layer over the store APIs #3911 exposed over HTTP (POST /__bili/storage/gc, POST /__bili/storage/restore): no new store semantics, no scheduler. - sessions ls: id, label, decoded size, last-updated, storage status (active/archived); DB rows unioned with the JSON tree like bili export. - sessions rm <id>: full delete via the existing deleteSession discipline (content refcount release incl. media), VACUUM afterwards (--no-vacuum). - sessions archive/restore <id>: wrap archiveSession/restoreArchived on the single append-only archive.jsonl log; unified-store-only commands refuse legacy lanes with the BILI_STORAGE_UNIFIED hint. - storage gc [--days N] [--vacuum full|incremental] [--dry-run]: prints the sweep plan (age-sweep candidates, zero-ref content rows, ~pages) from read-only queries, then runs the same unifiedGc() the HTTP endpoint calls; --dry-run stops after the plan. Additive store surface backing the plan/statistics: GcPlan/gcPlan (read-only prediction), contentZeroRefReport, sessionStorageStats, vacuum(mode), status column in sessionSummaries/webSummaryRows. Tests: tests/storage-maintenance-cli.test.ts demonstrates every command against scratch stores and pins gc parity with direct store.gc(). Docs: README en/zh 'Storage maintenance CLI' + CONFIGURATION en/zh.
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
…ve/restore + bili storage gc (#3928) Thin command layer over the store APIs #3911 exposed over HTTP (POST /__bili/storage/gc, POST /__bili/storage/restore): no new store semantics, no scheduler. - sessions ls: id, label, decoded size, last-updated, storage status (active/archived); DB rows unioned with the JSON tree like bili export. - sessions rm <id>: full delete via the existing deleteSession discipline (content refcount release incl. media), VACUUM afterwards (--no-vacuum). - sessions archive/restore <id>: wrap archiveSession/restoreArchived on the single append-only archive.jsonl log; unified-store-only commands refuse legacy lanes with the BILI_STORAGE_UNIFIED hint. - storage gc [--days N] [--vacuum full|incremental] [--dry-run]: prints the sweep plan (age-sweep candidates, zero-ref content rows, ~pages) from read-only queries, then runs the same unifiedGc() the HTTP endpoint calls; --dry-run stops after the plan. Additive store surface backing the plan/statistics: GcPlan/gcPlan (read-only prediction), contentZeroRefReport, sessionStorageStats, vacuum(mode), status column in sessionSummaries/webSummaryRows. Tests: tests/storage-maintenance-cli.test.ts demonstrates every command against scratch stores and pins gc parity with direct store.gc(). Docs: README en/zh 'Storage maintenance CLI' + CONFIGURATION en/zh.
This was referenced Oct 20, 2026
This was referenced Oct 22, 2026
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3901, fixes #3463, fixes #3762, fixes #3841, fixes #3798, fixes #3633
One PR, nine items — the storage wrap-up wave agreed in #3901. kernel-change: none (all src/, schema, tests, docs).
What
storage.archiveAfterDaysage sweep (default 365, 0=off) archives idle sessions (full export tosessions/archive/+ tombstone row keeps the label + releases content/media refs); manualPOST /__bili/storage/gc(?vacuum=full|incremental) andPOST /__bili/storage/restore;auto_vacuum=INCREMENTALalways onsessions.orphans_json: reaped blocks get their block rows deleted (refs released) and their contents journaled; views merge orphans into blockContents without resurrectingstate.blocks; restart no longer revives reaped blocks or pollutes envelopesmessage_media: image bytes content-addressed in the CAS, message rows store{"$media":hash}(byte-identical hydration on read), one copy per distinct image across sessions, refcount release on delete/archiveinc:canonical-ptrsordered row pointers replace the base-seq COUNT slice — resyncs that share history reuse rows below the cutoff instead of losing the prefix; resync invalidates pointers pre-response (crash window falls back to the old slice semantics, never the wrong branch); turn lock reordered: 409 turn-in-flight stays for the response phase, ack-then-wrap-up durability chain is awaited by the next turndsh.authoredCompactiondefault ON (0.1.203 dogfood window held; failure mode = checkpoint doesn't shrink, never data loss); envBILI_DSH_AUTHORED_COMPACTION=0or filefalserestores the #1729 refusal/__bili/replicas= checkpoints not podsTests (staked red first, then implementation)
tests/inc-canonical-pointers.test.ts(3) — identical resync keeps the shared prefix; edited-tail resync drops the replaced branch; malformed pointers force an honest resync, never a partial historytests/storage-reap-journal.test.ts(2) — reaped blocks don't survive restart; journal idempotenttests/storage-archive-gc.test.ts(3) — knob matrix; archive tombstone + shared content rows survive; restore + age sweeptests/media-cas.test.ts(2) — same image across two sessions stored once; byte-identical round-trip + refcount laddertests/incremental-lane.test.ts19/19,tests/storage-unified.test.ts14/14 (v7 expectation),tests/dsh-authored-compaction.test.ts16/16 (off-leg pins the explicit escape hatch), bounded/scale/replicas/goldens/writer-off/web-sessions/identity-registry/fold-reconcile families all green; tsc cleanNotes
CREATE TABLE IF NOT EXISTSmigrations; existing rows keep working — media rows hydrate lazily, old rows inline as before).incremental-lane.tskeeps the documented 409 contract bit-identical for concurrent senders (pinned by the existing turn-in-flight test) while making ack-then-next-turn ordering durable.