Repository navigation
[PR] fix(storage): move auto_vacuum before journal_mode so that newly created index.db is truly INCREMENTAL | fix(storage): 把 auto_vacuum 挪到 journal_mode 之前，新建的 index.db 才真正是 INCREMENTAL - #6324
Open
BBleae wants to merge 1 commit into
Conversation
…eally gets INCREMENTAL
SQLite only honours PRAGMA auto_vacuum while the database file is still
empty. storagePragmas() switched to WAL first, which already writes
page 1, and issued auto_vacuum = INCREMENTAL last, so every newly created
index.db kept auto_vacuum = 0. PRAGMA incremental_vacuum
(gc({vacuum:"incremental"}), `bili storage gc --vacuum incremental`,
?vacuum=incremental) then reclaimed nothing: the freelist only grew and
the file never shrank without the offline full VACUUM.
- storagePragmas(): auto_vacuum is now the first pragma, with a comment
on why it has to stay there.
- gc() doc comment: describes what incremental does and when it is a
no-op (comment only).
- tests/storage-auto-vacuum.test.ts: pragma order; a new index.db reads
auto_vacuum = 2; fill, delete, gc(incremental) empties the freelist and
shrinks the file; a file created the old way stays at 0 until one full
VACUUM converts it.
Existing databases are not migrated: on a non-empty file the pragma only
takes effect at the next full VACUUM.
ranxianglei
changed the title
There was a problem hiding this comment.
Copilot was unable to review this pull request due to rate limiting. Please try again later. See Usage limits for GitHub Copilot for details.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6323
改了什么、为什么
storagePragmas()原来先执行PRAGMA journal_mode = WAL，最后才执行PRAGMA auto_vacuum = INCREMENTAL。SQLite 只在数据库文件还是空的时候才认auto_vacuum，而切到 WAL 时已经写了第 1 页。所以在全新的index.db上这条 pragma 被忽略：文件头里留着auto_vacuum = 0，PRAGMA incremental_vacuum（gc({ vacuum: "incremental" })、bili storage gc --vacuum incremental、?vacuum=incremental）什么都回收不了。空闲页链表（freelist）只增不减，不跑离线的完整VACUUM，文件就不会变小。这不是 #3901 / #3463（PR #3911）的本意，也和 README 里「要在线安全地缩库用
--vacuum incremental」（#4612）对不上。修法是把
auto_vacuum那一行挪到列表最前面，别的都不变。kernel（kernel/**）没动。只用 SQLite 的最小核对（同样两条 pragma，两种顺序各跑一遍；写入 100 个 100 KB 的 blob 再删掉，然后
PRAGMA incremental_vacuum）：走
UnifiedStore（写入 20 个会话、全部删掉，再gc({ vacuum: "incremental" })）：auto_vacuum持久化面（按 §7.4 标出）
auto_vacuum = 2；文件带 pointer-map 页已有的库不迁移
迄今创建的
index.db，只要还没跑过完整的VACUUM，文件头里都是auto_vacuum = 0。用本 PR 打开它不会改变这一点：在非空文件上，这条 pragma 只是给同一连接上的下一次完整VACUUM预设模式。所以对这些文件，--vacuum incremental仍然什么都不做，直到跑一次完整 vacuum（bili storage gc --vacuum full/?vacuum=full）把文件重写一遍；之后就是auto_vacuum = 2，增量回收开始生效。tests/storage-auto-vacuum.test.ts把这两半都钉住了。我没有加自动转换：完整
VACUUM是 O(库大小) 的独占重写，#4612 已经把它定为大库上的离线操作，并且在另一个存活进程持有该数据库时会拒绝启动。想请维护者定：要不要提示已有安装的用户？下面两种我都可以在本 PR 或后续 PR 里加，看你们倾向哪种：
storage gc --vacuum incremental在PRAGMA auto_vacuum读出 0 时打印一行（"this database predates incremental auto-vacuum; run one--vacuum fullwhile no other instance is using it"）。--vacuum incremental那条建议旁边加一句。行为变化说明
index.db的PRAGMA auto_vacuumgc({ vacuum: "incremental" })VACUUM的库上VACUUM之前仍是空操作auto_vacuum的标准开销）改动
src/storage/schema.ts：PRAGMA auto_vacuum = INCREMENTAL挪到storagePragmas()的第一位，加注释说明它为什么必须留在第一位、在已有文件上会怎样。src/storage/store.ts：gc()的文档注释原来说 auto_vacuum "handles freed pages incrementally"，改成实际发生的情况（只改注释）。tests/storage-auto-vacuum.test.ts：4 个用例。auto_vacuum在journal_mode之前）；UnifiedStore.open打开的新index.db读出auto_vacuum = 2；gc({ vacuum: "incremental" })之后空闲页链表清空，文件缩到不到一半；vacuum("full")后变成 2。验证
npm run typecheck、check:unused-exports、check:test-import-graph：均无报错。tests/*.test.ts，Windows / Node 22，网络被沙箱隔离，--test-concurrency=4）：11009 个测试，10904 通过，8 失败，30 取消，67 跳过。失败和取消的这 38 个里，31 个出自 6 个文件（负载下对时序敏感），这 6 个文件串行重跑全部通过。仍然失败的 7 个都和本机环境有关（本机安全软件拦截回环 TLS、本机不在 UTC、沙箱挡了 DNS 和外网），所在代码本 PR 都没碰；我没有在 master 上跑全量做对照。以推上去的 head 上的 CI 为准（§7.2）。相关：#3463、#3901、#3911、#4612、#5599。