Feat/overlay support by dryajov · Pull Request #94 · durability-labs/archivist-node

dryajov · 2026-02-12T04:27:51Z

This PR introduces several important changes:

It switches to a CAS and Atomic kvstore instead of the old dumb datastore - this now allows consistent cross key operations, something that wasn't possible before.
It introduces the concept of Overlays, which are a fancy way of saying datasets, but more fitting since not all sets of blocks are datasets, i.e. slots
It consolidates expirations by moving them from blocks to the overlay. In the past each block would have a refcount and expiration, which lead to drift and inconsistencies. This changes remove this inconsistencies, blocks keep a refcount and can still be shared across several treeCids - original dataset, protected and verifiable, but allows them to have different lifecycles without stepping on each other. Expirations are handled atomically at the overlay level, so no multi-block updates are needed.

There are more improvements coming after this:

Consistent batched operations which should speed things up even further
Block exchange engine improvements that bring better connection handling and batched block transfers
True parallelized encoding in erasure coding (several encoding/decoding jobs running at the same time)
Parallel merkle tree building (tree splitting and merkelezation in parallel)
And probably a few more but those should improve speed and stability across the board

The change is quite big due to its crosscutting nature.

This depends on durability-labs/nim-kvstore#2, durability-labs/archivist-dht#7 and durability-labs/nim-metrics#1

benbierens · 2026-02-17T10:40:58Z

archivist/node.nim

+      leoDecoderProvider, self.taskpool,
+    )
+    encodedManifest = ?await erasure.encode(manifest, ecK, ecM)
+    manifestBlk = ?await self.repoStore.storeManifest(encodedManifest)


Adding manifestBlk = ?await self.repoStore.storeManifest(encodedManifest) causes a test failure.
manifestBlk variable is unused, and storing of the manifest is already performed at node.nim:587.

This extra storeManifest causes the protected-manifest to be stored in addition to the verifiable manifest. This isn't necessary since all marketplace and EC operations have been working with verifiable manifests. In the current flow, the protected manifest is never persisted. It's just an intermediate stage of going from basic to verifiable. We have no usecase where protected manifests are used.

The test fails because it expects to find the basic manifest and the verifiable manifest stored in the node. Instead of 2 it finds 3, but two of them have the exact same information (as far as the API is concerned) just a different CID.

(Test failure: DeletesExpiredDataUsedByStorageRequests at 8ba752c)

There is no harm of storing the protected manifest, even tho the verifiable manifest is an extension of the protected and the top level treeCid is is the same, for consistency reasons it might be fine to store the protected manifest - tho its not critical and I'll remove it for now...

benbierens · 2026-02-17T12:52:34Z

Upload performance has decreased by more than 10x. Here are some numbers:

7e8c0c8 - Feat/make submodules great again (#96)

Size	Upload	Download
1MB	(26 ms)	(112 ms)
10MB	(87 ms)	(111 ms)
100MB	(756 ms)	(432 ms)
1GB	(6 secs)	(4 secs)
10GB	(1 mins, 9 secs)	(43 secs)

1ad57bf - format

Size	Upload	Download
1MB	(286 ms)	(108 ms)
10MB	(2 secs)	(212 ms)
100MB	(20 secs)	(1 secs)
1GB	(4 mins, 56 secs)	(12 secs)
10GB	( > 30 mins)

I would expect the erasure-coding, and blockexchange performance to be affected as well since they also involve adding blocks to local storage. But I have no evidence of this because I haven't run the tests at this time.

(the above test involves a single node. Download numbers do not include blockexchange.)

benbierens · 2026-02-17T15:00:24Z

archivist/node.nim

    self: ArchivistNodeRef, manifestCid: Cid, expiry: SecondsSince1970
 ): Future[?!void] {.async: (raises: [CancelledError]).} =
-  without manifest =? await self.fetchManifest(manifestCid, expiry), error:
+  without manifest =? await self.fetchManifest(manifestCid), error:


Blocks for expired storage requests are not being cleaned up correctly. Blocks are downloaded as part of slots. The slots are successfully filled, but the request fails to start. (This is part of the test.) We expect the data blocks and the manifest of the failed request to be cleaned up in sync with the request's expiry as provided by the marketplace storeSlot callback.

Overlay does become dropped and reached "Deleting" state.

TRC 2026-02-17 14:25:43.169+00:00 Dropping overlay topics="archivist maintenance" tid=1 treeCid=zE2*VghTLV status=Failure expiry=1771424733 count=14647 TRC 2026-02-17 14:25:43.169+00:00 Dropping overlay and cleaning up blocks topics="archivist repostore overlays" tid=1 treeCid=zE2*VghTLV count=14648 TRC 2026-02-17 14:25:43.169+00:00 Overlay metadata stored topics="archivist repostore overlays" tid=1 treeCid=zE2*VghTLV status=Deleting count=14649

(Problem revealed by tests in DeceptiveContractTest at 8ba752c)

thanks I'll check it out - that is the point of this change, so...

dryajov · 2026-02-17T18:38:50Z

Upload performance has decreased by more than 10x. Here are some numbers:

7e8c0c8 - Feat/make submodules great again (#96)

Size Upload Download
1MB (26 ms) (112 ms)
10MB (87 ms) (111 ms)
100MB (756 ms) (432 ms)
1GB (6 secs) (4 secs)
10GB (1 mins, 9 secs) (43 secs)
1ad57bf - format

Size Upload Download
1MB (286 ms) (108 ms)
10MB (2 secs) (212 ms)
100MB (20 secs) (1 secs)
1GB (4 mins, 56 secs) (12 secs)
10GB ( > 30 mins)
I would expect the erasure-coding, and blockexchange performance to be affected as well since they also involve adding blocks to local storage. But I have no evidence of this because I haven't run the tests at this time.

(the above test involves a single node. Download numbers do not include blockexchange.)

Yeah, looking into this... there will be some overhead due to CAS and atomic operations, but it shouldn't be this much.

benbierens · 2026-02-20T17:06:27Z

There seems to be a crash. It's somewhere in the area of receiving/downloading blocks from peers and storing them, when quota runs out. It may not reproduce reliably... I need to test more. Which I will!

TRC 2026-02-20 16:38:06.814+00:00 Updating counters                          topics="archivist repostore" tid=1 quotaDelta=65536 reservedDelta=0 blocksDelta=1 count=765
TRC 2026-02-20 16:38:06.814+00:00 Updating block count to                    topics="archivist repostore" tid=1 totalBlocks=19 count=766
TRC 2026-02-20 16:38:06.814+00:00 Updating quota to                          topics="archivist repostore" tid=1 quotaUsed=1179775'NByte quotaReserved=0'NByte count=767
TRC 2026-02-20 16:38:06.814+00:00 Storing Leafs and Blocks                   topics="archivist repostore" tid=1 treeCid=zDz*LLZqHD totalItems=1 count=768
TRC 2026-02-20 16:38:06.814+00:00 Putting blocks                             topics="archivist repostore" tid=1 actualBlocks=1 totalSize=65536 treeCid=zDz*LLZqHD totalItems=1 count=769
ERR 2026-02-20 16:38:06.814+00:00 Unhandled exception in async proc, aborting topics="archivist" tid=1 msg="value out of range: -131199 notin 0 .. 9223372036854775807" count=770

dryajov · 2026-02-20T19:25:33Z

There seems to be a crash. It's somewhere in the area of receiving/downloading blocks from peers and storing them, when quota runs out. It may not reproduce reliably... I need to test more. Which I will!

TRC 2026-02-20 16:38:06.814+00:00 Updating counters                          topics="archivist repostore" tid=1 quotaDelta=65536 reservedDelta=0 blocksDelta=1 count=765
TRC 2026-02-20 16:38:06.814+00:00 Updating block count to                    topics="archivist repostore" tid=1 totalBlocks=19 count=766
TRC 2026-02-20 16:38:06.814+00:00 Updating quota to                          topics="archivist repostore" tid=1 quotaUsed=1179775'NByte quotaReserved=0'NByte count=767
TRC 2026-02-20 16:38:06.814+00:00 Storing Leafs and Blocks                   topics="archivist repostore" tid=1 treeCid=zDz*LLZqHD totalItems=1 count=768
TRC 2026-02-20 16:38:06.814+00:00 Putting blocks                             topics="archivist repostore" tid=1 actualBlocks=1 totalSize=65536 treeCid=zDz*LLZqHD totalItems=1 count=769
ERR 2026-02-20 16:38:06.814+00:00 Unhandled exception in async proc, aborting topics="archivist" tid=1 msg="value out of range: -131199 notin 0 .. 9223372036854775807" count=770

Keep in mind that I'm still in the process of improving perf. The degradation from CAS should be within 10-20%, but not 10x...

I'll look into the crash - thanks!

benbierens · 2026-03-02T14:15:59Z

Reran the 2GB upload test, just to keep an eye on performance.
Main ccbc239 - Revert "feat: re-enable http pipelining for json-rpc" = (12 secs)
Branch 0109665 - bump kvstore = (2 mins, 45 secs)

On previous commits of this branch, performance was so slow that the test would time out and fail. So performance has definitely improved. It's still far behind when compared to main.

markspanbroek · 2026-03-02T15:15:04Z

I'm currently reviewing this. I've reviewed about 20% of the files in this PR (excluding the dependency PRs) in one day, so this is going to take a while 😅

dryajov · 2026-03-02T18:49:37Z

Reran the 2GB upload test, just to keep an eye on performance. Main ccbc239 - Revert "feat: re-enable http pipelining for json-rpc" = (12 secs) Branch 0109665 - bump kvstore = (2 mins, 45 secs)

On previous commits of this branch, performance was so slow that the test would time out and fail. So performance has definitely improved. It's still far behind when compared to main.

Apparently, there is something going on under Linux, I'm getting abnormally slow uploads, so I'm still looking into it. On Mac, this is about %20-%30 faster than our current main.

markspanbroek

Thanks @dryajov, this is a much-needed change that ensures that concurrent updates to the repostore are handled correctly.

What I'm still missing is a way to handle multiple marketplace requests to store the same slot data, for instance when someone posts a new request for data that's about to expire on the network so that it stays on the network. These requests have different expiries for the slot data. But that's probably not something to address in this PR.

Also, the verbosity of the update mechanism in the kvstore makes this PR hard to read. Most updates boil down to the following form:

?await store.update(key, value):
  value.foo = bar
  value.baz = value.baz + qux

But you have to look through a lot of boilerplate to see this.

markspanbroek · 2026-03-02T10:12:35Z

.github/workflows/ci-reusable.yml

 jobs:
  build:
    strategy:
+      fail-fast: false


Probably a left-over from testing? I would not disable fail-fast in general, because we have too few github runners to use them on jobs in PRs that are failing anyway.

yeah, I'll clean it up before merge, but I need this to be able to see what breaks on which platform.

archivist/blockexchange/engine/advertiser.nim

archivist/blockexchange/engine/engine.nim

tests/archivist/marketplace/sales/mockstorage.nim

markspanbroek · 2026-03-05T14:56:36Z

tests/archivist/stores/repostore/testoverlays.nim

+    let inconsistencies = (await repo.verifyBlockBitState(treeCid)).tryGet()
+    check inconsistencies.len == 0
+
+  test "Concurrent put and delete operations maintain consistency":


I'm missing a test where there's concurrent putOverlay() and dropOverlay() for the same cid

good thinking!

ok, I reworked a lot of the tests, bottom line is that concurrent delete/puts are now handled properly. If an overlay is marked as deleted, which means that delete is in progress, puts to that overlay are no longer possible until the delete is stopped or finished.

note that "stopping the delete" is possible, because we're using a delete future handle, but it hasn't been fully exposed yet. I'll do this in subsequent iterations - this PR is way too big already.

markspanbroek · 2026-03-05T15:03:03Z

tests/archivist/stores/testrepostore.nim

  var
-    repoDs: Datastore
-    metaDs: Datastore
+    path = currentSourcePath() # get this file's name


It's probably nicer to use a temporary directory, instead of putting it next to the test sources. You can use createTempDir.

markspanbroek · 2026-03-05T15:23:42Z

tests/integration/30_minutes/testvalidator.nim


  test "validator marks proofs as missing":
-    let node = await testbed.node.persistence.start()
+    let node = await testbed.node.log("archivist", "validator").persistence.start()


I'm guessing that these are left-overs from some testing that you did? They probably shouldn't be committed.

For all integration tests, I've made sure that logging is disabled by default, so that when you enable it for some test on your local machine, you don't need to search through unrelated logs.

Also, when you start a validator, the "validator" log topic is added automatically.

markspanbroek · 2026-03-05T15:28:42Z

tests/testbed/builders/node.nim

    arguments.add("--block-ttl=" & $blockTtl)
  if blockMaintenanceInterval =? builder.blockMaintenanceInterval:
    arguments.add("--block-mi=" & $blockMaintenanceInterval)
+  arguments.add("--circuit-dir=" & builder.dataDirResolved / "circuits")


What is the reason for adding this? The --circom-r1cs, --circom-wasm, --circom-zkey and --circom-graph are already set above, and they use a different circuit directory (the one in hardhat, to match the circuit that is used for the smart contracts)

Without this, it will look for circuit files in the OS common directory first, if you ever ran archivist without a --data-dir passed, you would have them downloaded there and it obviously breaks the tests.

Testbed always passes --data-dir to a node:

archivist-node/tests/testbed/network/node.nim

Line 67 in ccbc239

arguments &= "--data-dir=" & $node.dataDir

reason: nimble is very inconsistent with larger dependency graphs, and doesn't allow overriding of stint with our forked version

Co-authored-by: markspanbroek <mark@spanbroek.net>

markspanbroek

The recent changes look good, thanks!

I'm approving this, so that you can merge it as soon as you've addressed the remaining review comments.

dryajov force-pushed the feat/overlay-support branch 3 times, most recently from 86b7062 to 4abbbea Compare February 15, 2026 08:51

dryajov marked this pull request as ready for review February 16, 2026 20:47

benbierens reviewed Feb 17, 2026

View reviewed changes

dryajov force-pushed the feat/overlay-support branch from 2bb7721 to 5b1513e Compare February 17, 2026 22:37

dryajov force-pushed the feat/overlay-support branch from 871859f to a6ccdb1 Compare February 27, 2026 22:30

markspanbroek reviewed Mar 5, 2026

View reviewed changes

dryajov and others added 14 commits March 5, 2026 11:59

wip

5cf67f0

fix(nimble): use submodules instead of nimble dependencies

df08dfc

reason: nimble is very inconsistent with larger dependency graphs, and doesn't allow overriding of stint with our forked version

wip: reworking with kvstore and overlays

4d71ea0

feat: fixing bugs and minor cleanup

fcbfd68

wip

b0846e3

feat: add overlay operations

9a91d70

rename keys

5606acb

feat: add putLeafsAndBlocks support

24b1567

misc

cb88ae7

add hash support for block types

aba5548

wip tests

1ed2914

use cid for hashing

bbce126

feat: handle bitseqs and duplicate blocks with different leafs

b1f18fe

mic cleanup

4505b73

dryajov and others added 10 commits March 5, 2026 11:59

format

eebca84

bump kvstore

9766b83

Update archivist/blockexchange/engine/advertiser.nim

b7b7b00

Co-authored-by: markspanbroek <mark@spanbroek.net>

Update archivist/blockexchange/engine/advertiser.nim

ec21319

Co-authored-by: markspanbroek <mark@spanbroek.net>

Update archivist/blockexchange/engine/advertiser.nim

d3d0c60

Co-authored-by: markspanbroek <mark@spanbroek.net>

Update tests/archivist/marketplace/sales/mockstorage.nim

94da497

Co-authored-by: markspanbroek <mark@spanbroek.net>

Update archivist/blockexchange/engine/engine.nim

b32a0bc

Co-authored-by: markspanbroek <mark@spanbroek.net>

feat: fixup grafana dashboard

3eee970

fix: fixup metrics

026c6bc

addressing review comments

930d1f4

dryajov force-pushed the feat/overlay-support branch from faeb0ce to 930d1f4 Compare March 5, 2026 19:33

dryajov added 18 commits March 5, 2026 13:36

addressing review comments

99b5ee8

remove log statments

d86cae0

feat: move verifyBlockBitState to test helpers

81d7171

feat: use checkBitmask more consistently

12530ca

feat: move to test helpers

1cbdbc5

formatting

626b01a

feat: batch proofItems

1cdadad

feat: add catchAsync to correctly propagate cancellations

9934a25

bump kvstore

cc14cfc

feat: don't doublecount batches

73dd3c1

fix: store the recovered data under the correct treeCid

de8116e

fix: don't decode invalid status enum vals

798f8fd

fix: don't decode invalid Naturals & enum vals

3a398d4

fix: invalidate cache and use catchAsync

2fda741

feat: delete only when refCount == 0

1dad3a2

feat: don't allow putting blocks for an overlay that is deleted

f79d000

feat: refactor repostore tests

dff0f52

fix: delete manifest blocks from repoDs directly.

836a855

markspanbroek approved these changes Mar 9, 2026

View reviewed changes

Conversation

dryajov commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benbierens Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benbierens commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

benbierens Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dryajov commented Feb 17, 2026

Uh oh!

benbierens commented Feb 20, 2026

Uh oh!

dryajov commented Feb 20, 2026

Uh oh!

benbierens commented Mar 2, 2026

Uh oh!

markspanbroek commented Mar 2, 2026

Uh oh!

dryajov commented Mar 2, 2026

Uh oh!

markspanbroek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markspanbroek left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dryajov commented Feb 12, 2026 •

edited

Loading

benbierens Feb 17, 2026 •

edited

Loading

benbierens commented Feb 17, 2026 •

edited

Loading

benbierens Feb 17, 2026 •

edited

Loading