Parallelize node preloads

[afterdusk]

Overview

This PR adds parallelization for node preloading. This primarily benefits publishing, which performs node preloading prior to node insertion and audit proof generation.

During node preloading, we perform a Breadth First Search on the tree starting from the root, fetching each level of the tree in sequence. This operation's latency is lower bounded since a certain minimum number of back-to-back round trips to DB is always required. However, the amount of work performed between each DB read is not trivial, and can contribute significantly to the overall preload latency when publishing hundreds of thousands of keys.

We can improve preload latency and bring it closer to the theoretical lower bound, by parallelizing the work performed between DB fetches. In this PR, I do so by having having each step in the BFS split the subsequent work for the next level into two chunks, having each chunk be processed concurrently. The storage layer might also benefit from performing DB operations in batches that are not overly large.

Note that node preloading is used in publishing and lookups, and parallelism is always disabled for the lookup path. This is because lookups are likely performed on a machine that services client requests. We do not want a single lookup to consume all the resources on the machine and starve other client requests.

Benchmark

Ran the azks benchmark, on trunk (fcd665a) and on this PR:

cargo bench -p akd --bench azks -F bench

Batch Insertion (1000 Initial Leaves, 1000 Inserted Leaves)

Trunk:

This PR:

Batch Insertion (200,000 Initial Leaves, 200,000 Inserted Leaves)

Trunk:

This PR:

On my 10-core Macbook Pro, we get a decent amount of improvement (~27%). This improvement should be larger on higher core count machines, and at higher load levels.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 99.20635% with 1 line in your changes missing coverage. Please review.

Project coverage is 88.04%. Comparing base (3ce5335) to head (079d2a4).
Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
akd/src/append_only_zks.rs	99.20%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #454      +/-   ##
==========================================
- Coverage   88.61%   88.04%   -0.58%     
==========================================
  Files          39       38       -1     
  Lines        9109     8288     -821     
==========================================
- Hits         8072     7297     -775     
+ Misses       1037      991      -46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[slawlor]

nifty! But I think we could do with making the bool arg an enum for safer usage (As I'm guessing @dillonrg would say 😛 )

[kevinlewi]

LGTM!