feat: blobs.

[MirandaWood]

The Blobbening

It's happening and I can only apologise.

Follows #8955.

Intro

More detailed stuff below, but the major changes are:

• Publish DA through blobs rather than calldata. This means:
  • No more txs effects hashing
  • Proving that our effects are included in a blob (see below for details) using base -> root rollup circuits
  • Accumulating tx effects in a blob sponge (SpongeBlob)
  • Building blocks by first passing a hint of how many tx effects the block will have
  • Updating forge to handle blobs
  • Tests for all the above

Major Issues

Things that we should resolve before merging:

• Run times massively increased:

  • This is largely because the nr code for blobs is written with the BigNum lib, which uses a lot of unconstrained code then a small amount of constrained code to verify results. Unfortunately this means we cannot simulate circuits containing blobs (currently block-root) using wasm or set nr tests to unconstrained because that causes a stack overflow in brillig.
  • To avoid straight up failures, I've created nr tests which are not unconstrained (meaning rollup-lib tests take 10mins or so to run) and I've forced circuit simulation to run in native ACVM rather than wasm (adding around 1min to any tests that simulate block-root).
  • Yes, there's more! All the above is happening while we only create one blob per block. This is definitely not enough space (we aim for 3 per block), but I imagine tripling the blob nr code would only cause more runtime issues.

• Data retrieval The below will be done in feat(blobs): Integrate beacon chain client/web2 blob getter #9101, and for now we use calldata just to keep the archiver working:

  • The current (interim) solution is to still publish the same block body calldata as before, just so the archiver actually runs. This calldata is no longer verified with the txs effects hash, but is checked (in ts) against the known blob hash, so a mismatch will still throw.
  • The actual blob contents will look different to the body calldata since we will be tightly packing effects and adding length markers before each section (like how log lengths work). I've added to/from methods to aid conversion in data-retrieval to use.

• Blob verification precompile gas Batching blob KZG proofs is being thought about (see Epic: Blobs #8955 for progression):

  • The current approach to verify that the published blob matches the tx effects coming from the rollup is to call the point evaluation precompile for each blob. This costs 50k gas each time, so is not sustainable.
  • We think it's possible to accumulate the KZG proofs used to validate blobs into one. Mike is thinking about this and whether it's doable using nr, so we can call the precompile once per epoch rather than 3 times per block.

General TODOs

Things I'm working on:

• Moving from 1 to 3 blobs per block
  • This will slow everything down massively so I'd prefer to solve the runtime issues before tackling this.
  • It's also going to be relatively complex, because the base rollup will need code to fill from one of three SpongeBlob instances and will need to know how to 'jump' from one full blob to the next at any possible index. Hopefully this does not lead to a jump in gates.

Description

The general maths in nr and replicated across foundation/blob is described here.

Old DA Flow

From the base rollup to L1, the previous flow for publishing DA was:

Nr:

• In the base rollup, take in all tx effects we wish to publish and sha256 hash them to a single value: tx_effects_hash
• This value is propogated up the rollup to the next merge (or block-root) circuit
• Each merge or block-root circuit simply sha256 hashes each 'child' tx_effects_hash from its left and right inputs
• Eventually, at block-root, we have one value: txs_effects_hash which becomes part of the header's content commitment

Ts:

• The txs_effects_hash is checked and propogated through the orchestrator and becomes part of the ts class L2Block in the header
• The actual tx effects to publish become the L2Block's .body
• The publisher sends the serialised block body and header to the L1 block propose function

Sol:

• In propose, we decode the block body and header
• The body is deconstructed per tx into its tx effects and then hashed using sha256, until we have N tx_effects_hashes (mimicing the calculation in the base rollup)
• Each tx_effects_hash is then input as leaves to a wonky tree and hashed up to the root (mimicing the calculation from base to block-root), forming the final txs_effects_hash
• This final value is checked to be equal to the one in the header's content commitment, then stored to be checked against for validating data availability
• *Later, when verifying a rollup proof, we use the above header values as public inputs. If they do not match what came from the circuit, the verification fails.

*NB: With batch rollups, I've lost touch with what currently happens at verification and how we ensure the txs_effects_hash matches the one calculated in the rollup, so this might not be accurate.

New DA Flow

The new flow for publishing DA is:

Nr:

• In the base rollup, we treat tx effects as we treat PartialStateReferences - injecting a hint to the start and end state we expect from processing this base's transaction
• We take all the tx effects to publish and absorb them into the given start SpongeBlob state. We then check the result is the same as the given end state
• Like with PartialStateReferences, each merge or block-root checks that the left input's end blob state is equal to the right input's start blob state
• Eventually, at block-root, we check the above and that the left's start blob state was empty. Now we have a sponge which has absorbed, as a flat array, all the tx effects in the block we wish to publish
• We inject the flat array of effects as a private input, along with the ts calculated blob commitment, and pass them and the sponge to the blob function
• The blob function:
  • Poseidon hashes the flat array of effects, and checks this matches the accumulated sponge when squeezed (this confirms that the flat array is indeed the same array of tx effects propogated from each base)
  • Computes the challenge z by hashing this ^ hash with the blob commitment
  • Evaluates the blob polynomial at z using the flat array of effects in the barycentric formula (more details on the engineering design link above), to return y
• The block-root adds this triple (z, y, and commitment C) to a new array of BlobPublicInputs
• *Like how we handle fees, each block-merge and root merges the left and right input arrays, so we end up with an array of each block's blob info

*NB: this will likely change to accumulating to a single set of values, rather than one per block, and is being worked on by Mike. The above also describes what happens for one blob per block for simplicity (it will actually be 3).

Ts:

• The BlobPublicInputs are checked against the ts calculated blob for each block in the orchestrator
• They form part of a serialised array of bytes called blobInput (plus the expected L1 blobHash and a ts generated KZG proof) sent to L1 to the propose function
• The propose transaction is now a special 'blob transaction' where all the tx effects (the same flat array as dealt with in the rollup) are sent as a sidecar
• *We also send the serialised block body, so the archiver can still read the data back until feat(blobs): Integrate beacon chain client/web2 blob getter #9101

*NB: this will change once we can read the blobs themselves from the beacon chain/some web2 client.

Sol:

• In propose, instead of recalcating the txs_effects_hash, we send the blobInput to a new validateBlob function. *This function:
  • Gathers the real blobHash from the EVM and checks it against the one in blobInput
  • Calls the point evaluation precompile and checks that our z, y, and C indeed correspond to the blob we claim
• We now have a verified link between the published blob and our blobInput, but still need to link this to our rollup circuit:
  • Each set of BlobPublicInputs is extracted from the bytes array and stored against its block number
  • When the root proof is verified, we reconstruct the array of BlobPublicInputs from the above stored values and use them in proof verification
  • If any of the BlobPublicInputs are incorrect (equivalently, if any of the published blobs were incorrect), the proof verification will fail
• To aid users/the archiver in checking their blob data matches a certain block, the EVM blobHash is been added to BlockLog once it has been verified by the precompile

*NB: As above, we will eventually call the precompile just once for many blobs with one set of BlobPublicInputs. This will still be used in verifying the root proof to ensure the tx effects match those from each base.

[MirandaWood]

@ TMNT (or any) team: is there a restriction anywhere on the size of the tx fee?

[LHerskind]

Don't think there really is a limit, there is probably some sensible value, but as base fees might go up a lot in congestion for now would just keep it as is. You can essentially see it as just a more domain specific public state update.

[iAmMichaelConnor]

Amazing!
I think this is mergeable. @LeilaWang, @LHerskind what do you think?

The circuits look good (with well documented todos) (and we'll be updating them for batching of blobs soon anyway).
The smart contracts look good (with well documented todos) (and we know they're not optimised, because we'll be moving to batching of blobs soon anyway).
The typescript is passing tests.
The "boxes" tests have been failing on master for a while, so no need to fix those here.

Ship?

[LHerskind]

There is a bit of naming that should probably be updated around single or multiple blobs, e.g., blobHash vs blobsHash. For the cases where there are multiple, would prefer the blobsHash to be used consistently to make it simpler to follow.

The TxsDecoder should also be pretty unused after this so maybe it can be deleted along with it.

[LHerskind]

Is there an issue for getting rid of this again?

[MirandaWood]

Not yet - made one here: #10147

[LHerskind]

Thanks.
Quick question, should this not be blobsHash (multiple blobs).

[LHerskind]

It is possible to get around this by using something similar to the ProposeArgs. Might be worth considering for the sake of your sanity 🫡

[LHerskind]

Unless you are submitting the proof in the same transaction as you set the values originally you will be paying extra to do this. Cleaning it here is essentially doing like gas-tokens did back in the days before pricing was changed.
Would remove.

[LHerskind]

🫡

[LHerskind]

As earlier, using a struct can be used to work around this while keeping the readability.

[LHerskind]

Is there an issue for this? You don't need to make the work, but if you can add an issue and throw it into #7820 that would be neat 👍

[MirandaWood]

Made one, thanks! #10148

[LHerskind]

The _blobHash should probably also be blobsHash (multiple)

[LHerskind]

Same down here, there is a bunch of blobHash that should probably be blobsHash.

[LHerskind]

Lets get it merged before the fees changes goes in.

[LHerskind]

What is PIs? Public Inputs? Personal Information

[LHerskind]

The EVM blob hash for this block is not correct anymore

[LHerskind]

Anther PIs, think it is public inputs, but would be easier to follow if written out

[LHerskind]

Why this?