fix(l2): checkpoint creation #5321

avilagaston9 · 2025-11-13T16:06:04Z

Motivation

This PR addresses two main issues:

Sealing a batch, storing the prover inputs, and creating the checkpoint for that batch are currently performed non-atomically. If the node restarts during any of these three operations, the committer will end up in an invalid state.
The batch checkpoint is created at the beginning of the building process. If an error occurs while building the batch, the l1_committer gets stuck with the following error:

2025-11-12T15:47:26.129936Z ERROR L1 Committer Error: Committer failed retrieve block from storage: Failed to create RocksDB checkpoint at "dev_ethrex_l2/checkpoint_batch_1": Invalid argument: Directory exists

Description

Stores the batch and prover inputs atomically in the rollup storage.
When the l1_committer encounters a batch generated in a previous iteration, it now checks whether the corresponding checkpoint exists. If it doesn't, the committer creates it by re-executing the batch.

Closes None

avilagaston9 · 2025-11-13T16:47:08Z

crates/l2/sequencer/l1_committer.rs

-    /// Blockchain instance using the current checkpoint store.
-    ///
-    /// It is used for witness generation.
-    current_checkpoint_blockchain: Arc<Blockchain>,


This was unused

github-actions · 2025-11-13T16:47:19Z

Lines of code report

Total lines added: 193
Total lines removed: 5
Total lines changed: 198

Detailed view

+----------------------------------------------------+-------+------+
| File                                               | Lines | Diff |
+----------------------------------------------------+-------+------+
| ethrex/crates/l2/sequencer/l1_committer.rs         | 1202  | +125 |
+----------------------------------------------------+-------+------+
| ethrex/crates/l2/sequencer/utils.rs                | 165   | -5   |
+----------------------------------------------------+-------+------+
| ethrex/crates/l2/storage/src/api.rs                | 134   | +6   |
+----------------------------------------------------+-------+------+
| ethrex/crates/l2/storage/src/store.rs              | 346   | +10  |
+----------------------------------------------------+-------+------+
| ethrex/crates/l2/storage/src/store_db/in_memory.rs | 360   | +15  |
+----------------------------------------------------+-------+------+
| ethrex/crates/l2/storage/src/store_db/sql.rs       | 914   | +26  |
+----------------------------------------------------+-------+------+
| ethrex/crates/storage/api.rs                       | 247   | +1   |
+----------------------------------------------------+-------+------+
| ethrex/crates/storage/store.rs                     | 1497  | +3   |
+----------------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/in_memory.rs        | 633   | +3   |
+----------------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/rocksdb.rs          | 1652  | +4   |
+----------------------------------------------------+-------+------+

avilagaston9 · 2025-11-13T17:30:36Z

crates/l2/sequencer/l1_committer.rs

-        // We need to guarantee that the checkpoint path is new
-        // to avoid causing a lock error under rocksdb feature.
-        let new_checkpoint_path = self
-            .checkpoints_dir
-            .join(batch_checkpoint_name(batch_number));


Creating the batch checkpoint at the beginning of the execution prevents us from recovering from errors, as we can't create a checkpoint on an already used path.

Copilot

Pull Request Overview

This PR addresses critical atomicity and checkpoint creation issues in the L2 sequencer, ensuring that batch sealing, prover input storage, and checkpoint creation happen atomically to prevent invalid committer states during node restarts.

Key Changes

Introduced seal_batch_with_prover_input() method to atomically store batches and prover inputs in a single database transaction
Added get_store_directory() method to enable checkpoint path validation
Implemented check_current_checkpoint() to verify and regenerate missing checkpoints by re-executing batch blocks

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`crates/storage/store_db/rocksdb.rs`	Added `get_store_directory()` to return the RocksDB path
`crates/storage/store_db/in_memory.rs`	Added `get_store_directory()` returning a placeholder path for in-memory store
`crates/storage/store.rs`	Exposed `get_store_directory()` through the Store API
`crates/storage/api.rs`	Added `get_store_directory()` to the StoreEngine trait
`crates/l2/storage/src/store_db/sql.rs`	Implemented atomic `seal_batch_with_prover_input()` using database transactions and extracted `seal_batch_in_tx()` helper
`crates/l2/storage/src/store_db/in_memory.rs`	Implemented `seal_batch_with_prover_input()` with sequential operations (acceptable for in-memory)
`crates/l2/storage/src/store.rs`	Exposed `seal_batch_with_prover_input()` with documentation
`crates/l2/storage/src/api.rs`	Added `seal_batch_with_prover_input()` to StoreEngineRollup trait
`crates/l2/sequencer/utils.rs`	Refactored `fetch_blocks_with_respective_fee_configs()` to accept Batch reference instead of batch number, improving error messages
`crates/l2/sequencer/l1_committer.rs`	Refactored batch production workflow: checkpoint creation moved after atomic batch/prover input storage, added checkpoint validation logic, extracted helper methods for one-time checkpoint management
`Cargo.lock`	Updated dependencies to use standard crates.io versions instead of patched versions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-14T03:57:57Z

crates/l2/sequencer/l1_committer.rs

+        let batch_prover_input = self.generate_batch_prover_input(&batch).await?;
+
+        self.rollup_store
+            .seal_batch_with_prover_input(batch.clone(), &self.git_commit_hash, batch_prover_input)
+            .await?;
+
+        // Create the next checkpoint from the one-time checkpoint used
+        let new_checkpoint_path = self
+            .checkpoints_dir
+            .join(batch_checkpoint_name(batch_number));
+        let (new_checkpoint_store, _) = self
+            .create_checkpoint(
+                &one_time_checkpoint_store,
+                &new_checkpoint_path,
+                &self.rollup_store,
+            )
+            .await?;



Resource leak: If generate_batch_prover_input, seal_batch_with_prover_input, or create_checkpoint fail, the one-time checkpoint at one_time_checkpoint_path is not cleaned up. Consider wrapping this section with error handling similar to lines 545-547 to ensure cleanup on failure.

Suggested change

let batch_prover_input = self.generate_batch_prover_input(&batch).await?;

self.rollup_store

.seal_batch_with_prover_input(batch.clone(), &self.git_commit_hash, batch_prover_input)

.await?;

// Create the next checkpoint from the one-time checkpoint used

let new_checkpoint_path = self

.checkpoints_dir

.join(batch_checkpoint_name(batch_number));

let (new_checkpoint_store, _) = self

.create_checkpoint(

&one_time_checkpoint_store,

&new_checkpoint_path,

&self.rollup_store,

)

.await?;

// Wrap the fallible section in error handling to ensure cleanup on failure

let result = async {

let batch_prover_input = self.generate_batch_prover_input(&batch).await?;

self.rollup_store

.seal_batch_with_prover_input(batch.clone(), &self.git_commit_hash, batch_prover_input)

.await?;

// Create the next checkpoint from the one-time checkpoint used

let new_checkpoint_path = self

.checkpoints_dir

.join(batch_checkpoint_name(batch_number));

let (new_checkpoint_store, _) = self

.create_checkpoint(

&one_time_checkpoint_store,

&new_checkpoint_path,

&self.rollup_store,

)

.await?;

Ok(new_checkpoint_store)

}.await;

let new_checkpoint_store = match result {

Ok(store) => store,

Err(e) => {

// Cleanup the one-time checkpoint on error

let _ = self.remove_one_time_checkpoint(&one_time_checkpoint_path);

return Err(e.into());

}

};

Agree with doing something like this

Copilot · 2025-11-14T03:57:57Z

crates/l2/sequencer/l1_committer.rs

+                &new_checkpoint_path,
+                &self.rollup_store,
+            )
+            .await?;


Resource leak: If create_checkpoint fails, the one-time checkpoint at one_time_checkpoint_path is not cleaned up. Consider adding error handling to ensure cleanup on failure, similar to the pattern used at line 419.

Suggested change

.await?;

.await

.inspect_err(|_| {

let _ = self.remove_one_time_checkpoint(&one_time_checkpoint_path);

})?;

Agree with this, but I think is better to do it once since two lines below we do it any way

tomip01 · 2025-11-14T15:19:58Z

crates/l2/sequencer/l1_committer.rs

+        .inspect_err(|_| {
+            let _ = self.remove_one_time_checkpoint(&one_time_checkpoint_path);
+        })?;


we could log the error here

tomip01 · 2025-11-14T15:40:57Z

crates/l2/sequencer/l1_committer.rs

+                &new_checkpoint_path,
+                &self.rollup_store,
+            )
+            .await?;


Agree with this, but I think is better to do it once since two lines below we do it any way

tomip01 · 2025-11-14T15:44:37Z

crates/l2/sequencer/l1_committer.rs

+        let batch_prover_input = self.generate_batch_prover_input(&batch).await?;
+
+        self.rollup_store
+            .seal_batch_with_prover_input(batch.clone(), &self.git_commit_hash, batch_prover_input)
+            .await?;
+
+        // Create the next checkpoint from the one-time checkpoint used
+        let new_checkpoint_path = self
+            .checkpoints_dir
+            .join(batch_checkpoint_name(batch_number));
+        let (new_checkpoint_store, _) = self
+            .create_checkpoint(
+                &one_time_checkpoint_store,
+                &new_checkpoint_path,
+                &self.rollup_store,
+            )
+            .await?;



Agree with doing something like this

**Motivation** This PR addresses two main issues: - Sealing a batch, storing the `prover inputs`, and creating the checkpoint for that batch are currently performed non-atomically. If the node restarts during any of these three operations, the committer will end up in an invalid state. - The batch checkpoint is created at the beginning of the building process. If an error occurs while building the batch, the `l1_committer` gets stuck with the following error: ``` 2025-11-12T15:47:26.129936Z ERROR L1 Committer Error: Committer failed retrieve block from storage: Failed to create RocksDB checkpoint at "dev_ethrex_l2/checkpoint_batch_1": Invalid argument: Directory exists ``` **Description** - Stores the batch and `prover inputs` atomically in the rollup storage. - When the `l1_committer` encounters a batch generated in a previous iteration, it now checks whether the corresponding checkpoint exists. If it doesn't, the committer creates it by re-executing the batch. Closes None

avilagaston9 added 2 commits November 13, 2025 13:05

fix(l2): checkpoint creation

3f3c48b

Merge branch 'main' into fix/l2/checkpoint_creation

b4d54c3

avilagaston9 commented Nov 13, 2025

View reviewed changes

Refactor using checkpoint helpers

f1aaf0b

avilagaston9 commented Nov 13, 2025

View reviewed changes

Minor improvements

91c4349

github-actions bot added the L2 Rollup client label Nov 13, 2025

avilagaston9 self-assigned this Nov 13, 2025

github-project-automation bot added this to ethrex_l2 Nov 13, 2025

avilagaston9 and others added 4 commits November 13, 2025 16:01

Fix retrieving batch blocks

350ccba

Merge branch 'main' into fix/l2/checkpoint_creation

1bdd791

fix: check if the current checkpoint is updated

f8de1d3

Fix restarts

d61b5da

avilagaston9 marked this pull request as ready for review November 14, 2025 03:53

Copilot AI review requested due to automatic review settings November 14, 2025 03:53

avilagaston9 requested a review from a team as a code owner November 14, 2025 03:53

avilagaston9 moved this to In Review in ethrex_l2 Nov 14, 2025

Copilot started reviewing on behalf of avilagaston9 November 14, 2025 03:53 View session

Copilot finished reviewing on behalf of avilagaston9 November 14, 2025 03:57

Copilot AI reviewed Nov 14, 2025

View reviewed changes

ilitteri approved these changes Nov 14, 2025

View reviewed changes

tomip01 reviewed Nov 14, 2025

View reviewed changes

tomip01 approved these changes Nov 14, 2025

View reviewed changes

tomip01 added this pull request to the merge queue Nov 14, 2025

Merged via the queue into main with commit 8825ef3 Nov 14, 2025
54 checks passed

tomip01 deleted the fix/l2/checkpoint_creation branch November 14, 2025 21:19

github-project-automation bot moved this from In Review to Done in ethrex_l2 Nov 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(l2): checkpoint creation #5321

fix(l2): checkpoint creation #5321

Uh oh!

avilagaston9 commented Nov 13, 2025 •

edited

Loading

Uh oh!

avilagaston9 Nov 13, 2025

Uh oh!

github-actions bot commented Nov 13, 2025 •

edited

Loading

Uh oh!

avilagaston9 Nov 13, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

tomip01 Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

tomip01 Nov 14, 2025 •

edited

Loading

Uh oh!

tomip01 Nov 14, 2025

Uh oh!

tomip01 Nov 14, 2025 •

edited

Loading

Uh oh!

tomip01 Nov 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix(l2): checkpoint creation #5321

fix(l2): checkpoint creation #5321

Uh oh!

Conversation

avilagaston9 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avilagaston9 Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Lines of code report

Uh oh!

avilagaston9 Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

tomip01 Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

tomip01 Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomip01 Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

tomip01 Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomip01 Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

avilagaston9 commented Nov 13, 2025 •

edited

Loading

github-actions bot commented Nov 13, 2025 •

edited

Loading

tomip01 Nov 14, 2025 •

edited

Loading

tomip01 Nov 14, 2025 •

edited

Loading