[Merged by Bors] - Fix broken Nethermind integration tests by macladson · Pull Request #4836 · sigp/lighthouse

macladson · 2023-10-12T05:39:15Z

Issue Addressed

CI is currently blocked by persistently failing integration tests.

Proposed Changes

Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes.
Also increase the timeout since I had some local timeouts.

michaelsproul · 2023-10-13T05:47:46Z

Thanks @macladson! I hijacked your branch and pushed some changes to get the latest version of Nethermind working.

Enabling Shapella from genesis seems to have done the trick. I nicked some Nethermind config JSON from src/Nethermind/Nethermind.Blockchain.Test/Specs/shanghai_from_genesis.json.

michaelsproul · 2023-10-13T05:50:07Z

testing/execution_engine_integration/src/nethermind.rs

            .join("Release")
            .join("net7.0")
-            .join("Nethermind.Runner")
+            .join("linux-x64")


this part of the path didn't exist on my machine, lets see if it exists on CI 🤞

It no longer runs on macos though :(

path on my machine: testing/execution_engine_integration/execution_clients/nethermind/src/Nethermind/Nethermind.Runner/bin/Release/net7.0/nethermind

I'm deleting it. I think maybe this was why the job was failing

jimmygchen · 2023-10-13T12:47:27Z

I ran into a panic locally due to "would clobber existing tag", and seems like I was running into this issue
.

Adding the --force flag below this line fixed it for me, but could well just be a local issue:

lighthouse/testing/execution_engine_integration/src/build_utils.rs

Lines 65 to 68 in 726d1b0

    
           output_to_result( 
        
               Command::new("git") 
        
                   .arg("fetch") 
        
                   .arg("--tags")

michaelsproul · 2023-10-15T03:35:45Z

Ah, the CI issue is due to us being out of disk space:

You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB

Received request to deprovision: The request was cancelled by the remote provider.

From: https://github.com/sigp/lighthouse/actions/runs/6514115416

michaelsproul · 2023-10-17T06:09:54Z

Pushed a new commit to run the tests on the self-hosted runners.

jimmygchen · 2023-10-18T00:52:03Z

testing/execution_engine_integration/src/nethermind.rs

+    let tests_dir = execution_clients_dir.join("nethermind/src/tests");
+    if let Err(e) = fs::remove_dir_all(tests_dir) {
+        eprintln!("Error while deleting folder: {}", e);
+    }


Wow just realized these are 2.44 GB on my machine, perhaps we don't even need to clone them in the first place? I think we can optionally exclude submodule during checkout:

lighthouse/testing/execution_engine_integration/src/build_utils.rs

Lines 44 to 49 in aee58b0

output_to_result(

Command::new("git")

.arg("submodule")

.arg("update")

.arg("--init")

.arg("--recursive")

jimmygchen

Changes look good! I only have a small comment above wrt excluding the submodule checkout, but happy to address it in a separate PR, as this is already better than what we had

jimmygchen · 2023-10-18T01:23:34Z

bors+

jimmygchen · 2023-10-18T02:21:14Z

Oops missed an r!
bors r+

## Issue Addressed CI is currently blocked by persistently failing integration tests. ## Proposed Changes Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes. Also increase the timeout since I had some local timeouts. Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: antondlr <anton@delaruelle.net> Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

bors · 2023-10-18T02:50:47Z

Build failed:

release-tests-windows

jimmygchen · 2023-10-18T02:57:26Z

bors retry

jimmygchen · 2023-10-18T03:13:10Z

bors r+

bors · 2023-10-18T03:13:11Z

Already running a review

## Issue Addressed CI is currently blocked by persistently failing integration tests. ## Proposed Changes Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes. Also increase the timeout since I had some local timeouts. Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: antondlr <anton@delaruelle.net> Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

bors · 2023-10-18T03:47:21Z

Build failed (retrying...):

release-tests-ubuntu

## Issue Addressed CI is currently blocked by persistently failing integration tests. ## Proposed Changes Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes. Also increase the timeout since I had some local timeouts. Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: antondlr <anton@delaruelle.net> Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

jimmygchen · 2023-10-18T04:06:24Z

.github/workflows/test-suite.yml

  execution-engine-integration-ubuntu:
    name: execution-engine-integration-ubuntu
-    runs-on: ubuntu-latest
+    runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "small"]') || 'ubuntu-latest'  }}


I've just realized that we're one runner short of running the entire workflow in parallel after adding this one, it may be worth considering adding one more runner instance.

bors · 2023-10-18T04:08:42Z

Build failed (retrying...):

execution-engine-integration-ubuntu

## Issue Addressed CI is currently blocked by persistently failing integration tests. ## Proposed Changes Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes. Also increase the timeout since I had some local timeouts. Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: antondlr <anton@delaruelle.net> Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

bors · 2023-10-18T04:40:23Z

Pull request successfully merged into unstable.

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

## Issue Addressed CI is currently blocked by persistently failing integration tests. ## Proposed Changes Use latest Nethermind release and apply the appropriate fixes as there have been breaking changes. Also increase the timeout since I had some local timeouts. Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: antondlr <anton@delaruelle.net> Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

macladson added 2 commits October 12, 2023 16:36

Initial fix

48a57c4

Clippy

a05228f

jmcph4 mentioned this pull request Oct 12, 2023

(fix): HTTP error handling in VC API #4606

Closed

Enable Shapella from genesis

57900aa

michaelsproul added ready-for-review The code is ready for review infra-ci labels Oct 13, 2023

Remove outdated comment

d743448

michaelsproul reviewed Oct 13, 2023

View reviewed changes

Tweak some things blindly

aee58b0

michaelsproul added 2 commits October 17, 2023 17:03

Run exec engine tests on self-hosted runner

b83ed10

Merge remote-tracking branch 'origin/unstable' into fix-nethermind-tests

9e22c61

michaelsproul and others added 7 commits October 17, 2023 17:58

Don't run rust update on runner

488f923

cleanup some disk space

bc75adb

cargo fmt

284d831

run on gh-hosted

f44b9a5

run on self-hosted with go added to path

e4b53a9

fix nethermind v1.21 path

6fc4266

Fix compilation error in release test

cce7ccf

jimmygchen reviewed Oct 18, 2023

View reviewed changes

jimmygchen approved these changes Oct 18, 2023

View reviewed changes

jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Oct 18, 2023

jimmygchen reviewed Oct 18, 2023

View reviewed changes

bors bot changed the title ~~Fix broken Nethermind integration tests~~ [Merged by Bors] - Fix broken Nethermind integration tests Oct 18, 2023

bors bot closed this Oct 18, 2023

jimmygchen mentioned this pull request Oct 18, 2023

Standard Liveness Endpoint #4853

Merged

	output_to_result(
	Command::new("git")
	.arg("submodule")
	.arg("update")
	.arg("--init")
	.arg("--recursive")

Conversation

macladson commented Oct 12, 2023

Issue Addressed

Proposed Changes

Uh oh!

michaelsproul commented Oct 13, 2023

Uh oh!

michaelsproul Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

jimmygchen Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

jimmygchen Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

michaelsproul Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

jimmygchen commented Oct 13, 2023

Uh oh!

michaelsproul commented Oct 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelsproul commented Oct 17, 2023

Uh oh!

jimmygchen Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen left a comment

Choose a reason for hiding this comment

Uh oh!

jimmygchen commented Oct 18, 2023

Uh oh!

jimmygchen commented Oct 18, 2023

Uh oh!

bors bot commented Oct 18, 2023

Uh oh!

jimmygchen commented Oct 18, 2023

Uh oh!

jimmygchen commented Oct 18, 2023

Uh oh!

bors bot commented Oct 18, 2023

Uh oh!

bors bot commented Oct 18, 2023

Uh oh!

jimmygchen Oct 18, 2023

Choose a reason for hiding this comment

Uh oh!

bors bot commented Oct 18, 2023

Uh oh!

bors bot commented Oct 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

michaelsproul commented Oct 15, 2023 •

edited

Loading

jimmygchen Oct 18, 2023 •

edited

Loading