Skip to content

feat: let nargo test use the new greybox fuzzer#8234

Merged
asterite merged 15 commits intomasterfrom
ab/nargo-test-use-new-fuzzer
Apr 25, 2025
Merged

feat: let nargo test use the new greybox fuzzer#8234
asterite merged 15 commits intomasterfrom
ab/nargo-test-use-new-fuzzer

Conversation

@asterite
Copy link
Copy Markdown
Collaborator

@asterite asterite commented Apr 25, 2025

Description

Problem

Part of #8214

Summary

Now nargo test will use the new fuzzer. For now:

  • The output of the test program (the print and println) calls will never be shown. Previously it did shown with --show-output but, like the first iteration in fix: let nargo test --show-output work for tests with arguments #8206, it showed all output of all runs, which is probably not desired. Given that --show-output didn't work for tests with arguments, this "preserves" that behavior and we can maybe fix it later.
  • It uses some default configurations: the default corpus directory, 1 timeout second and 1 thread (I thought about using the same number of threads as what's passed to nargo test but then we'd get more threads than that as nargo test already runs on tests in a thread pool). Given that previously there was no way to configure these when running nargo test I think it's fine to leave these for now and make them configurable in a later PR.

Then, the usual progress shown when running nargo fuzz (all of the "NEW", "LOOP", etc., outout) isn't shown when running nargo test. For example, for this program:

#[test]
fn fuzz(x: i8) {
    assert(x != 10);
}

we get this output:

$ nargo test
[one] Running 1 test function
[one] Testing fuzz ... FAIL
Execution failed: Failed to solve program: 'Cannot satisfy constraint': <no message>
Failing input: {"x":"0x000000000000000000000000000000000000000000000000000000000000000a"}

error: Failed constraint
  ┌─ src/main.nr:3:12
  │
3 │     assert(x != 10);
  │            -------
  │
  = Call stack:
    1. src/main.nr:3:12


[one] Failures:
     fuzz

[one] 1 test failed

For this program:

#[test]
fn fuzz(_: i8) {}

we get this output:

[one] Running 1 test function
[one] Testing fuzz ... ok
[one] 1 test passed

Additional Context

Documentation

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Compilation Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite Current: 3f16f7e Previous: ac7a20f Ratio
rollup-root 1.806 s 1.384 s 1.30

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench

@asterite asterite requested a review from Rumata888 April 25, 2025 19:08
Copy link
Copy Markdown
Collaborator

@Rumata888 Rumata888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thank you!

@asterite asterite added this pull request to the merge queue Apr 25, 2025
Merged via the queue into master with commit e483531 Apr 25, 2025
115 checks passed
@asterite asterite deleted the ab/nargo-test-use-new-fuzzer branch April 25, 2025 21:53
github-merge-queue bot pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Apr 28, 2025
Automated pull of nightly from the
[noir](https://github.com/noir-lang/noir) programming language, a
dependency of Aztec.
BEGIN_COMMIT_OVERRIDE
feat: let `nargo test` use the new greybox fuzzer
(noir-lang/noir#8234)
chore(docs): add fixed bugs to list
(noir-lang/noir#8201)
fix(parser): do not use `Ident::default()`
(noir-lang/noir#8224)
feat(fuzz): Add experimental comptime code generator
(noir-lang/noir#8207)
fix: let SSA parser parse calls to print
(noir-lang/noir#8232)
feat: added 1, -1 case for inversions in brillig
(noir-lang/noir#8225)
chore: clippy (noir-lang/noir#8220)
fix: don't take implicitly added named generics when checking where
clauses (noir-lang/noir#8184)
chore(docs): Defunctionalization and some minor cleanup
(noir-lang/noir#8217)
feat: add `--no-fuzz` and `--only-fuzz` to `nargo test`
(noir-lang/noir#8213)
chore: add regression test for panic encountered in bigcurve library.
(noir-lang/noir#8211)
fix: validate numeric types when passing them to a FunctionBuilder
(noir-lang/noir#8215)
chore: refactor constrainedness checks on trait impls
(noir-lang/noir#8170)
chore(test): Add smoke test for AST generation
(noir-lang/noir#8048)
chore: bump more dependencies
(noir-lang/noir#8208)
chore: Don't use `i1` in the AST fuzzer
(noir-lang/noir#8204)
fix: Do not panic if RHS constant in division has more bits than the
operand (noir-lang/noir#8197)
chore: bump cargo deps (noir-lang/noir#8205)
fix: Handle truncating constants to 128 bits
(noir-lang/noir#8180)
fix!: disallow the `i1` type
(noir-lang/noir#8172)
chore(fuzz): Make sure `main` makes at least one call
(noir-lang/noir#8202)
fix: Use `get_local_or_global_instruction` in
`try_optimize_array_set_from_previous_get`
(noir-lang/noir#8200)
fix: returns 0 for right shift overflow
(noir-lang/noir#8189)
feat: avoid overflow check when subtracting from a value that is the
maximum for its bitsize (noir-lang/noir#8190)
feat: optimize `checked_to_unchecked` to take into account
multiplications (noir-lang/noir#8188)
chore: document ssa `inline_simple_functions`
(noir-lang/noir#8169)
chore(docs): remove_unreachable SSA pass
(noir-lang/noir#8196)
chore: Change AST fuzzer recursion limit
(noir-lang/noir#8173)
chore(acir): Unify how we display witness indices
(noir-lang/noir#8192)
fix: Detect ABI change of fuzzing harnesses
(noir-lang/noir#8193)
chore(acir): Test that `BLACKBOX::RANGE` display the number of bits
(noir-lang/noir#8191)
fix(debugger): send idle at loop end + fix test
(noir-lang/noir#8187)
fix: Use `get_local_or_global_instruction` when simplifying `IfElse`
(noir-lang/noir#8185)
fix(parser): avoid using `Location::dummy()`
(noir-lang/noir#8178)
chore: Push a bug list (noir-lang/noir#8186)
feat(greybox_fuzzer): Should_fail and should_fail_with
(noir-lang/noir#8118)
chore: move unsigned overflow check from acir/brillig to ssa
(noir-lang/noir#8163)
feat: location tree for debug_info
(noir-lang/noir#7034)
fix: Use `IntegerConstant` for loop boundaries in `unrolling`
(noir-lang/noir#8094)
fix(ssa): Recursive shared Brillig entry points
(noir-lang/noir#8099)
chore: enable '--pedantic-solving' on more tests
(noir-lang/noir#7701)
chore(readme): Update `acvm-repo` READMEs
(noir-lang/noir#8150)
chore(test): add `test_programs` dir for expected-panic tests
(noir-lang/noir#8147)
chore(docs): minor updates in solidity doc
(noir-lang/noir#8160)
feat(debugger): debug test functions
(noir-lang/noir#7958)
chore: migrate recursive proof test to ultrahonk
(noir-lang/noir#8038)
chore: bump glob package (noir-lang/noir#8159)
chore: bump external pinned commits
(noir-lang/noir#8139)
chore(docs): patch web app tutorial in 1.0.0-beta3 versioned docs
(noir-lang/noir#8158)
END_COMMIT_OVERRIDE

---------

Co-authored-by: AztecBot <tech@aztecprotocol.com>
Co-authored-by: Ary Borenszweig <asterite@gmail.com>
Co-authored-by: TomAFrench <tom@tomfren.ch>
github-merge-queue bot pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Apr 28, 2025
Automated pull of nightly from the
[noir](https://github.com/noir-lang/noir) programming language, a
dependency of Aztec.
BEGIN_COMMIT_OVERRIDE
feat: let `nargo test` use the new greybox fuzzer
(noir-lang/noir#8234)
chore(docs): add fixed bugs to list
(noir-lang/noir#8201)
fix(parser): do not use `Ident::default()`
(noir-lang/noir#8224)
feat(fuzz): Add experimental comptime code generator
(noir-lang/noir#8207)
fix: let SSA parser parse calls to print
(noir-lang/noir#8232)
feat: added 1, -1 case for inversions in brillig
(noir-lang/noir#8225)
chore: clippy (noir-lang/noir#8220)
fix: don't take implicitly added named generics when checking where
clauses (noir-lang/noir#8184)
chore(docs): Defunctionalization and some minor cleanup
(noir-lang/noir#8217)
feat: add `--no-fuzz` and `--only-fuzz` to `nargo test`
(noir-lang/noir#8213)
chore: add regression test for panic encountered in bigcurve library.
(noir-lang/noir#8211)
fix: validate numeric types when passing them to a FunctionBuilder
(noir-lang/noir#8215)
chore: refactor constrainedness checks on trait impls
(noir-lang/noir#8170)
chore(test): Add smoke test for AST generation
(noir-lang/noir#8048)
chore: bump more dependencies
(noir-lang/noir#8208)
chore: Don't use `i1` in the AST fuzzer
(noir-lang/noir#8204)
fix: Do not panic if RHS constant in division has more bits than the
operand (noir-lang/noir#8197)
chore: bump cargo deps (noir-lang/noir#8205)
fix: Handle truncating constants to 128 bits
(noir-lang/noir#8180)
fix!: disallow the `i1` type
(noir-lang/noir#8172)
chore(fuzz): Make sure `main` makes at least one call
(noir-lang/noir#8202)
fix: Use `get_local_or_global_instruction` in
`try_optimize_array_set_from_previous_get`
(noir-lang/noir#8200)
fix: returns 0 for right shift overflow
(noir-lang/noir#8189)
feat: avoid overflow check when subtracting from a value that is the
maximum for its bitsize (noir-lang/noir#8190)
feat: optimize `checked_to_unchecked` to take into account
multiplications (noir-lang/noir#8188)
chore: document ssa `inline_simple_functions`
(noir-lang/noir#8169)
chore(docs): remove_unreachable SSA pass
(noir-lang/noir#8196)
chore: Change AST fuzzer recursion limit
(noir-lang/noir#8173)
chore(acir): Unify how we display witness indices
(noir-lang/noir#8192)
fix: Detect ABI change of fuzzing harnesses
(noir-lang/noir#8193)
chore(acir): Test that `BLACKBOX::RANGE` display the number of bits
(noir-lang/noir#8191)
fix(debugger): send idle at loop end + fix test
(noir-lang/noir#8187)
fix: Use `get_local_or_global_instruction` when simplifying `IfElse`
(noir-lang/noir#8185)
fix(parser): avoid using `Location::dummy()`
(noir-lang/noir#8178)
chore: Push a bug list (noir-lang/noir#8186)
feat(greybox_fuzzer): Should_fail and should_fail_with
(noir-lang/noir#8118)
chore: move unsigned overflow check from acir/brillig to ssa
(noir-lang/noir#8163)
feat: location tree for debug_info
(noir-lang/noir#7034)
fix: Use `IntegerConstant` for loop boundaries in `unrolling`
(noir-lang/noir#8094)
fix(ssa): Recursive shared Brillig entry points
(noir-lang/noir#8099)
chore: enable '--pedantic-solving' on more tests
(noir-lang/noir#7701)
chore(readme): Update `acvm-repo` READMEs
(noir-lang/noir#8150)
chore(test): add `test_programs` dir for expected-panic tests
(noir-lang/noir#8147)
chore(docs): minor updates in solidity doc
(noir-lang/noir#8160)
feat(debugger): debug test functions
(noir-lang/noir#7958)
chore: migrate recursive proof test to ultrahonk
(noir-lang/noir#8038)
chore: bump glob package (noir-lang/noir#8159)
chore: bump external pinned commits
(noir-lang/noir#8139)
chore(docs): patch web app tutorial in 1.0.0-beta3 versioned docs
(noir-lang/noir#8158)
END_COMMIT_OVERRIDE

---------

Co-authored-by: AztecBot <tech@aztecprotocol.com>
Co-authored-by: Ary Borenszweig <asterite@gmail.com>
Co-authored-by: TomAFrench <tom@tomfren.ch>
github-merge-queue bot pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Apr 28, 2025
Automated pull of nightly from the
[noir](https://github.com/noir-lang/noir) programming language, a
dependency of Aztec.
BEGIN_COMMIT_OVERRIDE
feat: let `nargo test` use the new greybox fuzzer
(noir-lang/noir#8234)
chore(docs): add fixed bugs to list
(noir-lang/noir#8201)
fix(parser): do not use `Ident::default()`
(noir-lang/noir#8224)
feat(fuzz): Add experimental comptime code generator
(noir-lang/noir#8207)
fix: let SSA parser parse calls to print
(noir-lang/noir#8232)
feat: added 1, -1 case for inversions in brillig
(noir-lang/noir#8225)
chore: clippy (noir-lang/noir#8220)
fix: don't take implicitly added named generics when checking where
clauses (noir-lang/noir#8184)
chore(docs): Defunctionalization and some minor cleanup
(noir-lang/noir#8217)
feat: add `--no-fuzz` and `--only-fuzz` to `nargo test`
(noir-lang/noir#8213)
chore: add regression test for panic encountered in bigcurve library.
(noir-lang/noir#8211)
fix: validate numeric types when passing them to a FunctionBuilder
(noir-lang/noir#8215)
chore: refactor constrainedness checks on trait impls
(noir-lang/noir#8170)
chore(test): Add smoke test for AST generation
(noir-lang/noir#8048)
chore: bump more dependencies
(noir-lang/noir#8208)
chore: Don't use `i1` in the AST fuzzer
(noir-lang/noir#8204)
fix: Do not panic if RHS constant in division has more bits than the
operand (noir-lang/noir#8197)
chore: bump cargo deps (noir-lang/noir#8205)
fix: Handle truncating constants to 128 bits
(noir-lang/noir#8180)
fix!: disallow the `i1` type
(noir-lang/noir#8172)
chore(fuzz): Make sure `main` makes at least one call
(noir-lang/noir#8202)
fix: Use `get_local_or_global_instruction` in
`try_optimize_array_set_from_previous_get`
(noir-lang/noir#8200)
fix: returns 0 for right shift overflow
(noir-lang/noir#8189)
feat: avoid overflow check when subtracting from a value that is the
maximum for its bitsize (noir-lang/noir#8190)
feat: optimize `checked_to_unchecked` to take into account
multiplications (noir-lang/noir#8188)
chore: document ssa `inline_simple_functions`
(noir-lang/noir#8169)
chore(docs): remove_unreachable SSA pass
(noir-lang/noir#8196)
chore: Change AST fuzzer recursion limit
(noir-lang/noir#8173)
chore(acir): Unify how we display witness indices
(noir-lang/noir#8192)
fix: Detect ABI change of fuzzing harnesses
(noir-lang/noir#8193)
chore(acir): Test that `BLACKBOX::RANGE` display the number of bits
(noir-lang/noir#8191)
fix(debugger): send idle at loop end + fix test
(noir-lang/noir#8187)
fix: Use `get_local_or_global_instruction` when simplifying `IfElse`
(noir-lang/noir#8185)
fix(parser): avoid using `Location::dummy()`
(noir-lang/noir#8178)
chore: Push a bug list (noir-lang/noir#8186)
feat(greybox_fuzzer): Should_fail and should_fail_with
(noir-lang/noir#8118)
chore: move unsigned overflow check from acir/brillig to ssa
(noir-lang/noir#8163)
feat: location tree for debug_info
(noir-lang/noir#7034)
fix: Use `IntegerConstant` for loop boundaries in `unrolling`
(noir-lang/noir#8094)
fix(ssa): Recursive shared Brillig entry points
(noir-lang/noir#8099)
chore: enable '--pedantic-solving' on more tests
(noir-lang/noir#7701)
chore(readme): Update `acvm-repo` READMEs
(noir-lang/noir#8150)
chore(test): add `test_programs` dir for expected-panic tests
(noir-lang/noir#8147)
chore(docs): minor updates in solidity doc
(noir-lang/noir#8160)
feat(debugger): debug test functions
(noir-lang/noir#7958)
chore: migrate recursive proof test to ultrahonk
(noir-lang/noir#8038)
chore: bump glob package (noir-lang/noir#8159)
chore: bump external pinned commits
(noir-lang/noir#8139)
chore(docs): patch web app tutorial in 1.0.0-beta3 versioned docs
(noir-lang/noir#8158)
END_COMMIT_OVERRIDE

---------

Co-authored-by: AztecBot <tech@aztecprotocol.com>
Co-authored-by: Ary Borenszweig <asterite@gmail.com>
Co-authored-by: TomAFrench <tom@tomfren.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants