Skip to content

Commit c256185

Browse files
thomhurstthomhurst
authored
chore: update benchmark results (#4498)
1 parent 1cb60e2 commit c256185

18 files changed

+366
-362
lines changed

‎docs/docs/benchmarks/AsyncTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 2
77
# AsyncTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.11.51 | 514.6 ms | 515.4 ms | 3.22 ms |
20-
| NUnit | 4.4.0 | 713.0 ms | 712.3 ms | 9.47 ms |
21-
| MSTest | 4.0.2 | 648.6 ms | 649.3 ms | 4.46 ms |
22-
| xUnit3 | 3.2.2 | 778.0 ms | 777.2 ms | 8.58 ms |
23-
| **TUnit (AOT)** | 1.11.51 | 122.6 ms | 122.7 ms | 0.28 ms |
19+
| **TUnit** | 1.11.64 | 510.6 ms | 510.8 ms | 3.62 ms |
20+
| NUnit | 4.4.0 | 683.8 ms | 683.5 ms | 12.08 ms |
21+
| MSTest | 4.0.2 | 624.1 ms | 625.0 ms | 8.34 ms |
22+
| xUnit3 | 3.2.2 | 748.3 ms | 749.6 ms | 7.78 ms |
23+
| **TUnit (AOT)** | 1.11.64 | 122.7 ms | 122.8 ms | 0.49 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
5858
xychart-beta
5959
title "AsyncTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 934
62-
bar [514.6, 713, 648.6, 778, 122.6]
61+
y-axis "Time (ms)" 0 --> 898
62+
bar [510.6, 683.8, 624.1, 748.3, 122.7]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-01-17T00:29:09.006Z*
75+
*Last generated: 2026-01-18T00:32:26.259Z*

‎docs/docs/benchmarks/BuildTime.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 8
77
# Build Performance Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -18,10 +18,10 @@ Compilation time comparison across frameworks:
1818

1919
| Framework | Version | Mean | Median | StdDev |
2020
|-----------|---------|------|--------|--------|
21-
| **TUnit** | 1.11.51 | 1.961 s | 1.958 s | 0.0142 s |
22-
| Build_NUnit | 4.4.0 | 1.564 s | 1.560 s | 0.0155 s |
23-
| Build_MSTest | 4.0.2 | 1.645 s | 1.647 s | 0.0118 s |
24-
| Build_xUnit3 | 3.2.2 | 1.574 s | 1.568 s | 0.0211 s |
21+
| **TUnit** | 1.11.64 | 1.827 s | 1.833 s | 0.0352 s |
22+
| Build_NUnit | 4.4.0 | 1.483 s | 1.488 s | 0.0256 s |
23+
| Build_MSTest | 4.0.2 | 1.545 s | 1.548 s | 0.0152 s |
24+
| Build_xUnit3 | 3.2.2 | 1.451 s | 1.451 s | 0.0095 s |
2525

2626
## 📈 Visual Comparison
2727

@@ -60,7 +60,7 @@ xychart-beta
6060
title "Build Time Comparison"
6161
x-axis ["Build_TUnit", "Build_NUnit", "Build_MSTest", "Build_xUnit3"]
6262
y-axis "Time (s)" 0 --> 3
63-
bar [1.961, 1.564, 1.645, 1.574]
63+
bar [1.827, 1.483, 1.545, 1.451]
6464
```
6565

6666
---
@@ -69,4 +69,4 @@ xychart-beta
6969
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7070
:::
7171

72-
*Last generated: 2026-01-17T00:29:09.009Z*
72+
*Last generated: 2026-01-18T00:32:26.261Z*

‎docs/docs/benchmarks/DataDrivenTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 3
77
# DataDrivenTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.11.51 | 448.59 ms | 446.24 ms | 7.606 ms |
20-
| NUnit | 4.4.0 | 573.81 ms | 573.61 ms | 8.975 ms |
21-
| MSTest | 4.0.2 | 588.86 ms | 590.31 ms | 10.409 ms |
22-
| xUnit3 | 3.2.2 | 620.75 ms | 618.07 ms | 7.196 ms |
23-
| **TUnit (AOT)** | 1.11.51 | 22.29 ms | 22.04 ms | 0.511 ms |
19+
| **TUnit** | 1.11.64 | 448.08 ms | 447.38 ms | 4.418 ms |
20+
| NUnit | 4.4.0 | 550.25 ms | 549.76 ms | 7.142 ms |
21+
| MSTest | 4.0.2 | 535.00 ms | 532.09 ms | 20.843 ms |
22+
| xUnit3 | 3.2.2 | 596.20 ms | 599.26 ms | 7.776 ms |
23+
| **TUnit (AOT)** | 1.11.64 | 22.53 ms | 22.47 ms | 0.486 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
5858
xychart-beta
5959
title "DataDrivenTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 745
62-
bar [448.59, 573.81, 588.86, 620.75, 22.29]
61+
y-axis "Time (ms)" 0 --> 716
62+
bar [448.08, 550.25, 535, 596.2, 22.53]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-01-17T00:29:09.007Z*
75+
*Last generated: 2026-01-18T00:32:26.260Z*

‎docs/docs/benchmarks/MassiveParallelTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 4
77
# MassiveParallelTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.11.51 | 544.4 ms | 544.8 ms | 2.13 ms |
20-
| NUnit | 4.4.0 | 1,221.7 ms | 1,221.5 ms | 8.34 ms |
21-
| MSTest | 4.0.2 | 2,975.5 ms | 2,975.6 ms | 4.50 ms |
22-
| xUnit3 | 3.2.2 | 3,099.3 ms | 3,098.9 ms | 8.68 ms |
23-
| **TUnit (AOT)** | 1.11.51 | 127.8 ms | 127.9 ms | 0.48 ms |
19+
| **TUnit** | 1.11.64 | 563.4 ms | 562.7 ms | 4.32 ms |
20+
| NUnit | 4.4.0 | 1,244.6 ms | 1,239.0 ms | 12.08 ms |
21+
| MSTest | 4.0.2 | 2,997.2 ms | 2,997.9 ms | 5.00 ms |
22+
| xUnit3 | 3.2.2 | 3,135.6 ms | 3,134.9 ms | 9.41 ms |
23+
| **TUnit (AOT)** | 1.11.64 | 128.9 ms | 128.9 ms | 0.52 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
5858
xychart-beta
5959
title "MassiveParallelTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 3720
62-
bar [544.4, 1221.7, 2975.5, 3099.3, 127.8]
61+
y-axis "Time (ms)" 0 --> 3763
62+
bar [563.4, 1244.6, 2997.2, 3135.6, 128.9]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-01-17T00:29:09.007Z*
75+
*Last generated: 2026-01-18T00:32:26.260Z*

‎docs/docs/benchmarks/MatrixTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 5
77
# MatrixTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.11.51 | 496.35 ms | 497.59 ms | 4.583 ms |
20-
| NUnit | 4.4.0 | 1,538.33 ms | 1,539.30 ms | 7.566 ms |
21-
| MSTest | 4.0.2 | 1,471.04 ms | 1,470.49 ms | 9.560 ms |
22-
| xUnit3 | 3.2.2 | 1,585.83 ms | 1,585.90 ms | 5.875 ms |
23-
| **TUnit (AOT)** | 1.11.51 | 74.54 ms | 74.62 ms | 0.266 ms |
19+
| **TUnit** | 1.11.64 | 504.74 ms | 505.48 ms | 2.053 ms |
20+
| NUnit | 4.4.0 | 1,541.76 ms | 1,539.88 ms | 10.153 ms |
21+
| MSTest | 4.0.2 | 1,477.12 ms | 1,473.99 ms | 8.968 ms |
22+
| xUnit3 | 3.2.2 | 1,598.46 ms | 1,595.92 ms | 8.635 ms |
23+
| **TUnit (AOT)** | 1.11.64 | 75.80 ms | 75.75 ms | 0.265 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
5858
xychart-beta
5959
title "MatrixTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 1903
62-
bar [496.35, 1538.33, 1471.04, 1585.83, 74.54]
61+
y-axis "Time (ms)" 0 --> 1919
62+
bar [504.74, 1541.76, 1477.12, 1598.46, 75.8]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-01-17T00:29:09.007Z*
75+
*Last generated: 2026-01-18T00:32:26.260Z*

‎docs/docs/benchmarks/ScaleTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 6
77
# ScaleTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.11.51 | 452.51 ms | 452.80 ms | 2.425 ms |
20-
| NUnit | 4.4.0 | 589.00 ms | 585.10 ms | 11.649 ms |
21-
| MSTest | 4.0.2 | 540.50 ms | 542.79 ms | 16.818 ms |
22-
| xUnit3 | 3.2.2 | 611.01 ms | 612.01 ms | 9.157 ms |
23-
| **TUnit (AOT)** | 1.11.51 | 39.57 ms | 39.42 ms | 3.153 ms |
19+
| **TUnit** | 1.11.64 | 456.44 ms | 454.60 ms | 2.931 ms |
20+
| NUnit | 4.4.0 | 609.90 ms | 610.90 ms | 4.504 ms |
21+
| MSTest | 4.0.2 | 597.46 ms | 598.47 ms | 9.649 ms |
22+
| xUnit3 | 3.2.2 | 637.48 ms | 634.86 ms | 10.064 ms |
23+
| **TUnit (AOT)** | 1.11.64 | 39.61 ms | 38.99 ms | 3.458 ms |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
5858
xychart-beta
5959
title "ScaleTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 734
62-
bar [452.51, 589, 540.5, 611.01, 39.57]
61+
y-axis "Time (ms)" 0 --> 765
62+
bar [456.44, 609.9, 597.46, 637.48, 39.61]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-01-17T00:29:09.008Z*
75+
*Last generated: 2026-01-18T00:32:26.261Z*

‎docs/docs/benchmarks/SetupTeardownTests.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 7
77
# SetupTeardownTests Benchmark
88

99
:::info Last Updated
10-
This benchmark was automatically generated on **2026-01-17** from the latest CI run.
10+
This benchmark was automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
1616

1717
| Framework | Version | Mean | Median | StdDev |
1818
|-----------|---------|------|--------|--------|
19-
| **TUnit** | 1.11.51 | 536.3 ms | 536.3 ms | 2.89 ms |
20-
| NUnit | 4.4.0 | 1,127.5 ms | 1,129.5 ms | 8.68 ms |
21-
| MSTest | 4.0.2 | 1,074.2 ms | 1,073.6 ms | 5.35 ms |
22-
| xUnit3 | 3.2.2 | 1,189.4 ms | 1,189.2 ms | 6.03 ms |
23-
| **TUnit (AOT)** | 1.11.51 | NA | NA | NA |
19+
| **TUnit** | 1.11.64 | 575.0 ms | 575.6 ms | 7.51 ms |
20+
| NUnit | 4.4.0 | 1,183.4 ms | 1,182.9 ms | 6.80 ms |
21+
| MSTest | 4.0.2 | 1,136.9 ms | 1,136.2 ms | 12.28 ms |
22+
| xUnit3 | 3.2.2 | 1,263.9 ms | 1,259.3 ms | 11.15 ms |
23+
| **TUnit (AOT)** | 1.11.64 | NA | NA | NA |
2424

2525
## 📈 Visual Comparison
2626

@@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-01-17** from the latest CI
5858
xychart-beta
5959
title "SetupTeardownTests Performance Comparison"
6060
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
61-
y-axis "Time (ms)" 0 --> 1428
62-
bar [536.3, 1127.5, 1074.2, 1189.4, 0]
61+
y-axis "Time (ms)" 0 --> 1517
62+
bar [575, 1183.4, 1136.9, 1263.9, 0]
6363
```
6464

6565
## 🎯 Key Insights
@@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
7272
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
7373
:::
7474

75-
*Last generated: 2026-01-17T00:29:09.008Z*
75+
*Last generated: 2026-01-18T00:32:26.261Z*

‎docs/docs/benchmarks/index.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebar_position: 1
77
# Performance Benchmarks
88

99
:::info Last Updated
10-
These benchmarks were automatically generated on **2026-01-17** from the latest CI run.
10+
These benchmarks were automatically generated on **2026-01-18** from the latest CI run.
1111

1212
**Environment:** Ubuntu Latest • .NET SDK 10.0.102
1313
:::
@@ -37,7 +37,7 @@ These benchmarks compare TUnit against the most popular .NET testing frameworks:
3737

3838
| Framework | Version Tested |
3939
|-----------|----------------|
40-
| **TUnit** | 1.11.51 |
40+
| **TUnit** | 1.11.64 |
4141
| **xUnit v3** | 3.2.2 |
4242
| **NUnit** | 4.4.0 |
4343
| **MSTest** | 4.0.2 |
@@ -80,4 +80,4 @@ These benchmarks run automatically daily via [GitHub Actions](https://github.com
8080
Each benchmark runs multiple iterations with statistical analysis to ensure accuracy. Results may vary based on hardware and test characteristics.
8181
:::
8282

83-
*Last generated: 2026-01-17T00:29:09.009Z*
83+
*Last generated: 2026-01-18T00:32:26.262Z*

‎docs/static/benchmarks/AsyncTests.json‎

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"timestamp": "2026-01-17T00:29:09.006Z",
2+
"timestamp": "2026-01-18T00:32:26.260Z",
33
"category": "AsyncTests",
44
"environment": {
55
"benchmarkDotNetVersion": "BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.3 LTS (Noble Numbat)",
@@ -9,43 +9,43 @@
99
"results": [
1010
{
1111
"Method": "TUnit",
12-
"Version": "1.11.51",
13-
"Mean": "514.6 ms",
14-
"Error": "3.45 ms",
15-
"StdDev": "3.22 ms",
16-
"Median": "515.4 ms"
12+
"Version": "1.11.64",
13+
"Mean": "510.6 ms",
14+
"Error": "4.08 ms",
15+
"StdDev": "3.62 ms",
16+
"Median": "510.8 ms"
1717
},
1818
{
1919
"Method": "NUnit",
2020
"Version": "4.4.0",
21-
"Mean": "713.0 ms",
22-
"Error": "10.68 ms",
23-
"StdDev": "9.47 ms",
24-
"Median": "712.3 ms"
21+
"Mean": "683.8 ms",
22+
"Error": "13.63 ms",
23+
"StdDev": "12.08 ms",
24+
"Median": "683.5 ms"
2525
},
2626
{
2727
"Method": "MSTest",
2828
"Version": "4.0.2",
29-
"Mean": "648.6 ms",
30-
"Error": "4.77 ms",
31-
"StdDev": "4.46 ms",
32-
"Median": "649.3 ms"
29+
"Mean": "624.1 ms",
30+
"Error": "8.91 ms",
31+
"StdDev": "8.34 ms",
32+
"Median": "625.0 ms"
3333
},
3434
{
3535
"Method": "xUnit3",
3636
"Version": "3.2.2",
37-
"Mean": "778.0 ms",
38-
"Error": "9.17 ms",
39-
"StdDev": "8.58 ms",
40-
"Median": "777.2 ms"
37+
"Mean": "748.3 ms",
38+
"Error": "9.96 ms",
39+
"StdDev": "7.78 ms",
40+
"Median": "749.6 ms"
4141
},
4242
{
4343
"Method": "TUnit_AOT",
44-
"Version": "1.11.51",
45-
"Mean": "122.6 ms",
46-
"Error": "0.32 ms",
47-
"StdDev": "0.28 ms",
48-
"Median": "122.7 ms"
44+
"Version": "1.11.64",
45+
"Mean": "122.7 ms",
46+
"Error": "0.53 ms",
47+
"StdDev": "0.49 ms",
48+
"Median": "122.8 ms"
4949
}
5050
]
5151
}

0 commit comments

Comments
 (0)