test(performance): Refactor runner

[rlve]

Summary

Refactors performance and reliability test orchestration. We move from a single “all scenarios in one image + shell runner” setup to a base image plus scenario‑driven orchestration. Each scenario now starts and cleans up its own fresh containers, picks the transport (TCP or QUIC) at runtime, and can inject pre/post commands. This makes tests more isolated, easier to iterate on, and less flaky.

Before

1. Built one Docker image containing every scenario.
2. A shell script runner launched 10 containers, streamed logs, and handled cleanup.
3. Scenarios ran sequentially inside reused containers.
4. Hidden/shared state could leak between tests.

After

1. Build a base test image.
2. Scenarios live outside the image and orchestrate their own containers.
3. Transport (TCP or QUIC) is chosen at runtime.
4. Each scenario gets fresh containers, no cross-test state.
5. Entrypoint supports pre- and post-test hooks for setup/teardown actions.

New Directory Structure

performance/
├── runner.nim                    # Orchestrates Docker containers, networking, log streaming
├── scenarios_performance.nim     # Performance test suite
├── scenarios_reliability.nim     # Network reliability test suite
├── types.nim                     # Shared type definitions (stats, charts, CI config)
│
├── test_node/                    # Distributed test node implementation
│   ├── main.nim                  # Test entry point
│   ├── test_node.nim             # Core scenario logic
│   ├── utils.nim                 # Test utilities
│   ├── Dockerfile
│   └── entrypoint.sh             # Supports pre/post test hooks
│
├── scripts/                      # Post-test data processing
│   ├── common.nim                # Shared utilities for chart generation
│   ├── process_docker_stats.nim  # Parse Docker resource usage logs
│   ├── process_latency_stats.nim # Aggregate latency metrics
│   ├── docker_stats_charts.nim   # Generate resource usage charts
│   └── latency_history_chart.nim # Generate latency history charts

[github-actions]

🏁 Performance Summary

Commit: 1325cee0cb101bbbc60df70d05c0135f68bff935

Scenario	Nodes	Total messages sent	Total messages received	Latency min (ms)	Latency max (ms)	Latency avg (ms)
Base Test (TCP)	10	100	900	0.319	2.441	1.103
Base Test (QUIC)	10	100	900	0.271	2.006	0.949

📊 View Container Resources in the Workflow Summary

Latency History

_{🔵 Min • 🟢 Avg • 🔴 Max}

%%{init: {"xyChart": {"width": 500, "height": 200}}}%%
xychart-beta
    title "TCP"
    x-axis "PR Number" 1599 --> 1742
    y-axis "Latency (ms)"
    line "Min" [0.315, 0.429, 0.308, 0.349, 0.397, 0.259, 0.317, 0.333, 0.266, 0.278, 0.322, 0.300, 0.301, 0.302, 0.263, 0.338, 0.288, 0.308, 0.318, 0.313, 0.358, 0.317, 0.274, 0.293, 0.278, 0.312, 0.308, 0.280, 0.296, 0.292, 0.296, 0.296, 0.437, 0.265, 0.444, 0.396, 0.323, 0.305, 0.305, 0.293, 0.333, 0.322, 0.370, 0.273, 0.338, 0.275, 0.318, 0.322, 0.339, 0.278, 0.325, 0.297, 0.275, 0.297, 0.280, 0.277, 0.407, 0.304, 0.329, 0.319, 0.372, 0.289, 0.317, 0.262, 0.298, 0.376, 0.331, 0.257, 0.286, 0.299, 0.267, 0.312, 0.277, 0.240, 0.266, 0.424, 0.260, 0.305, 0.261, 0.364, 0.422, 0.274, 0.297, 0.313, 0.291, 0.281, 0.356, 0.274, 0.222, 0.290, 0.264, 0.319, 0.268, 0.303, 0.246, 0.304, 0.272]
    line "Avg" [1.042, 1.134, 1.015, 1.035, 1.177, 0.899, 1.040, 1.085, 0.860, 1.015, 1.020, 0.945, 0.986, 1.074, 0.899, 0.975, 0.958, 1.027, 0.959, 1.047, 1.103, 1.046, 0.908, 0.882, 0.896, 1.064, 1.011, 0.957, 0.991, 0.923, 0.867, 0.959, 1.126, 0.910, 1.125, 1.085, 0.983, 1.049, 1.053, 1.056, 1.080, 0.989, 1.137, 1.007, 1.062, 0.874, 1.065, 1.137, 1.104, 0.852, 1.080, 0.991, 1.003, 1.017, 0.971, 1.068, 1.026, 1.054, 1.100, 1.054, 1.087, 0.961, 0.886, 0.878, 1.028, 1.247, 1.023, 0.887, 1.015, 0.952, 0.875, 0.985, 1.003, 0.997, 0.982, 1.178, 1.097, 0.980, 0.839, 1.092, 1.110, 0.850, 1.026, 0.996, 0.959, 0.951, 1.083, 0.907, 0.852, 0.933, 0.919, 1.103, 0.932, 1.033, 0.974, 0.901, 0.880]
    line "Max" [2.298, 2.513, 2.647, 2.370, 2.486, 2.258, 2.316, 2.407, 2.065, 3.120, 2.328, 2.193, 2.359, 2.398, 2.082, 2.241, 2.339, 2.187, 2.267, 2.268, 2.480, 2.591, 2.231, 1.950, 2.493, 2.596, 2.270, 2.198, 2.587, 2.539, 1.811, 2.355, 2.373, 2.516, 2.368, 2.412, 2.166, 2.482, 2.401, 2.505, 2.396, 2.211, 2.686, 2.876, 2.527, 2.543, 2.441, 2.495, 2.430, 2.073, 2.354, 2.257, 2.217, 2.562, 2.383, 3.300, 2.337, 2.394, 2.521, 2.316, 2.490, 2.352, 2.218, 2.233, 2.267, 2.836, 2.352, 2.316, 2.371, 2.306, 1.972, 2.273, 2.457, 2.517, 2.295, 2.685, 13.159, 2.661, 1.988, 2.495, 2.309, 2.154, 2.482, 2.433, 2.387, 2.369, 2.335, 2.411, 2.018, 2.145, 2.091, 2.441, 2.129, 2.243, 2.354, 2.225, 6.287]

Loading

%%{init: {"xyChart": {"width": 500, "height": 200}}}%%
xychart-beta
    title "QUIC"
    x-axis "PR Number" 1718 --> 1742
    y-axis "Latency (ms)"
    line "Min" [0.253, 0.000, 0.325, 0.229, 0.000, 0.000, 0.000, 0.223, 0.000, 0.000, 0.000, 0.256, 0.271, 0.203, 0.186, 0.000, 0.219, 0.000]
    line "Avg" [0.825, 0.687, 1.020, 0.856, 0.725, 0.602, 0.683, 0.875, 0.661, 0.557, 0.731, 0.745, 0.949, 0.726, 0.849, 0.761, 0.718, 0.463]
    line "Max" [2.129, 1.504, 1.944, 1.917, 1.681, 1.573, 1.715, 1.857, 1.782, 1.274, 1.783, 1.750, 2.006, 1.943, 2.179, 1.918, 1.755, 1.093]

Loading

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (test-gossipsub-performance-13@e02acc3). Learn more about missing BASE report.

Additional details and impacted files

@@                       Coverage Diff                        @@
##             test-gossipsub-performance-13    #1735   +/-   ##
================================================================
  Coverage                                 ?   82.58%           
================================================================
  Files                                    ?      142           
  Lines                                    ?    22509           
  Branches                                 ?        0           
================================================================
  Hits                                     ?    18589           
  Misses                                   ?     3920           
  Partials                                 ?        0           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[richard-ramos]

Left few comments and nitpicks