Some nginx test runs fail to complete #340
Description
Some nginx test runs fail to complete.
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 1]
Test 1 of 7
Estimated Trial Run Count: 3
Estimated Test Run-Time: 4 Minutes
Estimated Time To Completion: 25 Minutes [15:25 UTC]
Running Pre-Test Script @ 15:01:26
Started Run 1 @ 15:01:35
The test quit with a non-zero exit status.
Started Run 2 @ 15:01:40
The test quit with a non-zero exit status.
Started Run 3 @ 15:01:44
The test quit with a non-zero exit status.
Running Post-Test Script @ 15:01:46
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 20]
Test 2 of 7
Estimated Trial Run Count: 3
Estimated Test Run-Time: 4 Minutes
Estimated Time To Completion: 21 Minutes [15:22 UTC]
Running Pre-Test Script @ 15:01:57
Started Run 1 @ 15:02:02
The test quit with a non-zero exit status.
Started Run 2 @ 15:02:06
The test quit with a non-zero exit status.
Started Run 3 @ 15:02:10
The test quit with a non-zero exit status.
Running Post-Test Script @ 15:02:12
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 100]
Test 3 of 7
Estimated Trial Run Count: 3
Estimated Test Run-Time: 4 Minutes
Estimated Time To Completion: 18 Minutes [15:19 UTC]
Running Pre-Test Script @ 15:02:23
Started Run 1 @ 15:02:28
The test quit with a non-zero exit status.
Started Run 2 @ 15:02:32
The test quit with a non-zero exit status.
Started Run 3 @ 15:02:36
The test quit with a non-zero exit status.
Running Post-Test Script @ 15:02:38
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 200]
Test 4 of 7
Estimated Trial Run Count: 3
Estimated Test Run-Time: 4 Minutes
Estimated Time To Completion: 14 Minutes [15:16 UTC]
Running Pre-Test Script @ 15:02:49
Started Run 1 @ 15:02:54
The test quit with a non-zero exit status.
Started Run 2 @ 15:02:58
The test quit with a non-zero exit status.
Started Run 3 @ 15:03:02
The test quit with a non-zero exit status.
Running Post-Test Script @ 15:03:04
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 500]
Test 5 of 7
Estimated Trial Run Count: 3
Estimated Test Run-Time: 5 Minutes
Estimated Time To Completion: 11 Minutes [15:13 UTC]
Running Pre-Test Script @ 15:03:15
Started Run 1 @ 15:03:20
Started Run 2 @ 15:04:55
Started Run 3 @ 15:06:29
Running Post-Test Script @ 15:08:02
Connections: 500:
94410.2
93610.17
92799.85
Average: 93606.74 Requests Per Second
Deviation: 0.86%
Comparison of 2,643 OpenBenchmarking.org samples since 9 September 2022; median result: 97908 Requests Per Second. Box plot of samples:
[|*--------------###################*#!*#################*##*#*#*###----------------------*-------*-*---------*-----------------------------------------------------*----------------*-----------*---|* ]
^ This Result (47th Percentile): 93607
AMD EPYC 4245P: 101491 ^ AMD EPYC 7643: 167324 ^ Intel Xeon Gold 6421N: 255110 ^ AMD Ryzen Threadripper 9970X: 427978 ^ AMD EPYC 9755: 516074 ^
AMD EPYC 4465P: 162377 ^ ARMv8 Neoverse-V1: 288066 ^ AMD EPYC 9965: 503520 ^
AMD EPYC 4584PX: 156140 ^ AMD Ryzen 9 9950X3D: 233810 ^ AMD EPYC 9845: 471521 ^
^ ARMv8 Cortex-A76: 6117 Intel Xeon Gold 6342: 149161 ^ Intel Xeon Max 9480: 260506 ^
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 1000]
Test 6 of 7
Estimated Trial Run Count: 3
Estimated Test Run-Time: 2 Minutes
Estimated Time To Completion: 6 Minutes [15:13 UTC]
Running Pre-Test Script @ 15:08:18
Started Run 1 @ 15:08:23
Started Run 2 @ 15:09:58
The test quit with a non-zero exit status.
Started Run 3 @ 15:10:08
Started Run 4 @ 15:11:42 *
The test quit with a non-zero exit status.
E: Segmentation fault (core dumped)
Running Post-Test Script @ 15:11:50
Connections: 1000:
90011.31
89066.19
Average: 89538.75 Requests Per Second
Samples: 2
Comparison of 2,208 OpenBenchmarking.org samples since 9 September 2022; median result: 86582 Requests Per Second. Box plot of samples:
[ |-------------------########################*!*#######*###*####*####*###########------------------*-------*-*-----------------------*------------------*--*--------------------| * * ]
This Result (52nd Percentile): 89539 ^
AMD EPYC 4344P: 84907 ^ AMD EPYC 7413: 129683 ^ 2 x Intel Xeon Gold 6342: 200468 ^ ARMv8 Neoverse-V1: 289806 ^ 2 x AMD EPYC 9755: 379062 ^
AMD EPYC 4484PX: 119800 ^ Intel Xeon Platinum 8480+: 283883 ^ 2 x AMD EPYC 9965: 358843 ^
AMD EPYC 4464P: 110850 ^ Intel Core i9-14900K: 185258 ^ Intel Xeon Gold 6421N: 248638 ^
AMD EPYC 4345P: 103583 ^ AMD EPYC 4585PX: 203974 ^
nginx 1.23.2:
pts/nginx-3.0.1 [Connections: 4000]
Test 7 of 7
Estimated Trial Run Count: 3
Estimated Time To Completion: 5 Minutes [15:16 UTC]
Running Pre-Test Script @ 15:12:06
Started Run 1 @ 15:12:11
The test quit with a non-zero exit status.
Started Run 2 @ 15:12:18
The test quit with a non-zero exit status.
Started Run 3 @ 15:12:24
The test quit with a non-zero exit status.
E: Segmentation fault (core dumped)
Running Post-Test Script @ 15:12:28
The following tests failed to properly run:
- pts/nginx-3.0.1: Connections: 1
- pts/nginx-3.0.1: Connections: 20
- pts/nginx-3.0.1: Connections: 100
- pts/nginx-3.0.1: Connections: 200
- pts/nginx-3.0.1: Connections: 4000
This wasn't easy to figure out, but when running test manually:
$ cd ~/.phoronix-test-suite/installed-tests/pts/nginx-3.0.1/
$ ./nginx_/sbin/nginx -g "worker_processes auto;"
$ LOG_FILE=/tmp/nginx.log NUM_CPU_CORES=256 ./nginx -d 90s -c 1 https://127.0.0.1:8089/test.html
I saw in log file, so basically it's pointless to run these low connection tests on such high core CPUs
number of connections must be >= threads
Usage: wrk <options> <url>
Options:
-c, --connections <N> Connections to keep open
-d, --duration <T> Duration of test
-t, --threads <N> Number of threads to use
-s, --script <S> Load Lua script file
-H, --header <H> Add header to request
--latency Print latency statistics
--timeout <T> Socket/request timeout
-v, --version Print version details
Numeric arguments may include a SI unit (1k, 1M, 1G)
Time arguments may include a time unit (2s, 2m, 2h)
and another
$ LOG_FILE=/tmp/nginx.log NUM_CPU_CORES=256 ./nginx -d 90s -c 4000 https://127.0.0.1:8089/test.html
unable to create thread 67: Too many open files
Segmentation fault (core dumped)
This failed because ran out of nofile ulimit.
Increasing ulimit -n 524288 it succeeded.
So basically it seems this test should also increase ulimit before running because it takes a lot of time to figure this out. Also error message from log would be useful too because when next run runs then previous log is deleted.