[Perf] Linux/x64: 160 Regressions on 8/17/2022 6:09:21 PM #7981
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | ubuntu 18.04 |
| Baseline | 543bcc5ee7d6a2b9471b016770227421c43a756e |
| Compare | f142128e89b63577a9bbba7e2b760ec82102a7a9 |
| Diff | Diff |
Regressions in System.Tests.Perf_String
Repro
git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'Details
Payloads
Histogram
System.Tests.Perf_String.Format_OneArg(s: "Testing {0}, {0:C}, {0:E} - {0:F4}{0:G}{0:N} , !!", o: 3.14159)
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.5516874625643573 > 2.350611819642669.
IsChangePoint: Marked as a change because one of 6/24/2022 5:01:29 PM, 6/28/2022 9:06:37 PM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -38.79578953518803 (T) = (0 -2563.7163581969467) / Math.Sqrt((454.87117869714154 / (14)) + (930.0274088828977 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -0.1398142456624475 = (2249.2404950658806 - 2563.7163581969467) / 2249.2404950658806 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'i', newChar: 'I')
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 125.90268824435714 > 101.73812095373543.
IsChangePoint: Marked as a change because one of 6/24/2022 12:39:19 PM, 6/29/2022 4:06:39 AM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -28.19418272941041 (T) = (0 -128.3050576893062) / Math.Sqrt((11.774123315218525 / (14)) + (18.26513705560435 / (29))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (29) - 2, .025) and -0.36332811572868035 = (94.11164943277716 - 128.3050576893062) / 94.11164943277716 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_Char(text: "Hello", oldChar: 'l', newChar: '!')
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 98.36287430188123 > 68.20174507294821.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -32.646923659413666 (T) = (0 -100.2224874025874) / Math.Sqrt((6.4138230698879735 / (14)) + (21.48488045099303 / (29))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (29) - 2, .025) and -0.5544478253206164 = (64.47465509620162 - 100.2224874025874) / 64.47465509620162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Format_OneArg(s: "Testing {0}, {0:C}, {0:D5}, {0:E} - {0:F4}{0:G}{0:N} {0:X} !!", o: 8)
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4344054878048782 > 1.1605645880607696.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/29/2022 1:11:29 AM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -44.261218747176585 (T) = (0 -1471.663986324827) / Math.Sqrt((314.28462297584457 / (14)) + (1327.0239262407026 / (29))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (29) - 2, .025) and -0.33047659631188225 = (1106.117905722145 - 1471.663986324827) / 1106.117905722145 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 363.608794745425 > 143.77835024641746.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -124.96765950555645 (T) = (0 -347.3291751657902) / Math.Sqrt((10.319928796325312 / (14)) + (57.48832107905517 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -1.5063241969077996 = (138.5811043895721 - 347.3291751657902) / 138.5811043895721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 764.759308027231 > 167.68205416757223.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -134.87212820251094 (T) = (0 -782.548426892428) / Math.Sqrt((13.322034129219215 / (14)) + (565.8316900027708 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -3.826408896964876 = (162.13885802020204 - 782.548426892428) / 162.13885802020204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y')
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 289.35599893092285 > 25.287906320219577.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -99.98218219727052 (T) = (0 -288.377274892454) / Math.Sqrt((3.3652378870109554 / (14)) + (188.46364743346518 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -10.821810471096704 = (24.393664202070514 - 288.377274892454) / 24.393664202070514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.IndexOfAny
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 802.4409741362872 > 73.09304220658181.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -182.10788141307557 (T) = (0 -813.7766362361342) / Math.Sqrt((22.196872375281252 / (14)) + (404.2380861527335 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -10.18313001926548 = (72.76823526456539 - 813.7766362361342) / 72.76823526456539 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "bad", newValue: "nice")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 280.1536335030515 > 35.636487595757515.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -273.36779454336 (T) = (0 -281.2164450470928) / Math.Sqrt((2.8023748373775828 / (14)) + (15.494713214793327 / (25))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (25) - 2, .025) and -7.350228598819173 = (33.677694175565485 - 281.2164450470928) / 33.677694175565485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_Char(text: "Hello", oldChar: 'a', newChar: 'b')
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 76.84936149358121 > 18.282816393967657.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -46.79705112899162 (T) = (0 -65.5071128647858) / Math.Sqrt((7.556530895537607 / (14)) + (15.565143026298111 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -2.964579874608719 = (16.523090702328446 - 65.5071128647858) / 16.523090702328446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "b")
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 233.28303305149277 > 152.85767414876875.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -83.0389948012548 (T) = (0 -237.70080891605662) / Math.Sqrt((9.63630059919198 / (14)) + (13.794542493264334 / (27))) is less than -2.0226909200346674 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (27) - 2, .025) and -0.6195919590567344 = (146.76586135590338 - 237.70080891605662) / 146.76586135590338 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Replace_Char(text: "yfesgj0sg1ijslnjsb3uofdz3tbzf6ysgblu3at20nfab2wei1kxfbvsbpzwhanjczcqa2psra3aacxb67qnwbnfp2tok6v0a58l", oldChar: 'b', newChar: '+')
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 364.95583104041975 > 181.89205555824867.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -116.16669505847699 (T) = (0 -353.66583335705974) / Math.Sqrt((12.87366151610382 / (14)) + (40.051677336329575 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -1.014191277467959 = (175.58701465615187 - 353.66583335705974) / 175.58701465615187 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Format_MultipleArgs
```log
Description of detection logic
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 854.3849195442017 > 547.6817332098481.
IsChangePoint: Marked as a change because one of 6/24/2022 10:40:04 AM, 6/27/2022 8:54:23 AM, 8/5/2022 10:47:59 PM, 8/23/2022 11:50:53 AM falls between 8/5/2022 10:47:59 PM and 8/23/2022 11:50:53 AM.
IsRegressionStdDev: Marked as regression because -97.8970278454593 (T) = (0 -852.4908035609617) / Math.Sqrt((88.87602574889881 / (14)) + (136.77708491956312 / (28))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (14) + (28) - 2, .025) and -0.6257128366247446 = (524.3796963127123 - 852.4908035609617) / 524.3796963127123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository