slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1408248	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127779
1408239	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127778
2967054	dotnet-performance	Performance nativeaot_scenarios windows Win11 x64 Viper main	Log
1408038	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
1408133	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#127636
1408117	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#127773
1408102	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#127751
1408083	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#127573
1408071	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127578
1408034	dotnet/aspnetcore	Tests: macOS	Log
1407917	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#126150
1407953	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127741
1407974	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53847
1407916	dotnet/maui	Controls Shell	Log	dotnet/maui#31755
1407924	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127228
1407922	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#127228
1407929	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127604
2966698	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#60646
1407268	dotnet/maui	Controls Shell	Log	dotnet/maui#29952
1407709	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127498
1407685	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#125437
1407664	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126954
1407657	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126916
1407646	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127760
1407648	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127760
1407609	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127755
1407595	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127749
1407589	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127697
1407568	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127319
1407554	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127743
1407518	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127510
1407551	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127523
1407405	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127016
1407407	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127016
1407378	dotnet/runtime	osx-x64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#117261
1407345	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127742
1407499	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127750
1407242	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126351
1407019	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Release	Log
1406879	dotnet/maui	Controls Layout	Log
1407162	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127730
2966608	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2966651	dotnet-performance	Performance powershell ubuntu 2204 arm64 Ampere main	Log
1407130	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127724
1407117	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127720
1407115	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127720
1407076	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127712
1407067	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127709
1407048	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR		dotnet/runtime#127705
1407034	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127702
1407021	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66487
1407018	dotnet/aspnetcore	Tests: macOS	Log
1403849	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66519
1406989	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127112
1406959	dotnet/runtime	osx-x64 release Runtime_Release	Log
1406955	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127005
1406925	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127603
1406826	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2966337	dotnet-performance	Performance powershell ubuntu 2204 arm64 Ampere main	Log
1406235	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127613
1406718	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127689
1406683	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127690
2966150	dotnet-performance	Performance powershell ubuntu 2204 arm64 Ampere main	Log
2966039	dotnet-maui	Pack macOS	Log
1406427	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2965935	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper 8.0	Log
1406336	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127689
1406292	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66569
1406262	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127615
1406198	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#127683
1406155	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#126932
2965725	dotnet-performance	Performance powershell ubuntu 2204 x64 Viper 9.0	Log
2965690	dotnet-runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log
1406046	dotnet/maui	Controls Shell	Log
1405962	dotnet/maui	Controls CollectionView	Log
1406006	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127675
1405999	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1405922	dotnet/aspnetcore	Tests: macOS	Log
2965439	dotnet-performance	Performance powershell ubuntu 2204 arm64 Ampere main	Log
1405703	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#127613
1405692	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#123920
1405593	dotnet/aspnetcore	Tests: macOS	Log
1405528	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127613
1405414	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126911
2965036	dotnet-maui	Controls (vlatest) CollectionView	Log
1405315	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127649
1405168	dotnet/maui	Controls CollectionView	Log
1405288	dotnet/roslyn	Build_macOS_Debug	Log
1405200	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127642
2965035	dotnet-maui	RunOniOS_MauiDebug ARM64	Log
2965047	dotnet-performance	Performance blazor_scenarios windows Win11 x64 Viper 8.0	Log
1405179	dotnet/aspnetcore	Tests: macOS	Log
1405163	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127596
1405091	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127555
1404759	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1404983	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127604
1404669	dotnet/maui	Controls CollectionView	Log
1404816	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127635
1404798	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127300
1404795	dotnet/aspnetcore	Tests: macOS	Log
Displaying 100 of 1375 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
57	428	1375

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.