Azure artifacts restore failures

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2017199

Build leg reported

All of them (in attempt 1)

Pull Request

No response

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Response status code does not indicate success: 503 (Service Unavailable)",
   "BuildRetry": false
}

Additional information about the issue reported

I'm seeing some restore failures accross repositories this morning. Opening a known issue to track the ones that happen in the public side. The message is a little broad as it will catch every 503 from Azure devops. If we do end up catching strays not related to feeds, I'll try to narrow it down a bit.

Report

Build	Definition	Step Name	Console log	Pull Request
48004	dotnet/runtime	Build managed product components and packages	Log	dotnet/runtime#76868
48566	dotnet/roslyn	Build	Log
48266	dotnet/runtime	Build product	Log	dotnet/runtime#76898
47179	dotnet/runtime	Build and Package	Log	dotnet/runtime#76847
48132	dotnet/installer	Build	Log	dotnet/installer#14715
48141	dotnet/sdk	Build	Log	dotnet/sdk#28468
48149	dotnet/sdk	Build	Log	dotnet/sdk#28470
48146	dotnet/sdk	Build	Log	dotnet/sdk#28469
48130	dotnet/installer	Build	Log	dotnet/installer#14713
48117	dotnet/sdk	Build	Log	dotnet/sdk#28467
48084	dotnet/roslyn	Build and Test	Log	dotnet/roslyn#64634
48085	dotnet/roslyn	Build and Test	Log	dotnet/roslyn#64634
47985	dotnet/runtime	Prepare TestHost with runtime CoreCLR	Log	dotnet/runtime#76692
48034	dotnet/aspnetcore	Build	Log
48020	dotnet/runtime	Build product	Log	dotnet/runtime#76870
48014	dotnet/runtime	Build product	Log	dotnet/runtime#74393
47823	dotnet/installer	Create Tarball	Log	dotnet/installer#14549
47980	dotnet/arcade	Unix Build / Publish	Log	#11230
47930	dotnet/runtime	mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release	Log	dotnet/runtime#76878
47934	dotnet/runtime	Build product	Log	dotnet/runtime#75597
47952	dotnet/sdk	Build	Log
44017	dotnet/sdk	Windows_NT Build_Release	Log	dotnet/sdk#28380
47927	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#28458
47928	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#28464
47884	dotnet/runtime	Build native test components	Log	dotnet/runtime#76874
47917	dotnet/runtime	CoreCLR Product Build windows arm64 release	Log	dotnet/runtime#74820
47920	dotnet/runtime	Build product	Log	dotnet/runtime#74820
47912	dotnet/aspnetcore	Run build.sh	Log
47903	dotnet/roslyn-sdk	CmdLine	Log	dotnet/roslyn-sdk#1026
47689	dotnet/runtime	Restore blob feed tasks (Unix)	Log	dotnet/runtime#76743
47454	dotnet/runtime	Copy native test components to test output folder	Log	dotnet/runtime#76850
47887	dotnet/runtime	Build product	Log	dotnet/runtime#76874
47899	dotnet/installer	Build	Log
47881	dotnet/runtime	Build product	Log	dotnet/runtime#76811
47885	dotnet/runtime	Build product	Log	dotnet/runtime#76874
47886	dotnet/runtime	Build product	Log	dotnet/runtime#76874
47879	dotnet/runtime	Restore and Build Product	Log	dotnet/runtime#76811
47743	dotnet/runtime	Restore blob feed tasks (Unix)	Log	dotnet/runtime#76868
47851	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#28459
47765	dotnet/runtime	Copy native test components to test output folder	Log	dotnet/runtime#76871
45192	dotnet/runtime	Build product	Log	dotnet/runtime#75597
47853	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#28460
47852	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#28461
47862	dotnet/runtime	Build CLR and Libraries	Log
47869	dotnet/aspnetcore	Build	Log
47864	dotnet/runtime	Build CLR and Libraries	Log
47815	dotnet/runtime	Build managed product components and packages	Log	dotnet/runtime#76873
47866	dotnet/runtime	Build CLR and Libraries	Log
47867	dotnet/runtime	Build CLR and Libraries	Log
47822	dotnet/aspnetcore	Build ARM64	Log
47739	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#76870
47827	dotnet/sdk	Run dotnet new Integration Tests	Log	dotnet/sdk#28416
47826	dotnet/installer	Build Tarball	Log	dotnet/installer#14705
47724	dotnet/runtime	Send tests to Helix (Windows)	Log
47843	dotnet/roslyn-sdk	CmdLine	Log	dotnet/roslyn-sdk#1026
46563	dotnet/runtime	Build tests	Log	dotnet/runtime#76339
47828	dotnet/installer	Build	Log	dotnet/installer#14709
47814	dotnet/sdk	Windows_NT_TemplateEngine Build_Release	Log	dotnet/sdk#28456
47816	dotnet/runtime	Build	Log	dotnet/runtime#76873
47824	dotnet/sdk	Darwin_AoT_Tests Build_Release	Log	dotnet/sdk#28457
47802	dotnet/sdk	Darwin_TemplateEngine Build_Release	Log	dotnet/sdk#28455
47800	dotnet/sdk	Windows_NT_FullFramework Build_Release	Log	dotnet/sdk#28454
47801	dotnet/sdk	Windows_NT Build_Release	Log	dotnet/sdk#28453
47767	dotnet/runtime	Build	Log	dotnet/runtime#76871
47721	dotnet/runtime	Libraries Build OSX x64 Release	Log
47718	dotnet/runtime	Libraries Build OSX x64 Release	Log
47723	dotnet/runtime	Libraries Build OSX arm64 Release	Log
47719	dotnet/runtime	Libraries Build Linux arm Release	Log
47621	dotnet/runtime	Build product	Log
47622	dotnet/sdk	Build	Log
47607	dotnet/aspnetcore	Tests: Helix x64	Log	dotnet/aspnetcore#44426
47530	dotnet/runtime	Build product	Log
47490	dotnet/runtime	Send to Helix	Log
47488	dotnet/runtime	Copy native test components to test output folder	Log
47460	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#76860
47447	dotnet/runtime	Copy native test components to test output folder	Log
47524	dotnet/runtime	Build product	Log
47487	dotnet/runtime	Copy native test components to test output folder	Log
47519	dotnet/sdk	Run Tests in Helix and non Helix in parallel	Log	dotnet/sdk#28447
47448	dotnet/runtime	Generate test wrappers	Log
47500	dotnet/aspnetcore	Build ARM64	Log	dotnet/aspnetcore#44465
47506	dotnet/sdk	Run Tests in Helix and non Helix in parallel	Log
47517	dotnet/roslyn	Build	Log
47515	dotnet/roslyn	Build - Validate determinism	Log
47518	dotnet/roslyn	Build and Test	Log
47489	dotnet/runtime	Build managed product components and packages	Log
47516	dotnet/roslyn	Build and Test	Log
47514	dotnet/sdk	Build	Log
47510	dotnet/sdk	Build	Log
47509	dotnet/performance	Prepare scenarios	Log	dotnet/performance#2641
47502	dotnet/aspnetcore	Build shared fx	Log	dotnet/aspnetcore#44465
47495	dotnet/sdk	Windows_NT_FullFramework Build_Debug	Log
47364	dotnet/sdk	Build	Log
47358	dotnet/runtime	Build product	Log	dotnet/runtime#76853
45803	dotnet/sdk	Windows_NT_FullFramework Build_Release	Log	dotnet/sdk#28416
47300	dotnet/roslyn	Correctness_Build_Artifacts	Log	dotnet/roslyn#64617
47239	dotnet/roslyn	Build and Test	Log	dotnet/roslyn#64618
47237	dotnet/runtime	Build tvOS arm64 Release AllSubsets_Mono	Log
47209	dotnet/runtime	Build product	Log	dotnet/runtime#74623
47211	dotnet/runtime	Build	Log	dotnet/runtime#74623
Displaying 100 of 155 results

Build	Definition	Test	Pull Request
47927	dotnet/sdk	Microsoft.DotNet.Cli.Sln.Add.Tests.GivenDotnetSlnAdd.WhenValidProjectIsPassedTheSlnBuilds	dotnet/sdk#28458
47777	dotnet/aspnetcore	Templates.Tests--net7.0.WorkItemExecution
47770	dotnet/aspnetcore	Templates.Blazor.Server.Tests--net7.0.WorkItemExecution	dotnet/aspnetcore#44469
47764	dotnet/aspnetcore	Templates.Mvc.Tests--net7.0.WorkItemExecution
47899	dotnet/installer	Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishAFrameworkDependentApp.It_errors_when_using_app_host_with_older_target_framework
47800	dotnet/sdk	Microsoft.NET.Sdk.Razor.Tests.BuildIntegrationTest.Build_WithP2P_Referencing21Project_Works	dotnet/sdk#28454
47754	dotnet/aspnetcore	Templates.Blazor.Server.Tests--net7.0.WorkItemExecution
47765	dotnet/runtime	LibraryImportGenerator.UnitTests.Compiles.ValidateRequireAllowUnsafeBlocksDiagnosticNoTrigger	dotnet/runtime#76871
47811	dotnet/performance	Partition4.WorkItemExecution	dotnet/performance#2644
47739	dotnet/runtime	LibraryImportGenerator.UnitTests.Compiles.ValidateRequireAllowUnsafeBlocksDiagnosticNoTrigger	dotnet/runtime#76870
47495	dotnet/sdk	dotnet.Tests.dll.8.WorkItemExecution
47519	dotnet/sdk	Microsoft.DotNet.PackageValidation.Tests.ValidatePackageTargetTests.InvalidPackage	dotnet/sdk#28447
47509	dotnet/performance	Partition4.WorkItemExecution	dotnet/performance#2641
47439	dotnet/runtime	Microsoft.NET.HostModel.Tests.BundleLegacy.TestNetCoreApp3xApp(minorVersion: 1)	dotnet/runtime#76705
47149	dotnet/sdk	Microsoft.NET.Sdk.Razor.Tests.dll.1.WorkItemExecution
45870	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.2.WorkItemExecution	dotnet/sdk#28397
47168	dotnet/sdk	Microsoft.DotNet.Cli.Test.Tests.GivenDotnetTestBuildsAndRunsTestFromCsprojForMultipleTFM.MStestMultiTFM
47034	dotnet/sdk	Microsoft.NET.Build.Tests.dll.5.WorkItemExecution
47201	dotnet/performance	SDK Razor Class Library Template Clean Build.WorkItemExecution
46486	dotnet/sdk	Microsoft.NET.Build.Tests.dll.11.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
161	175	175

[michellemcdaniel]

We noticed this in the release pipeline as well. Do things seem to work on retry?

I've had some luck retrying the builds I've seen in this state this morning.

@MattGal is engaged with the azure artifacts team, these 503s are in their radar.

[michellemcdaniel]

heads up @dcwhittaker

[ulisesh]

Closed in favor of #11239