Revert "Revert "Upgrade Ubuntu and Azure Linux VMs and Containers""#62304
Open
richlander wants to merge 1 commit intomainfrom
Open
Revert "Revert "Upgrade Ubuntu and Azure Linux VMs and Containers""#62304richlander wants to merge 1 commit intomainfrom
richlander wants to merge 1 commit intomainfrom
Conversation
github-actions
bot
added
the
area-mvc
Open
3 tasks
Member
Author
|
The aspnetcore-helix-matrix pipeline ran clean. https://dev.azure.com/dnceng-public/public/_build/results?buildId=1064551&view=results |
Member
|
The AzureLinux machine on PRs is still borked. |
Member
|
We do get console logs now, at least. This is from the "failed" Mvc.FunctionalTests workitem:
Not sure why that's getting reported as a failure |
dotnet-policy-service
bot
added
the
pending-ci-rerun
Member
Author
|
/azp run |
dotnet-policy-service
bot
removed
the
pending-ci-rerun
|
Azure Pipelines successfully started running 3 pipeline(s). |
Member
Author
|
From the logs. The first doesn't look very OS specific. The second looks like it could be the VM. 2025-06-24T22:33:16.5064416Z [xUnit.net 00:00:19.87] | [0.040s] Templates.Mvc.Test.WebApiTemplateTest Information: ==> /datadisks/disk1/work/AB9C09E3/p/dotnet-cli/dotnet new "web" --debug:disable-sdk-templates --debug:custom-hive "/datadisks/disk1/work/AB9C09E3/w/AB1409A8/e/Hives/.templateEngine" [/datadisks/disk1/work/AB9C09E3/w/AB1409A8/e/]
2025-06-24T22:33:16.5064472Z [xUnit.net 00:00:19.87] | [0.715s] Templates.Mvc.Test.WebApiTemplateTest Information: Process exited.
2025-06-24T22:33:16.5122271Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] No templates or subcommands found matching: 'web'.
2025-06-24T22:33:16.5122329Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR]
2025-06-24T22:33:16.5122477Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] To list installed templates similar to 'web', run:
2025-06-24T22:33:16.5122521Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] dotnet new list web
2025-06-24T22:33:16.5122571Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] To search for the templates on NuGet.org, run:
2025-06-24T22:33:16.5122732Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] dotnet new search web
2025-06-24T22:33:16.5122770Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR]
2025-06-24T22:33:16.5122809Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR]
2025-06-24T22:33:16.5123077Z [xUnit.net 00:00:19.87] | [1.259s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] For details on the exit code, refer to https://aka.ms/templating-exit-codes#103
2025-06-24T22:33:16.5123113Z [xUnit.net 00:00:19.87] | [1.264s] Templates.Mvc.Test.WebApiTemplateTest Information: ==> /datadisks/disk1/work/AB9C09E3/p/dotnet-cli/dotnet new "webapp" --debug:disable-sdk-templates --debug:custom-hive "/datadisks/disk1/work/AB9C09E3/w/AB1409A8/e/Hives/.templateEngine" [/datadisks/disk1/work/AB9C09E3/w/AB1409A8/e/]
2025-06-24T22:33:16.5123352Z [xUnit.net 00:00:19.87] | [1.478s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] No templates or subcommands found matching: 'webapp'.
2025-06-24T22:33:16.5123386Z [xUnit.net 00:00:19.87] | [1.481s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR]
2025-06-24T22:33:16.5123421Z [xUnit.net 00:00:19.87] | [1.481s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] To list installed templates similar to 'webapp', run:
2025-06-24T22:33:16.5151090Z [xUnit.net 00:00:19.87] | [1.481s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] dotnet new list webapp
2025-06-24T22:33:16.5151200Z [xUnit.net 00:00:19.87] | [1.481s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] To search for the templates on NuGet.org, run:
2025-06-24T22:33:16.6151750Z [xUnit.net 00:00:19.87] | [1.481s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] dotnet new search webapp
2025-06-24T22:33:16.6151924Z [xUnit.net 00:00:19.87] | [1.481s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR]
2025-06-24T22:33:16.6151962Z [xUnit.net 00:00:19.87] | [1.483s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR]
2025-06-24T22:33:16.6151992Z [xUnit.net 00:00:19.87] | [1.484s] Templates.Mvc.Test.WebApiTemplateTest Information: [ERROR] For details on the exit code, refer to https://aka.ms/templating-exit-codes#103And --- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.12/logging/__init__.py", line 1164, in emit
self.flush()
File "/usr/lib/python3.12/logging/__init__.py", line 1144, in flush
self.stream.flush()
OSError: [Errno 28] No space left on device
Call stack:
File "/datadisks/disk1/work/AB9C09E3/p/reporter/run.py", line 68, in <module>
main()
File "/datadisks/disk1/work/AB9C09E3/p/reporter/run.py", line 65, in main
reporter.report_results(all_results)
File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 30, in report_results
self._log.info("Packing %d test reports to '%s'", len(results), path)
Message: "Packing %d test reports to '%s'"
Arguments: (963, '/datadisks/disk1/work/AB9C09E3/w/C3C70A88/e/__test_report.json')
2025-06-24T22:37:49.529Z INFO run.py packing_test_reporter(33) report_results Packed 344460 bytes
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
if self.shouldRollover(record):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/logging/handlers.py", line 193, in shouldRollover
pos = self.stream.tell()
^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
File "/datadisks/disk1/work/AB9C09E3/p/reporter/run.py", line 68, in <module>
main()
File "/datadisks/disk1/work/AB9C09E3/p/reporter/run.py", line 65, in main
reporter.report_results(all_results)
File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 33, in report_results
self._log.info("Packed %d bytes", os.path.getsize(path))
Message: 'Packed %d bytes'
Arguments: (344460,)
+ exit 0
['Microsoft.AspNetCore.SignalR.Client.FunctionalTests--net10.0' END OF WORK ITEM LOG: Command exited with 0] |
Member
The first one isn't a test error. It's normal expected output. Like previous runs, the helix run seems to have passed but still shows as a failure in AzDo
|
Member
Author
|
Thanks for the analysis. That suggests a problem on the helix side, somehow. I assume the second one is a legit issue, however. Yes? |
dotnet-policy-service
bot
added
the
pending-ci-rerun
Member
Author
|
/azp run |
dotnet-policy-service
bot
removed
the
pending-ci-rerun
|
Azure Pipelines successfully started running 3 pipeline(s). |
Member
Author
|
Just re-ran the pipeline. Seems the same as before (on Azure Linux). This is the end of the console log. Seems like "no space" is the culprit. 2025-07-09T14:53:51.0128010Z Data collector 'Blame' message: All tests finished running, Sequence file will not be generated.
2025-07-09T14:53:51.1295893Z Passed Microsoft.AspNetCore.SignalR.Client.FunctionalTests.HubConnectionTests.WebSocketsCanConnectOverHttp2 [451 ms]
2025-07-09T14:53:51.1304271Z Passed Microsoft.AspNetCore.SignalR.Client.FunctionalTests.HubConnectionTests.StopCausesPollToReturnImmediately [299 ms]
2025-07-09T14:53:52.6826384Z Results File: /datadisks/disk1/work/9D490945/w/AFE00999/e/TestResults/TestResults.xml
2025-07-09T14:53:52.6842264Z
2025-07-09T14:53:52.6967982Z Test Run Successful.
2025-07-09T14:53:52.6968073Z Total tests: 963
2025-07-09T14:53:52.6968102Z Passed: 959
2025-07-09T14:53:52.6968129Z Skipped: 4
2025-07-09T14:53:52.7081311Z Total time: 5.7686 Minutes
2025-07-09T14:53:52.7877173Z [dotnet test - 07/09/2025 14:53:52.787]No artifact found, post-processing won't run.
2025-07-09T14:53:52.8092329Z '/datadisks/disk1/work/9D490945/p/dotnet-cli/dotnet test Microsoft.AspNetCore.SignalR.Client.FunctionalTests.dll --diag:/datadisks/disk1/work/9D490945/w/AFE00999/uploads/vstest.log --logger xunit --logger "console;verbosity=normal" --blame-crash --blame-hang-timeout 15m --filter "Quarantined!=true|Quarantined=false"' completed with exit code '0'
2025-07-09T14:53:52.8102029Z Running tests complete
2025-07-09T14:53:52.8102101Z Uploading test results
2025-07-09T14:53:52.8107667Z Trying to upload results...
2025-07-09T14:53:52.8107972Z Copying TestResults/TestResults.xml to ./testResults.xml
2025-07-09T14:53:52.8657161Z Copying artifacts/log/ to /datadisks/disk1/work/9D490945/w/AFE00999/uploads/
2025-07-09T14:53:52.8660178Z Copying TestResults/**/Sequence*.xml to /datadisks/disk1/work/9D490945/w/AFE00999/uploads/
2025-07-09T14:53:52.8660885Z Test results uploaded
2025-07-09T14:53:52.8660957Z Completed Helix job with exit code '0'
Finished tests...exit_code=0
+ export _commandExitCode=0
+ _commandExitCode=0
+ /usr/bin/python3.12 /datadisks/disk1/work/9D490945/p/reporter/run.py https://dev.azure.com/dnceng-public/ public 29684050 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6InRzOGNhWXZ4UGNORXBpQnFhVG1FTlpwRmhZMCJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiI4YmVmOGY3Zi0xMzg2LTRlYzAtYjI5Zi00YjIzODkyMjI4YmMiLCJzaWQiOiI3ZjE0MzI2MS05MjJmLTRhNjUtOTk5ZS1lMWQ3NDAxOTU1M2EiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzEwODk4MjIiLCJEZWZJZCI6IjgzIiwiam9icmVmIjoiN2FmNWIwYjQtZmFhMy00NTBiLWFmZGUtZTBlNDZjNjM0MTNjOjNmNmQ0ZTBmLTFiNzEtNTZiNS0zNjFlLWQ5NWI2ZTZkYTE1YSIsInBwaWQiOiJ2c3RmczovLy9CdWlsZC9CdWlsZC8xMDg5ODIyIiwib3JjaGlkIjoiN2FmNWIwYjQtZmFhMy00NTBiLWFmZGUtZTBlNDZjNjM0MTNjLmJ1aWxkLmhlbGl4X3g2NC5fX2RlZmF1bHQiLCJyZXBvSWRzIjoiIiwiaXNzIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbSIsImF1ZCI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb218dnNvOjZmY2M5MmU1LTczYTctNGY4OC04ZDEzLWQ5MDQ1YjQ1ZmIyNyIsIm5iZiI6MTc1MjA2OTQwMSwiZXhwIjoxNzUyMDg1NjAxfQ.h41wlkhzXbyDyKt0-mdeyPYbJM4AkDdJ3qzLBiF9KXv3GseYLst90tntCs27oC1dc6CGmBcSAFo0HGAWlacKVyKsf0N_ihxo9ztt1LNmKsbDL11NxEMUX_h5K0hN3D5Ep41-Hd_ggD249siYyZBVf60tcVUGS7pQHkKyo5GsaqXh4AIxd4DWH0X_CRd0oJBo_o_MNhoPQsUaSgAZLLi-SbOqzcdvdueWnAyTMrWo_nAV_LK1rt21TNeYkJjZuPbY3hft5wKlqjR_cpSIyvtgVqpxhs7tg9jAYcBsIfGYccnuqCyvNLlGZnnj-MtPE83PqM_Z8-CTFfWlBRlj1MhoFQ
/etc/helix/scripts/azure/__init__.py:5: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
2025-07-09T14:53:53.966Z INFO run.py managed_identity(145) __init__ ManagedIdentityCredential will use IMDS with client_id: 3423231b-6b9e-4177-8817-76276b647404
2025-07-09T14:53:53.966Z INFO run.py azure_utils(32) get_credential_and_access_token VMSSManagedIdentity VmssMIClientIdForUpload Auth type will be used
2025-07-09T14:53:53.971Z INFO run.py _universal(511) on_request Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=REDACTED&resource=REDACTED&client_id=REDACTED'
Request method: 'GET'
Request headers:
'Metadata': 'REDACTED'
'User-Agent': 'azsdk-python-identity/1.23.0 Python/3.12.9 (Linux-6.6.92.2-1.azl3-x86_64-with-glibc2.38)'
No body was attached to the request
2025-07-09T14:53:53.981Z INFO run.py _universal(554) on_response Response status: 200
Response headers:
'Content-Type': 'application/json; charset=utf-8'
'Server': 'IMDS/150.870.65.1730'
'x-ms-request-id': '7eeb8f41-1a48-430a-a66e-57fa3d4839a3'
'Date': 'Wed, 09 Jul 2025 14:53:53 GMT'
'Content-Length': '2264'
2025-07-09T14:53:53.981Z INFO run.py msal_managed_identity_client(165) _get_token_base ImdsCredential.get_token succeeded
2025-07-09T14:53:53.981Z INFO run.py decorators(24) wrapper ManagedIdentityCredential.get_token succeeded
2025-07-09T14:53:53.982Z INFO run.py azure_utils(64) get_credential_and_access_token Credentials are valid
2025-07-09T14:53:53.983Z INFO run.py msal_managed_identity_client(165) _get_token_base ImdsCredential.get_token_info succeeded
2025-07-09T14:53:53.983Z INFO run.py decorators(24) wrapper ManagedIdentityCredential.get_token_info succeeded
2025-07-09T14:53:54.183Z INFO run.py run(48) main Beginning reading of test results.
2025-07-09T14:53:54.183Z INFO run.py __init__(42) read_results Searching '/datadisks/disk1/work/9D490945/w/AFE00999/e' for test results files
2025-07-09T14:53:54.185Z INFO run.py __init__(48) read_results Found results file /datadisks/disk1/work/9D490945/w/AFE00999/e/testResults.xml with format xunit
2025-07-09T14:53:54.402Z INFO run.py __init__(42) read_results Searching '/datadisks/disk1/work/9D490945/w/AFE00999/uploads' for test results files
2025-07-09T14:53:54.402Z INFO run.py packing_test_reporter(30) report_results Packing 963 test reports to '/datadisks/disk1/work/9D490945/w/AFE00999/e/__test_report.json'
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.12/logging/__init__.py", line 1164, in emit
self.flush()
File "/usr/lib/python3.12/logging/__init__.py", line 1144, in flush
self.stream.flush()
OSError: [Errno 28] No space left on device
Call stack:
File "/datadisks/disk1/work/9D490945/p/reporter/run.py", line 68, in <module>
main()
File "/datadisks/disk1/work/9D490945/p/reporter/run.py", line 65, in main
reporter.report_results(all_results)
File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 30, in report_results
self._log.info("Packing %d test reports to '%s'", len(results), path)
Message: "Packing %d test reports to '%s'"
Arguments: (963, '/datadisks/disk1/work/9D490945/w/AFE00999/e/__test_report.json')
2025-07-09T14:53:54.469Z INFO run.py packing_test_reporter(33) report_results Packed 344460 bytes
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.12/logging/handlers.py", line 73, in emit
if self.shouldRollover(record):
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/logging/handlers.py", line 193, in shouldRollover
pos = self.stream.tell()
^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Call stack:
File "/datadisks/disk1/work/9D490945/p/reporter/run.py", line 68, in <module>
main()
File "/datadisks/disk1/work/9D490945/p/reporter/run.py", line 65, in main
reporter.report_results(all_results)
File "/etc/helix/scripts/helix/test_reporting/packing_test_reporter.py", line 33, in report_results
self._log.info("Packed %d bytes", os.path.getsize(path))
Message: 'Packed %d bytes'
Arguments: (344460,)
+ exit 0
['Microsoft.AspNetCore.SignalR.Client.FunctionalTests--net10.0' END OF WORK ITEM LOG: Command exited with 0] |
3 tasks
dotnet-policy-service
bot
added
the
pending-ci-rerun
Member
Author
|
/azp run |
|
Azure Pipelines successfully started running 3 pipeline(s). |
dotnet-policy-service
bot
removed
the
pending-ci-rerun
This was referenced Jul 17, 2025
Member
Author
|
The SignalR test run is still failing but it is now less obvious why. All the tests pass.
I saw this. If true, it doesn't seem like this change is ready for GA. 11-30 is servicing. We shouldn't be in a position where we need to make this style of change in servicing. |
Member
|
Could be related to dotnet/dnceng#4756 & the version of python used? CC @akoeplinger @jkotas in case they remember anything about the issue (Looks like Doug had the most context) |
Member
|
There's definitely something weird in the infrastructure. The normal https://helix.dot.net/api/2019-06-17/jobs/6a0440aa-b53f-42be-a53e-bfb4129b60a5/workitems/Microsoft.AspNetCore.SignalR.Client.FunctionalTests--net10.0/console shows "not found" for me. If you go to https://helix.dot.net/api/2019-06-17/jobs/6a0440aa-b53f-42be-a53e-bfb4129b60a5/workitems/Microsoft.AspNetCore.SignalR.Client.FunctionalTests--net10.0/ you see this: |
Member
|
Ah yeah we also see this in the
|
dotnet-policy-service
bot
added
the
pending-ci-rerun
Member
Author
|
/azp run |
dotnet-policy-service
bot
removed
the
pending-ci-rerun
|
Azure Pipelines successfully started running 4 pipeline(s). |
Contributor
|
Looks like this PR hasn't been active for some time and the codebase could have been changed in the meantime. |
dotnet-policy-service
bot
added
the
pending-ci-rerun
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts #62265
Re-enable new OS images.