crossgen-comparison Linux arm checked fails with timeouts

[sandreenko]

It happens in all runs, the timeout is 60 minutes. We have not seen that in coreclr.

cc @echesakovMSFT @BruceForstall

Log example:

Starting: Run native crossgen and compare output to baseline (Unix)
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.151.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
$BUILD_SOURCESDIRECTORY/eng/common/msbuild.sh $BUILD_SOURCESDIRECTORY/eng/common/helixpublish.proj /restore /t:Test /bl:$BUILD_SOURCESDIRECTORY/artifacts/log/$BuildConfig/SendToHelix.binlog
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /__w/_temp/7553b9eb-4204-4db6-9f64-3d08371fe862.sh
/__w/3/s/.dotnet/sdk/5.0.100-alpha1-015772/MSBuild.dll /nologo -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/__w/3/s/.dotnet/sdk/5.0.100-alpha1-015772/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/__w/3/s/.dotnet/sdk/5.0.100-alpha1-015772/dotnet.dll -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/3/s/artifacts/log/Checked/SendToHelix.binlog /clp:Summary /nr:true /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /restore /t:Test /warnaserror /__w/3/s/eng/common/helixpublish.proj
  Restore completed in 599.36 ms for /__w/3/s/eng/common/helixpublish.proj.
  Restore completed in 2.35 ms for /__w/3/s/eng/common/helixpublish.proj.
  Starting Azure Pipelines Test Run (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226
  Uploading payloads for Job on (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226...
  Finished uploading payloads for Job on (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226...
  Sending Job to (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226...
  Sent Helix Job 40b0be17-33d6-41c3-bdce-fd02aba91a98
  Waiting for completion of job 40b0be17-33d6-41c3-bdce-fd02aba91a98
##[error]The operation was canceled.
Finishing: Run native crossgen and compare output to baseline (Unix)

https://dev.azure.com/dnceng/public/_build/results?buildId=471248

[BruceForstall]

Is this a regression, or has it always failed in runtime?

[sandreenko]

I saw it green a few times, for example here https://dev.azure.com/dnceng/public/_build/results?buildId=456520&view=logs&j=e14ed261-6507-56f9-47b6-1ad25f52112f.

[jashook]

Generally this is failure happens due to us running at capacity for the helix queue. With the amount of machines that were added last month, there is most likely something that has changed with what jobs are submitted and from where to cause this, or there are a lot of machine that have gone offline.

[jashook]

Is this a regression, or has it always failed in runtime?

This has not always failed, although the amount of jobs submitted has probably increased this month over last month.

[sandreenko]

Thanks, @jashook . We can make a quick surgical fix and stop running everything twice (under runtime and under runtime-coreclr pipelines, delete duplicates from https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime-official.yml and https://github.com/dotnet/runtime/blame/master/eng/pipelines/coreclr/pr.yml) or wait for a few weeks when @safern replaces existing pipelines.

[jashook]

Seems like the wait times are not too high in the queue, if I remember right these jobs ran close to the timeout. I would suggest upping the timeout by 15 to 30 minutes.

[safern]

or wait for a few weeks when @safern replaces existing pipelines.

I should be removing duplication this week. Hope to have a PR before Wednesday.