Run Categories Separately

[PureWeen]

Description

iOS and Catalyst device tests seem to have hit a critical mass of tests when running on CI. iOS has been problematic for a bit and catalyst just started following suit. This PR splits up the ios/catalyst runs so that it does a run per category

• Split up ios/catalyst to run per category
• add a property for ios/catalyst runs so you can filter by category --test-filter="Category=Editor" this only works on ios/catalyst so don't really want to advertise it too much yet
• Organized the code a bit
• Added a retry in cake so if a category fails it retries just that category

[Copilot]

Pull Request Overview

This PR refactors how test categories are handled to allow running them separately. Key changes include modifying MauiApp creation methods to accept a TestOptions lambda, updating test category attributes to use the new TestCategory values, and introducing helper functions and pipeline logic to support category‐based test execution.

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/Core/tests/DeviceTests/MauiProgram.cs	Updated MauiApp creation to use a lambda returning TestOptions with SkipCategories.
src/Core/tests/DeviceTests/Handlers/*	Changed Category attributes to reference updated TestCategory values.
src/Core/tests/DeviceTests.Shared/MauiProgramDefaults.cs	Added a new CreateMauiApp overload accepting a delegate and updated ConfigureTests usage.
src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs	Introduced helper methods for extracting and excluding test categories.
src/Controls/tests/DeviceTests/*	Mirrored changes from Core tests to support new TestOptions and Category attributes.
eng/pipelines and eng/devices/*	Modified pipeline and build scripts to run tests per category, including setting environment filters and handling test result file renaming.

Comments suppressed due to low confidence (2)

src/Core/tests/DeviceTests.Shared/MauiProgramDefaults.cs:59

• Passing null to the options delegate may be risky if the delegate assumes a non-null IServiceProvider. Consider refactoring to either supply a valid IServiceProvider instance or adjust the lambda to handle a null input explicitly.

if (options(null).Assemblies.Any(a => a.FullName.Contains("Controls.DeviceTests", StringComparison.OrdinalIgnoreCase)))

eng/devices/devices-shared.cake:139

• Ensure that 'testFilter' is declared and properly initialized in the current context, as it is used here without a visible definition.

if (!string.IsNullOrEmpty(testFilter))

[Copilot]

The addition of this CreateMauiApp overload introduces a change to the public API. Verify that appropriate documentation and test cases are updated accordingly to support this new API.

[PureWeen]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[jsuarezruiz]

Can we get some metrics on how this change impacts the build time? Has there been a noticeable effect?

[PureWeen]

Can we get some metrics on how this change impacts the build time? Has there been a noticeable effect?

It doesn't effect build time

I'm just looping over the same build and running each test by category.

Here's the run times on this PR

Here's the runtime on a PR that doesn't have these changes. Notice how iOS just timed out after 2 hours.
This PR definitely adds about 5 or 10 minutes since there's more time starting devices, but our iOS tests have been consistently freezing and taking 2 hours.

If we're worried about length of time then we could split this up to separate lanes if we want to and go that way, but for now this makes everything way more reliable and actually finish.

[PureWeen]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 3 pipeline(s).

[rmarinho]

Everything seems working, the test results artifacts seems have a folder that doesn t make sense, and logs are also on test results folder but I think theres no issue with that .