diff --git a/ci/builders/README.md b/ci/builders/README.md index df53742a5f01b..8a9a38417205a 100644 --- a/ci/builders/README.md +++ b/ci/builders/README.md @@ -374,9 +374,130 @@ be relative to the checkout directory. ### Global Tests -Global tests follow the same format as build tests but defined at the top -level. The main difference from local tests is that global tests have access -to the sub-build artifacts. +Tests in this section run on a separate bot as independent sub-builds. +As opposed to tests running within builds, global tests have access to the +the outputs of all the builds running in the same orchestrator build. A +use case for global tests is to run flutter/framework tests using the +artifacts generated by an specific engine build. + +Global tests currently support two different scenarios: + +* flutter/flutter tests with [tester](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine_v2/tester.py) + recipe. This workflow checks out flutter/flutter to run any of the existing + sharded tests using the engine artifacts archived to GCS. +* complicated engine tests that require the outputs from multiple subbuilds + with [tester_engine](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine_v2/tester_engine.py). + This workflow checks out [flutter/engine] and operates over the dependencies passed to it using cas. + +Note: the supported scenarios can be later extended to support running devicelab tests although a +[smart scheduler](https://github.com/flutter/flutter/issues/128294) is a prerequisite for +it to be scalable(build/test separation model). + +Framework test example: + +```json +{ + "tests": [ + { + "name": "web-tests-1", + "shard": "web_tests", + "subshard": "1", + "test_dependencies": [ + { + "dependency": "chrome_and_driver", + "version": "version:111.0a" + } + ] + } + ] +} +``` + +The property's description is as follows: + +* **name** the name that will be assigned to the sub-build. +* **shard** the flutter/flutter shard test to run. The supported shard names can be found +on the flutter framework [test.dart](https://github.com/flutter/flutter/blob/master/dev/bots/test.dart#L244). +* **subshard** one of the accepted subshard values for shard. Sub-shards are defined as part +of the shard implementation, please look at the corresponding shard implementation to find the +accepted values. +* **test_dependencies** a list of [dependencies](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipe_modules/flutter_deps/api.py#75) + required for the test to run. + +Engine test example: + +```json +{ + "tests": [ + { + "name": "test: lint android_debug_arm64", + "recipe": "engine_v2/tester_engine", + "drone_dimensions": [ + "device_type=none", + "os=Linux" + ], + "dependencies": [ + "host_debug", + "android_debug_arm64" + ], + "tasks": [ + { + "name": "test: lint android_debug_arm64", + "parameters": [ + "--variant", + "android_debug_arm64", + "--lint-all", + "--shard-id=0", + "--shard-variants=host_debug" + ], + "max_attempts": 1, + "script": "flutter/ci/lint.sh" + } + ] + } + ] +} +``` + +The property's description is as follows: + +* **name** the name to assign to the sub-build. +* **recipe** the recipe name to use if different than tester. +* **drone_dimensions** a list of strings with key values to select the + bot where the test will run. +* **dependencies** a list of build outputs required + by the test. These build outputs are referenced by the name of build + generating the output. This type of dependency is shared using CAS and + the contents are mounted in checkout/src/out. E.g. a build configuration + building the `host_engine` configuration will upload the content of + checkout/src/out/host_engine to CAS and a global test with a `host_engine` + dependency will mount the content of host engine in the same location of + the bot running the test. +* **tasks** a list of dictionaries representing scripts and parameters to run them. + +Example task configuration: + +```json +{ + "name": "test: lint android_debug_arm64", + "parameters": [ + "--variant", + "android_debug_arm64", + "--lint-all", + "--shard-id=0", + "--shard-variants=host_debug" + ], + "max_attempts": 1, + "script": "flutter/ci/lint.sh" +} +``` + +The property's description is as follows: + +* **name** the name assigned to the step running the script. +* **parameters** a list of parameters passed to the script execution. +* **max_attempts** an integer with the maximum number of runs in case of failure. +* **script** the path relative to checkout/src/ to run. ### Global Generators