From 6f3d24ffefef5b6828c0c7b2e209648dd61df331 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 11:41:42 +0300 Subject: [PATCH 01/14] Clean up the doc for the library testing on Android --- .../testing/libraries/testing-android.md | 25 ++++++++++++++++--- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 23b6e07d924140..322358bd629402 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -1,18 +1,23 @@ # Testing Libraries on Android +## Prerequisites + The following dependencies should be installed in order to be able to run tests: +- OpenJDK - Android NDK - Android SDK -- OpenJDK -- OpenSSL + +To manage the dependencies, you can install them via terminal or using Android Studio. + +### Using a terminal OpenJDK can be installed on Linux (Ubuntu) using `apt-get`: ```bash sudo apt-get install openjdk-8 zip unzip ``` -Android SDK, NDK and OpenSSL can be automatically installed via the following script: +Android SDK and NDK can be automatically installed via the following script: ```bash #!/usr/bin/env bash set -e @@ -21,7 +26,6 @@ NDK_VER=r21b SDK_VER=6200805_latest SDK_API_LEVEL=29 SDK_BUILD_TOOLS=29.0.3 -OPENSSL_VER=1.1.1g-alpha-1 if [[ "$OSTYPE" == "darwin"* ]]; then HOST_OS=darwin @@ -47,8 +51,21 @@ yes | ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROI ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROID_SDK_ROOT} "platform-tools" "platforms;android-${SDK_API_LEVEL}" "build-tools;${SDK_BUILD_TOOLS}" ``` +### Using Android Studio + +Android Studio offers a convenient UI: +- to install all the dependencies; +- to manage android virtual devices; +- to make easy use of adb logs. + ## Building Libs and Tests for Android +Before running a build you might want to set the Android SDK and NDK environment variables: +``` +export ANDROID_SDK_ROOT= +export ANDROID_NDK_ROOT= +``` + Now we're ready to build everything for Android: ``` ./build.sh mono+libs -os Android -arch x64 From a17fe25a37d3256455360c19418e7b286639f484 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 12:27:31 +0300 Subject: [PATCH 02/14] Add a section about functional tests --- docs/workflow/testing/libraries/testing-android.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 322358bd629402..55fbf7dcfdc65e 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -83,6 +83,17 @@ The following shows how to run tests for a specific library ./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64 ``` +### Running the functional tests + +There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform. + +A functional test can be run the same way as any library test suite, e.g.: +``` +./dotnet.sh build /t:Test /p:TargetOS=Android /p:TargetArchitecture=x64 /p:Configuration=Release src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj +``` + +Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. + ### Test App Design Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed. From 6a10f2dd088b6a6562de617fe1b77262f9207f33 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 14:08:23 +0300 Subject: [PATCH 03/14] Update the doc for testing on apple mobile platforms --- .../testing/libraries/testing-apple.md | 47 ++++++++++++++++--- 1 file changed, 41 insertions(+), 6 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 99adf6a87176b8..df0523ddcc2455 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -1,18 +1,43 @@ -# Testing Libraries on iOS and tvOS +# Testing Libraries on iOS, tvOS, and MacCatalyst -In order to build libraries and tests for iOS or tvOS you need recent version of XCode installed (e.g. 11.3 or higher). +## Prerequisites -Build Libraries for iOS Simulator: +- XCode 11.3 or higher + +## Building Libs and Tests + +You can build and run the library tests: +- on a simulator; +- on a device. + +Run the following command in a terminal: +``` +./build.sh mono+libs -os -arch x64 +``` +where `` is one of the following: + +| simulator | device| +|:--------------|:------| +| iOSSimulator | iOS | +| tvOSSimulator | tvOS | +| MacCatalyst | | + + +e.g., to build for a iOS simulator, run: ``` ./build.sh mono+libs -os iOSSimulator -arch x64 ``` + Run tests one by one for each test suite on a simulator: ``` ./build.sh libs.tests -os iOSSimulator -arch x64 -test ``` + +### Building for a device + In order to run the tests on a device: -- Set the os to `iOS` instead of `iOSSimulator` -- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`): +- Set the `-os` parameter to a device-related value (see above); +- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`), e.g.: ``` ./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5 ``` @@ -24,11 +49,21 @@ In order to run the tests on a device: ./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64 ``` +### Running the functional tests + +There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform. + +A functional test can be run the same way as any library test suite, e.g.: +``` +./dotnet.sh build /t:Test /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 /p:Configuration=Release src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj +``` + +Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. + ### Test App Design iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets. ### Existing Limitations -- Most of the test suites crash on devices due to #35674 - Simulator uses JIT mode only at the moment (to be extended with FullAOT and Interpreter) - Interpreter is not enabled yet. From 9a3f04441b82cb4fe627dfa87b1deed6b268e190 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 16:20:06 +0300 Subject: [PATCH 04/14] Initial note on testing various configurations on apple mobile platforms --- .../testing/libraries/testing-apple.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index df0523ddcc2455..4e8524f190a16b 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -60,6 +60,24 @@ A functional test can be run the same way as any library test suite, e.g.: Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. +### Testing various configurations + +It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoEnableInterpreter`, and some more. + +1. Interpreter Only + +This configuration is necessary for hot reload scenarios. + +To enable the interpreter, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=true` to a build command. + +2. AOT only + +To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command. + +3. AOT-LLVM + +To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command. + ### Test App Design iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets. From c550324b6e2497df687444ae93f5278f799b1f86 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 16:25:32 +0300 Subject: [PATCH 05/14] Trailing spaces --- docs/workflow/testing/libraries/testing-apple.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 4e8524f190a16b..90870d977f2263 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -72,11 +72,11 @@ To enable the interpreter, add `/p:RunAOTCompilation=true /p:MonoEnableInterpret 2. AOT only -To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command. +To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command. 3. AOT-LLVM -To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command. +To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command. ### Test App Design iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test From 886789d65008e90338391e40fcd760f33d9de476 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 16:43:07 +0300 Subject: [PATCH 06/14] Fix links --- docs/workflow/testing/libraries/testing-apple.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 90870d977f2263..22588c65ff65de 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -36,12 +36,14 @@ Run tests one by one for each test suite on a simulator: ### Building for a device In order to run the tests on a device: -- Set the `-os` parameter to a device-related value (see above); -- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`), e.g.: +- Set the `-os` parameter to a device-related value (see above) +- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`). + +For example: ``` ./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5 ``` -[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices. +[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices. ### Running individual test suites - The following shows how to run tests for a specific library: @@ -79,7 +81,7 @@ To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpre To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command. ### Test App Design -iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test +iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets. ### Existing Limitations From f3a23f43b4f6f895bb9fb93b2937cbc8e492ce9a Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 16:50:59 +0300 Subject: [PATCH 07/14] Fix a link --- docs/workflow/testing/libraries/testing-android.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 55fbf7dcfdc65e..83693fe976d90f 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -95,7 +95,7 @@ A functional test can be run the same way as any library test suite, e.g.: Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. ### Test App Design -Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test +Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed. ### Obtaining the logs From d045edbac1014a3429eecf4d2ca50d9a89e2dec5 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 16:51:21 +0300 Subject: [PATCH 08/14] Fix a typo --- docs/workflow/testing/libraries/testing-android.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 83693fe976d90f..53eac7c69d0bce 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -103,7 +103,7 @@ XHarness for Android doesn't talk much and only saves test results to a file. Ho ``` adb logcat -s "DOTNET" ``` -Or simply open `logcat` window in Android Studio or Visual Stuido. +Or simply open `logcat` window in Android Studio or Visual Studio. ### AVD Manager If Android Studio is installed, [AVD Manager](https://developer.android.com/studio/run/managing-avds) can be used from the IDE to create and start Android virtual devices. Otherwise, the Android SDK provides the [`avdmanager` command line tool](https://developer.android.com/studio/command-line/avdmanager). From d550d51e45652435a0b2192dbf263c3d17df984e Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Tue, 21 Sep 2021 16:54:15 +0300 Subject: [PATCH 09/14] Trailing spaces --- docs/workflow/testing/libraries/testing-apple.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 22588c65ff65de..e86a373dfd0dbe 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -37,8 +37,8 @@ Run tests one by one for each test suite on a simulator: In order to run the tests on a device: - Set the `-os` parameter to a device-related value (see above) -- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`). - +- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`). + For example: ``` ./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5 From 494d96e9b1ddab72b0672bf346435ff606accd9f Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Wed, 22 Sep 2021 09:39:51 +0300 Subject: [PATCH 10/14] Use a build parameter for configuration Co-authored-by: Viktor Hofer --- docs/workflow/testing/libraries/testing-android.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 53eac7c69d0bce..449f319bd9119e 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -89,7 +89,7 @@ There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tes A functional test can be run the same way as any library test suite, e.g.: ``` -./dotnet.sh build /t:Test /p:TargetOS=Android /p:TargetArchitecture=x64 /p:Configuration=Release src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj +./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj ``` Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. From 819c8524e1c4492bf227ecd66f6ddcd75e75eefc Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Wed, 22 Sep 2021 09:40:32 +0300 Subject: [PATCH 11/14] Use a build parameter for configuration Co-authored-by: Viktor Hofer --- docs/workflow/testing/libraries/testing-apple.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index e86a373dfd0dbe..7b52ac60a645e5 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -57,7 +57,7 @@ There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tes A functional test can be run the same way as any library test suite, e.g.: ``` -./dotnet.sh build /t:Test /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 /p:Configuration=Release src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj +./dotnet.sh build /t:Test -c Release /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj ``` Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. From ea6d84938f51d9c5a18ee0a96072941bd5f3c43a Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Thu, 23 Sep 2021 16:16:35 +0300 Subject: [PATCH 12/14] Add the functional tests section to the Mono general testing document --- docs/workflow/testing/mono/testing.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/workflow/testing/mono/testing.md b/docs/workflow/testing/mono/testing.md index 4c37f634de45c5..3e2eec366153d1 100644 --- a/docs/workflow/testing/mono/testing.md +++ b/docs/workflow/testing/mono/testing.md @@ -79,7 +79,20 @@ For example, the following command is for running System.Runtime tests: make run-tests-corefx-System.Runtime ``` ### Mobile targets and WebAssembly -Build and run library tests against Webassembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/) +Build and run library tests against WebAssembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/) + +## Running the functional tests + +There are the [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on Android, iOS-like platforms (iOS/tvOS + simulators, MacCatalyst), and WebAssembly. + +A functional test can be run the same way as any library test suite, e.g.: +``` +./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj +``` + +Currently the functional tests are expected to return `42` as a success code so please be careful when adding a new one. + +For more details, see instructions located in [Library testing document folder](../libraries/). # Running the Mono samples There are a few convenient samples located in `$(REPO_ROOT)/src/mono/sample`, which could help you test your program easily with different flavors of Mono or do a sanity check on the build. The samples are set up to work with a specific configuration; please refer to the relevant Makefile for specifics. If you would like to work with a different configuration, you can edit the Makefile. From 9af3c51949b4f1b2249966e750e6c70bde56ce00 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Mon, 4 Oct 2021 14:49:54 +0300 Subject: [PATCH 13/14] address the feedback --- .../testing/libraries/testing-android.md | 16 +++++++ .../testing/libraries/testing-apple.md | 46 +++++++++++++++---- 2 files changed, 53 insertions(+), 9 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 449f319bd9119e..e3c1c4e79c38d0 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -94,6 +94,22 @@ A functional test can be run the same way as any library test suite, e.g.: Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. +### Testing various configurations + +It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoForceInterpreter`, and some more. + +1. AOT + +To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoForceInterpreter=false` to a build command. + +2. AOT-LLVM + +To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoForceInterpreter=false /p:MonoEnableLLVM=true` to a build command. + +3. Interpreter + +To build for Interpreter mode, add `/p:RunAOTCompilation=false /p:MonoForceInterpreter=true` to a build command. + ### Test App Design Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed. diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 7b52ac60a645e5..7045ae66d0c263 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -3,7 +3,10 @@ ## Prerequisites - XCode 11.3 or higher - +- a certificate and provisioning profile if using a device +- a simulator with a proper device type and OS version. +Go `XCode > Window > Devices and Simulators` to revise the list of the available simulators and then `"+" button on bottom left > OS Version dropdown selection > Download more simulator runtimes` in case you need to download more simulators. + ## Building Libs and Tests You can build and run the library tests: @@ -12,18 +15,20 @@ You can build and run the library tests: Run the following command in a terminal: ``` -./build.sh mono+libs -os -arch x64 +./build.sh mono+libs -os -arch ``` where `` is one of the following: +- iOSSimulator +- tvOSSimulator +- MacCatalyst +- iOS +- tvOS -| simulator | device| -|:--------------|:------| -| iOSSimulator | iOS | -| tvOSSimulator | tvOS | -| MacCatalyst | | - +and `` is one of the following: +- x64 +- arm64 (for device) -e.g., to build for a iOS simulator, run: +e.g., to build for an iOS simulator, run: ``` ./build.sh mono+libs -os iOSSimulator -arch x64 ``` @@ -46,11 +51,29 @@ For example: [AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices. ### Running individual test suites + - The following shows how to run tests for a specific library: ``` ./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64 ``` +Also you can run the built test app through Xcode by opening the corresponding `.xcodeproj` and setting up the right scheme, app, and even signing if using a local device. + +There's also an option to run a `.app` through `xcrun`, which is simulator specific. Consider the following shell script: + +``` + xcrun simctl shutdown + xcrun simctl boot + open -a Simulator + xcrun simctl install + xcrun simctl launch --console booted +``` + +where +`` is a name of the simulator to start, e.g. `"iPhone 11"`, +`` is a path to the iOS test app bundle, +`` is a name of the iOS test app + ### Running the functional tests There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform. @@ -62,6 +85,11 @@ A functional test can be run the same way as any library test suite, e.g.: Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. +### Viewing logs +- see the logs generated by the XHarness tool +- use the `Console` app on macOS: +`Command + Space`, type in `Console`, search the appropriate process (System.Buffers.Tests for example). + ### Testing various configurations It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoEnableInterpreter`, and some more. From 186372836f3de44e3a053a5f33d37526384676b5 Mon Sep 17 00:00:00 2001 From: Maxim Lipnin Date: Mon, 4 Oct 2021 16:33:32 +0300 Subject: [PATCH 14/14] Trailing spaces --- docs/workflow/testing/libraries/testing-apple.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 7045ae66d0c263..c303fc737362d2 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -4,9 +4,9 @@ - XCode 11.3 or higher - a certificate and provisioning profile if using a device -- a simulator with a proper device type and OS version. +- a simulator with a proper device type and OS version. Go `XCode > Window > Devices and Simulators` to revise the list of the available simulators and then `"+" button on bottom left > OS Version dropdown selection > Download more simulator runtimes` in case you need to download more simulators. - + ## Building Libs and Tests You can build and run the library tests: @@ -22,7 +22,7 @@ where `` is one of the following: - tvOSSimulator - MacCatalyst - iOS -- tvOS +- tvOS and `` is one of the following: - x64 @@ -69,9 +69,9 @@ There's also an option to run a `.app` through `xcrun`, which is simulator speci xcrun simctl launch --console booted ``` -where -`` is a name of the simulator to start, e.g. `"iPhone 11"`, -`` is a path to the iOS test app bundle, +where +`` is a name of the simulator to start, e.g. `"iPhone 11"`, +`` is a path to the iOS test app bundle, `` is a name of the iOS test app ### Running the functional tests @@ -87,7 +87,7 @@ Currently functional tests are expected to return `42` as a success code so plea ### Viewing logs - see the logs generated by the XHarness tool -- use the `Console` app on macOS: +- use the `Console` app on macOS: `Command + Space`, type in `Console`, search the appropriate process (System.Buffers.Tests for example). ### Testing various configurations