Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 34 additions & 6 deletions docs/workflow/testing/libraries/testing-android.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,23 @@
# Testing Libraries on Android

## Prerequisites

The following dependencies should be installed in order to be able to run tests:

- OpenJDK
- Android NDK
- Android SDK
- OpenJDK
- OpenSSL

To manage the dependencies, you can install them via terminal or using Android Studio.

### Using a terminal

OpenJDK can be installed on Linux (Ubuntu) using `apt-get`:
```bash
sudo apt-get install openjdk-8 zip unzip
```

Android SDK, NDK and OpenSSL can be automatically installed via the following script:
Android SDK and NDK can be automatically installed via the following script:
```bash
#!/usr/bin/env bash
set -e
Expand All @@ -21,7 +26,6 @@ NDK_VER=r21b
SDK_VER=6200805_latest
SDK_API_LEVEL=29
SDK_BUILD_TOOLS=29.0.3
OPENSSL_VER=1.1.1g-alpha-1

if [[ "$OSTYPE" == "darwin"* ]]; then
HOST_OS=darwin
Expand All @@ -47,8 +51,21 @@ yes | ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROI
${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROID_SDK_ROOT} "platform-tools" "platforms;android-${SDK_API_LEVEL}" "build-tools;${SDK_BUILD_TOOLS}"
```

### Using Android Studio

Android Studio offers a convenient UI:
- to install all the dependencies;
- to manage android virtual devices;
- to make easy use of adb logs.

## Building Libs and Tests for Android

Before running a build you might want to set the Android SDK and NDK environment variables:
```
export ANDROID_SDK_ROOT=<PATH-TO-ANDROID-SDK>
export ANDROID_NDK_ROOT=<PATH-TO-ANDROID-NDK>
```

Now we're ready to build everything for Android:
```
./build.sh mono+libs -os Android -arch x64
Expand All @@ -66,16 +83,27 @@ The following shows how to run tests for a specific library
./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
```

### Running the functional tests

There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform.

A functional test can be run the same way as any library test suite, e.g.:
```
./dotnet.sh build /t:Test /p:TargetOS=Android /p:TargetArchitecture=x64 /p:Configuration=Release src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj
```

Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one.

### Test App Design
Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed.

### Obtaining the logs
XHarness for Android doesn't talk much and only saves test results to a file. However, you can also subscribe to live logs via the following command:
```
adb logcat -s "DOTNET"
```
Or simply open `logcat` window in Android Studio or Visual Stuido.
Or simply open `logcat` window in Android Studio or Visual Studio.

### AVD Manager
If Android Studio is installed, [AVD Manager](https://developer.android.com/studio/run/managing-avds) can be used from the IDE to create and start Android virtual devices. Otherwise, the Android SDK provides the [`avdmanager` command line tool](https://developer.android.com/studio/command-line/avdmanager).
Expand Down
71 changes: 63 additions & 8 deletions docs/workflow/testing/libraries/testing-apple.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,89 @@
# Testing Libraries on iOS and tvOS
# Testing Libraries on iOS, tvOS, and MacCatalyst

In order to build libraries and tests for iOS or tvOS you need recent version of XCode installed (e.g. 11.3 or higher).
## Prerequisites

Build Libraries for iOS Simulator:
- XCode 11.3 or higher

## Building Libs and Tests

You can build and run the library tests:
- on a simulator;
- on a device.

Run the following command in a terminal:
```
./build.sh mono+libs -os <TARGET_OS> -arch x64
```
where `<TARGET_OS>` is one of the following:

| simulator | device|
|:--------------|:------|
| iOSSimulator | iOS |
| tvOSSimulator | tvOS |
| MacCatalyst | |


e.g., to build for a iOS simulator, run:
```
./build.sh mono+libs -os iOSSimulator -arch x64
```

Run tests one by one for each test suite on a simulator:
```
./build.sh libs.tests -os iOSSimulator -arch x64 -test
```

### Building for a device

In order to run the tests on a device:
- Set the os to `iOS` instead of `iOSSimulator`
- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`):
- Set the `-os` parameter to a device-related value (see above)
- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`).

For example:
```
./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5
```
[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices.
[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices.

### Running individual test suites
- The following shows how to run tests for a specific library:
```
./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64
```

### Running the functional tests

There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform.

A functional test can be run the same way as any library test suite, e.g.:
```
./dotnet.sh build /t:Test /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 /p:Configuration=Release src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj
```

Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one.

### Testing various configurations

It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoEnableInterpreter`, and some more.

1. Interpreter Only

This configuration is necessary for hot reload scenarios.

To enable the interpreter, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=true` to a build command.

2. AOT only

To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command.

3. AOT-LLVM

To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command.

### Test App Design
iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets.

### Existing Limitations
- Most of the test suites crash on devices due to #35674
- Simulator uses JIT mode only at the moment (to be extended with FullAOT and Interpreter)
- Interpreter is not enabled yet.