[JNIEnv] Add support for Kotlin's unsigned types. (#4054)

Fixes: dotnet/java-interop#525
Fixes: dotnet/java-interop#543

Context: dotnet/java-interop@476597b...4565369
Context: dotnet/java-interop@71afce5
Context: https://github.com/Kotlin/KEEP/blob/13b67668ccc5b4741ecc37d0dd050fd77227c035/proposals/unsigned-types.md
Context: https://kotlinlang.org/docs/reference/basic-types.html#unsigned-integers

Bumps to xamarin/Java.Interop/master@45653697

  * dotnet/java-interop@4565369: [generator] Improve generic type lookup (#552)
  * dotnet/java-interop@71afce5: [generator] Support Kotlin's unsigned types (#539)
  * dotnet/java-interop@f26bc27: [Java.Interop] use Dictionary<string, *>(StringComparer.Ordinal) (#550)

Kotlin has experimental (!) support for unsigned types, providing the
following value types:

  * `kotlin.UByte`: marshals as a Java `byte`/JNI `B`;
    equivalent to C# `byte`
  * `kotlin.UInt`: marshals as an Java `int`/JNI `I`;
    equivalent to C# `uint`
  * `kotlin.ULong`: marshals as a Java `long`/JNI `J`;
    equivalent to C# `ulong`
  * `kotlin.UShort`: marshals as a Java `short`/JNI `S`;
    equivalent to C# `ushort`

Kotlin also provides arrays of these types:

  * `kotlin.UByteArray`: marshals as a Java `byte[]`/JNI `[B`;
    equivalent to C# `byte[]`, 
  * `kotlin.UIntArray`: marshals as an Java `int[]`/JNI `[I`;
    equivalent to C# `uint[]`.
  * `kotlin.ULongArray`: marshals as a Java `long[]`/JNI `[J`;
    equivalent to C# `ulong[]`.
  * `kotlin.UShortArray`: marshals as a Java `short[]`/JNI `[S`;
    equivalent to C# `ushort[]`.

Kotlin methods which contain unsigned types are "mangled", containing
characters which cannot be part of a valid Java identifier.  As such,
they cannot be invoked from Java code.

They *can* be invoked via JNI.

To bind these, we have two options:

 1. We could ignore all members which use unsigned types, or
 2. We could present a "Java-esque" view of the methods.

(1) is potentially problematic, as it means that abstract classes and
interfaces which contain them could become unbindable, making life
more complicated than is necessarily ideal.

(2) is viable, but ugly.

Consider this Kotlin type:

	// Kotlin
	package example;
	public open class ExampleBase {
	  public fun foo(value : UInt) {}
	}

Run `javap` on the resulting output, and we observe:

	Compiled from "hello.kt"
	public class example.ExampleBase {
	  public final void foo-WZ4Q5Ns(int);
	  public example.ExampleBase();
	}

We could bind `example.ExampleBase.foo-WZ4Q5Ns(int)` in C# by
replacing invalid characters with `_`, e.g.:

	// C# Binding
	partial class ExampleBase : Java.Lang.Object {
	  [Register ("foo-WZ4Q5Ns", …)]
	  public void Foo_WZ4Q5Ns (int value) {…}
	}

...but that would be *really* ugly.

We could instead just take everything before the `-` as the method
name for the C# method name -- resulting in `ExampleBase.Foo()` --
which results in a friendlier name from C#, but opens us up to
"collisions" with method overloads:

	// Kotlin
	public open class ExampleBase2 {
	  public fun foo(value : Int) {}
	  public fun foo(value : UInt) {}
	}

The C# `ExampleBase` type can't bind *both* `ExampleBase2.foo()`
methods as `Foo(int)`.

The chosen solution is to "more natively" support Kotlin unsigned
types, which allows reasonable disambiguation:

	// C# binding
	partial class ExampleBase2 : Java.Lang.Object {
	  [Register ("foo", …)]
	  public void Foo (int value) {…}

	  [Register ("foo-WZ4Q5Ns", …)]
	  public void Foo (uint value) {…}
	}

Java.Interop added support for emitting C# binding code which supports
Kotlin unsigned types.

For `kotlin.UByte`, `kotlin.UInt`, `kotlin.ULong`, and `kotlin.UShort`,
no additional runtime changes are needed to support the binding.

The `*Array` counterparts *do* require additional runtime changes.

Consider this Kotlin type:

	// Kotlin
	class UnsignedInstanceMethods {
	  public open fun uintArrayInstanceMethod (value: UIntArray) : UIntArray {
	    return value;
	  }
	}

The binding for this method would be:

	// C# binding
	partial class UnsignedInstanceMethods {
	  [Register ("uintArrayInstanceMethod--ajY-9A", "([I)[I", "")]
	  public unsafe uint[] UintArrayInstanceMethod (uint[] value)
	  {
	    const string __id = "uintArrayInstanceMethod--ajY-9A.([I)[I";
	    IntPtr native_value = JNIEnv.NewArray ((int[])(object)value);
	    try {
	      JniArgumentValue* __args = stackalloc JniArgumentValue [1];
	      __args [0] = new JniArgumentValue (native_value);
	      var __rm = _members.InstanceMethods.InvokeNonvirtualObjectMethod (__id, this, __args);
	      return (uint[]) JNIEnv.GetArray (__rm.Handle, JniHandleOwnership.TransferLocalRef, typeof (uint));
	    } finally {
	      if (value != null) {
	        JNIEnv.DeleteLocalRef (native_value);
	      }
	    }
	  }
	}

The problem is the `JNIEnv.GetArray(…, typeof(uint))` invocation.
This eventually hits various dictionaries within `JNIEnv`, which
requires additional support so that `uint` can be marshaled properly.

Previous versions of Xamarin.Android do not contain support for
marshaling arrays of unsigned types.

As such, support for using Kotlin bindings which contain unsigned
types will require Xamarin.Android 10.2.0 and later.


~~ Installer ~~

Add `protobuf-net.dll` to our installers, as it is required to read
the metadata that Kotlin puts within `.class` files.


~~ Unit Tests ~~

On-device tests for calling Kotlin code that uses unsigned types have
been added.

As we don't currently have a way of acquiring the Kotlin compiler for
builds, the test `.jar` and Kotlin's
`org.jetbrains.kotlin.kotlin-stdlib.jar` are copied into the repo.
They are *not* redistributed.

The source of the test `.jar` is provided for future use.


~~ Warnings ~~

Kotlin unsigned types are still experimental.  If Kotlin changes how
they are represented in Java bytecode, bindings which use them will
break.

Methods which use unsigned types *cannot* be virtual, nor can they
be used in bound interfaces.  This is because Java source code is
currently used as an "intermediary" for Java <-> Managed transitions,
and Java cannot override Kotlin "mangled" methods.  If the virtual
method is declared on a class, it will be bound as a non-`virtual`
method.  If the method is on an interface, it will be bound, but the
interface will not actually be implementable; a
[XA4213 build-time error][0] will be generated.

[0]: dotnet/java-interop@3bf5333

Author: jpobst

build-tools/installers/create-installers.targets

@@ -188,6 +188,7 @@
     <_MSBuildFiles Include="$(MSBuildSrcDir)\java_runtime.dex" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\java_runtime_fastdev.dex" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\manifestmerger.jar" />
+    <_MSBuildFiles Include="$(MSBuildSrcDir)\protobuf-net.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\SgmlReaderDll.dll" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Xamarin.Android.Aapt.targets" />
     <_MSBuildFiles Include="$(MSBuildSrcDir)\Xamarin.Android.Aapt2.targets" />

external/Java.Interop

@@ -1123,16 +1123,31 @@ static Dictionary<Type, Func<Type, IntPtr, int, Array>> CreateNativeArrayToManag
 					CopyArray (source, r);
 					return r;
 				} },
+				{ typeof (ushort), (type, source, len) => {
+					var r = new ushort [len];
+					CopyArray (source, r);
+					return r;
+				} },
 				{ typeof (int), (type, source, len) => {
 					var r = new int[len];
 					CopyArray (source, r);
 					return r;
 				} },
+				{ typeof (uint), (type, source, len) => {
+					var r = new uint[len];
+					CopyArray (source, r);
+					return r;
+				} },
 				{ typeof (long), (type, source, len) => {
 					var r = new long[len];
 					CopyArray (source, r);
 					return r;
 				} },
+				{ typeof (ulong), (type, source, len) => {
+					var r = new ulong[len];
+					CopyArray (source, r);
+					return r;
+				} },
 				{ typeof (float), (type, source, len) => {
 					var r = new float[len];
 					CopyArray (source, r);
@@ -1592,6 +1607,29 @@ public static Java.Lang.Object[] ToObjectArray<T> (T[] array)
 			return ret;
 		}
 
+		public static unsafe void CopyArray (IntPtr src, uint[] dest)
+		{
+			if (src == IntPtr.Zero)
+				return;
+			fixed (uint* __p = dest)
+				JniEnvironment.Arrays.GetIntArrayRegion (new JniObjectReference (src), 0, dest.Length, (int*) __p);
+		}
+
+		public static unsafe void CopyArray (IntPtr src, ushort[] dest)
+		{
+			if (src == IntPtr.Zero)
+				return;
+			fixed (ushort* __p = dest)
+				JniEnvironment.Arrays.GetShortArrayRegion (new JniObjectReference (src), 0, dest.Length, (short*) __p);
+		}
+
+		public static unsafe void CopyArray (IntPtr src, ulong[] dest)
+		{
+			if (src == IntPtr.Zero)
+				return;
+			fixed (ulong* __p = dest)
+				JniEnvironment.Arrays.GetLongArrayRegion (new JniObjectReference (src), 0, dest.Length, (long*) __p);
+		}
 #if ANDROID_8
 		[DllImport ("libjnigraphics.so")]
 		static extern int AndroidBitmap_getInfo (IntPtr env, IntPtr jbitmap, out Android.Graphics.AndroidBitmapInfo info);

src/Mono.Android/Android.Runtime/JNIEnv.cs

@@ -0,0 +1,92 @@
+using System;
+using Com.Xamarin.Android;
+using Foo;
+using Java.Lang;
+using NUnit.Framework;
+
+namespace Xamarin.Android.JcwGenTests
+{
+	[TestFixture]
+	public class KotlinUnsignedTypesTests
+	{
+		[Test]
+		public void TestUnsignedTypeMembers ()
+		{
+			var foo = new Foo.UnsignedInstanceMethods ();
+
+			Assert.AreEqual (uint.MaxValue, foo.UnsignedInstanceMethod (uint.MaxValue));
+
+			Assert.AreEqual (3u, foo.UnsignedInstanceProperty);
+			foo.UnsignedInstanceProperty = uint.MaxValue;
+			Assert.AreEqual (uint.MaxValue, foo.UnsignedInstanceProperty);
+
+			Assert.AreEqual (ushort.MaxValue, foo.UshortInstanceMethod (ushort.MaxValue));
+
+			Assert.AreEqual (3u, foo.UshortInstanceProperty);
+			foo.UshortInstanceProperty = ushort.MaxValue;
+			Assert.AreEqual (ushort.MaxValue, foo.UshortInstanceProperty);
+
+			Assert.AreEqual (ulong.MaxValue, foo.UlongInstanceMethod (ulong.MaxValue));
+
+			Assert.AreEqual (3u, foo.UlongInstanceProperty);
+			foo.UlongInstanceProperty = ulong.MaxValue;
+			Assert.AreEqual (ulong.MaxValue, foo.UlongInstanceProperty);
+
+			Assert.AreEqual (byte.MaxValue, foo.UbyteInstanceMethod (byte.MaxValue));
+
+			Assert.AreEqual (3u, foo.UbyteInstanceProperty);
+			foo.UbyteInstanceProperty = byte.MaxValue;
+			Assert.AreEqual (byte.MaxValue, foo.UbyteInstanceProperty);
+		}
+
+		[Test]
+		public void TestUnsignedArrayTypeMembers ()
+		{
+			var foo = new Foo.UnsignedInstanceMethods ();
+
+			var uint_array = new uint [] { 1u, 2u, uint.MaxValue };
+			var ushort_array = new ushort [] { 1, 2, ushort.MaxValue };
+			var ulong_array = new ulong [] { 1u, 2u, ulong.MaxValue };
+			var ubyte_array = new byte [] { 1, 2, byte.MaxValue };
+
+			Assert.AreEqual (uint_array, foo.UintArrayInstanceMethod (uint_array));
+			Assert.AreEqual (ushort_array, foo.UshortArrayInstanceMethod (ushort_array));
+			Assert.AreEqual (ulong_array, foo.UlongArrayInstanceMethod (ulong_array));
+			Assert.AreEqual (ubyte_array, foo.UbyteArrayInstanceMethod (ubyte_array));
+		}
+
+		[Test]
+		public void TestUnsignedTypeInterfaceImplementedMembers ()
+		{
+			var foo = new Foo.UnsignedInterfaceImplementedMethods ();
+
+			var uint_array = new uint [] { 1u, 2u, uint.MaxValue };
+
+			Assert.AreEqual (uint.MaxValue, foo.UnsignedInterfaceMethod (uint.MaxValue));
+
+			Assert.AreEqual (3u, foo.UnsignedInterfaceProperty);
+			foo.UnsignedInterfaceProperty = uint.MaxValue;
+			Assert.AreEqual (uint.MaxValue, foo.UnsignedInterfaceProperty);
+
+			Assert.AreEqual (uint_array, foo.UnsignedArrayInterfaceMethod (uint_array));
+		}
+
+		[Test]
+		public void TestUnsignedTypeAbstractImplementedMembers ()
+		{
+			var foo = new Foo.UnsignedAbstractImplementedMethods ();
+
+			Assert.AreEqual (uint.MaxValue, foo.UnsignedAbstractMethod (uint.MaxValue));
+
+			Assert.AreEqual (3u, foo.UnsignedAbstractClassProperty);
+			foo.UnsignedAbstractClassProperty = uint.MaxValue;
+			Assert.AreEqual (uint.MaxValue, foo.UnsignedAbstractClassProperty);
+		}
+
+		[Test]
+		public void TestUnsignedConstant ()
+		{
+			Assert.AreEqual (3u, Foo.UnsignedMethodsKt.SubsystemDeprecated);
+		}
+	}
+}

tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/KotlinUnsignedTypesTests.cs

@@ -52,6 +52,7 @@
   </ItemGroup>
   <ItemGroup>
     <Compile Include="BindingTests.cs" />
+    <Compile Include="KotlinUnsignedTypesTests.cs" />
     <Compile Include="DimBindingTests.cs" />
     <Compile Include="ExceptionTests.cs" />
     <Compile Include="MainActivity.cs" />

tests/CodeGen-Binding/Xamarin.Android.JcwGen-Tests/Xamarin.Android.JcwGen-Tests.csproj

@@ -0,0 +1,74 @@
+// Compile with:
+// kotlinc UnsignedMethods.kt -d KotlinUnsignedTypes.jar
+
+package foo;
+
+@kotlin.ExperimentalUnsignedTypes
+const val SUBSYSTEM_DEPRECATED: UInt = 3u
+
+@kotlin.ExperimentalUnsignedTypes
+open class UnsignedInstanceMethods {
+    public open fun unsignedInstanceMethod (value: UInt) : UInt { return value; }
+    
+    public open fun ushortInstanceMethod (value: UShort) : UShort { return value; }
+    
+    public open fun ulongInstanceMethod (value: ULong) : ULong { println ("kotlin: " + value); return value; }
+    
+    public open fun ubyteInstanceMethod (value: UByte) : UByte { return value; }
+    
+    var signedInstanceProperty: Int = 3
+  
+    var unsignedInstanceProperty: UInt = 3u
+  
+    var ushortInstanceProperty: UShort = 3u
+  
+    var ulongInstanceProperty: ULong = 3u
+  
+    var ubyteInstanceProperty: UByte = 3u
+
+    public open fun uintArrayInstanceMethod (value: UIntArray) : UIntArray { return value; }
+
+    public open fun ushortArrayInstanceMethod (value: UShortArray) : UShortArray { return value; }
+
+    public open fun ulongArrayInstanceMethod (value: ULongArray) : ULongArray { return value; }
+
+    public open fun ubyteArrayInstanceMethod (value: UByteArray) : UByteArray { return value; }
+
+    public open fun intArrayInstanceMethod (value: IntArray) : IntArray { return value; }
+}
+
+@kotlin.ExperimentalUnsignedTypes
+open class UnsignedInterfaceImplementedMethods : UnsignedInterface {
+    public override open fun unsignedInterfaceMethod (value: UInt) : UInt { return value; }
+    
+    public override open fun unsignedArrayInterfaceMethod (value: UIntArray) : UIntArray { return value; }
+
+    override var unsignedInterfaceProperty: UInt = 3u
+    
+    override var signedInterfaceProperty: Int = 3
+}
+
+@kotlin.ExperimentalUnsignedTypes
+open class UnsignedAbstractImplementedMethods : UnsignedAbstractClass () {
+    public override open fun unsignedAbstractMethod (value: UInt) : UInt { return value; }
+
+    override var unsignedAbstractClassProperty: UInt = 3u
+}
+
+@kotlin.ExperimentalUnsignedTypes
+abstract class UnsignedAbstractClass {
+    abstract fun unsignedAbstractMethod (value: UInt) : UInt
+    
+    abstract var unsignedAbstractClassProperty: UInt
+}
+
+@kotlin.ExperimentalUnsignedTypes
+interface UnsignedInterface {
+    fun unsignedInterfaceMethod (value: UInt) : UInt
+  
+    fun unsignedArrayInterfaceMethod (value: UIntArray) : UIntArray
+    
+    var unsignedInterfaceProperty: UInt
+
+    var signedInterfaceProperty: Int
+}

tests/CodeGen-Binding/Xamarin.Android.McwGen-Tests/Jars/KotlinUnsignedTypes.jar

@@ -134,6 +134,12 @@
       <OutputFile>Jars/xamarin-test.jar</OutputFile>
     </TestJarEntry>
   </ItemGroup>
+  <ItemGroup>
+    <EmbeddedJar Include="Jars\KotlinUnsignedTypes.jar" />
+  </ItemGroup>
+  <ItemGroup>
+    <EmbeddedReferenceJar Include="Jars\org.jetbrains.kotlin.kotlin-stdlib.jar" />
+  </ItemGroup>
   <Import Project="$(MSBuildExtensionsPath)\Xamarin\Android\Xamarin.Android.Bindings.targets" />
   <Import Project="..\..\..\build-tools\scripts\Jar.targets" />
   <Import Project="Xamarin.Android.McwGen-Tests.targets" />