Native libraries put in MonoBundle on Mac NativeAot which dlopen cannot find

[charlesroddie]

Steps to Reproduce

Use a package involving native library (e.g. Realm, SkiaSharp) on a ios and mac project with dotnet8 NativeAOT.

Expected Behavior

The app works in both projects.

Actual Behavior

The app works on ios but on macos fails. For example with Realm (using rd.xml or TrimmerRootDescriptor to preserve types since Realm has trim incompatibilities), it fails with:

TypeInitialization_Type_NoTypeAvailable (System.TypeInitializationException)
   at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xb0
   at System.Runtime.CompilerServices.ClassConstructorRunner.CheckStaticClassConstructionReturnNonGCStaticBase(StaticClassConstructionContext*, IntPtr) + 0x14
   at Realms.SharedRealmHandle.Open(Configuration) + 0x6c
   at Realms.RealmConfiguration.CreateHandle(???) + 0x34
   at Realms.RealmConfigurationBase.CreateRealm() + 0x9c
   at LocalDbTypesQueries.DictionaryLoadSave.GetFromDictionary(DictionaryKey) + 0x58
   at LocalDb.Dictionary.getFromDictionary[a](DictionaryKey, FSharpFunc`2) + 0x1c
   at LocalDb.Dictionary.getZoom() + 0x30
   at Views.App..ctor(FSharpValueOption`1, Boolean, String, IPlatformData, FSharpOption`1) + 0x148
   at Views.App.Mac(String, IPlatformData, Func`2) + 0x68
   at Mac.AppDelegate.DidFinishLaunching(NSNotification notification) + 0x9c
   at Summatic!<BaseAddress>+0x88d2c0
DllNotFound_Mac, realm-wrappers,
dlopen(realm-wrappers.dylib, 0x0001): tried: 'realm-wrappers.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSrealm-wrappers.dylib' (no such file), '/usr/lib/realm-wrappers.dylib' (no such file, not in dyld cache), 'realm-wrappers.dylib' (no such file)
dlopen(librealm-wrappers.dylib, 0x0001): tried: 'librealm-wrappers.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibrealm-wrappers.dylib' (no such file), '/usr/lib/librealm-wrappers.dylib' (no such file, not in dyld cache), 'librealm-wrappers.dylib' (no such file)
dlopen(realm-wrappers, 0x0001): tried: 'realm-wrappers' (no such file), '/System/Volumes/Preboot/Cryptexes/OSrealm-wrappers' (no such file), '/usr/lib/realm-wrappers' (no such file, not in dyld cache), 'realm-wrappers' (no such file)
dlopen(librealm-wrappers, 0x0001): tried: 'librealm-wrappers' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibrealm-wrappers' (no such file), '/usr/lib/librealm-wrappers' (no such file, not in dyld cache), 'librealm-wrappers' (no such file)
(System.DllNotFoundException)
   at System.Runtime.InteropServices.NativeLibrary.LoadLibErrorTracker.Throw(String) + 0x4c
   at Internal.Runtime.CompilerHelpers.InteropHelpers.FixupModuleCell(InteropHelpers.ModuleFixupCell*) + 0x140
   at Internal.Runtime.CompilerHelpers.InteropHelpers.ResolvePInvokeSlow(InteropHelpers.MethodFixupCell*) + 0x40
   at Realms.SynchronizationContextScheduler.install_scheduler_callbacks(SynchronizationContextScheduler.get_context, SynchronizationContextScheduler.post_on_context, SynchronizationContextScheduler.release_context, SynchronizationContextScheduler.is_on_context) + 0x30
   at Realms.SynchronizationContextScheduler.Initialize() + 0xd4
   at Realms.NativeCommon.Initialize() + 0x24
   at System.Runtime.CompilerServices.ClassConstructorRunner.EnsureClassConstructorRun(StaticClassConstructionContext*) + 0xbc

The native dlls are placed in a MonoBundle folder which dlopen doesn't know about.
If the native libraries are placed where dlopen can find them then the app works.

Environment

Version information

Visual Studio Community 2022 for Mac
Version 17.6.7 (build 417)
Installation UUID: b89e8f6f-8fae-431f-8ba4-b760ec93affa
Runtime
.NET 7.0.3 (64-bit)
Architecture: X64
Microsoft.macOS.Sdk 13.1.1007; git-rev-head:8afca776a0a96613dfb7200e0917bb57f9ed5583; git-branch:release/7.0.1xx-xcode14.2
Roslyn (Language Service)
4.6.0-3.23180.6+99e956e42697a6dd886d1e12478ea2b27cceacfa
NuGet
Version: 6.4.0.117
.NET SDK (x64)
SDK: /usr/local/share/dotnet/sdk/8.0.100/Sdks
SDK Versions:
8.0.100
7.0.311
6.0.417
MSBuild SDKs: /Applications/Visual [Studio.app/Contents/MonoBundle/MSBuild/Current/bin/Sdks](http://studio.app/Contents/MonoBundle/MSBuild/Current/bin/Sdks)
.NET Runtime (x64)
Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
8.0.0
7.0.14
6.0.25
Xamarin.Profiler
Version: 1.8.0.49
Location: /Applications/Xamarin [Profiler.app/Contents/MacOS/Xamarin](http://profiler.app/Contents/MacOS/Xamarin) Profiler
Updater
Version: 11
Apple Developer Tools
Xcode: 15.0.1 22266
Build: 15A507
Xamarin.Mac
Version: 9.3.0.23 Visual Studio Community
Hash: 9defd91b3
Branch: xcode14.3
Build date: 2023-10-23 16:14:59-0400
Xamarin.iOS
Version: 16.4.0.23 Visual Studio Community
Hash: 9defd91b3
Branch: xcode14.3
Build date: 2023-10-23 16:15:00-0400
Xamarin Designer
Version: 17.6.3.9
Hash: 2648399ae8
Branch: remotes/origin/d17-6
Build date: 2023-11-10 12:19:21 UTC
Xamarin.Android
Version: 13.2.2.0 (Visual Studio Community)
Commit: xamarin-android/d17-5/45b0e14
Android SDK: /Users/user/Library/Developer/Xamarin/android-sdk-macosx
Supported Android versions:
13.0 (API level 33)
SDK Command-line Tools Version: 7.0
SDK Platform Tools Version: 34.0.3
SDK Build Tools Version: 32.0.0
Build Information:
Mono: d9a6e87
Java.Interop: xamarin/java.interop/d17-5@149d70fe
SQLite: xamarin/sqlite/3.40.1@68c69d8
Xamarin.Android Tools: xamarin/xamarin-android-tools/d17-5@ca1552d
Microsoft Build of OpenJDK
Java SDK: /Library/Java/JavaVirtualMachines/microsoft-11.jdk
11.0.16.1
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL
Eclipse Temurin JDK
Java SDK: /Library/Java/JavaVirtualMachines/temurin-8.jdk
1.8.0.302
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL
Android SDK Manager
Version: 17.6.0.50
Hash: a715dca
Branch: HEAD
Build date: 2023-11-10 12:19:27 UTC
Android Device Manager
Version: 0.0.0.1309
Hash: 06e3e77
Branch: HEAD
Build date: 2023-11-10 12:19:27 UTC
Build Information
Release ID: 1706070417
Git revision: 179aa48a5702754e41a8218d0403ddc3be46706f
Build date: 2023-11-10 12:17:47+00
Build branch: release-17.6
Build lane: release-17.6
Operating System
Mac OS X 14.1.1
Darwin 23.1.0 Darwin Kernel Version 23.1.0
Mon Oct 9 21:27:27 PDT 2023
root:xnu-10002.41.9~6/RELEASE_X86_64 x86_6

Build Logs

To be added on request

Example Project (If Possible)

This can be simply reproduced with SkiaSharp. We are putting relevant code here. Can share a project privately (tricky on ios/mac because builds are hard, there are certificates, etc.).

csproj:

<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0-macos</TargetFramework>
<OutputType>Exe</OutputType>
<Nullable>enable</Nullable>
<ImplicitUsings>true</ImplicitUsings>
<SupportedOSPlatformVersion>10.15</SupportedOSPlatformVersion>
<RuntimeIdentifiers>osx-x64</RuntimeIdentifiers>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
<CreatePackage>false</CreatePackage>
<UseSGen>false</UseSGen>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<CreatePackage>false</CreatePackage>
<UseSGen>false</UseSGen>
<PublishAot>true</PublishAot>
<_IsPublishing>true</_IsPublishing>
<TrimmerSingleWarn>false</TrimmerSingleWarn>
<IsAotCompatible>true</IsAotCompatible>
<TrimMode>full</TrimMode>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="SkiaSharp.Views" Version="2.88.6" />
</ItemGroup>
</Project>

ViewController.cs (largely taken from https://github.com/mattleibow/skiasharp-samples/blob/main/Basic/macOS/SkiaSharpSample/ViewController.cs )

public partial class ViewController : NSViewController {
	protected ViewController (NativeHandle handle) : base (handle)
	{
		// This constructor is required if the view controller is loaded from a xib or a storyboard.
		// Do not put any initialization here, use ViewDidLoad instead.
	}
    private SkiaSharp.Views.Mac.SKCanvasView skiaView;
    public override void ViewDidLoad()
    {
        base.ViewDidLoad();
        if (skiaView is null)
        {
            skiaView = new SKCanvasView
            {
                Frame = new CGRect(0, 0, 600, 400)
            };
            this.View = skiaView;
        }
        skiaView.IgnorePixelScaling = true;
        skiaView.PaintSurface += OnPaintSurface;
    }
 
    private void OnPaintSurface(object sender, SKPaintSurfaceEventArgs e)
    {
        // the the canvas and properties
        var canvas = e.Surface.Canvas;
 
        // make sure the canvas is blank
        canvas.Clear(SKColors.White);
 
        // draw some text
        var paint = new SKPaint
        {
            Color = SKColors.Black,
            IsAntialias = true,
            Style = SKPaintStyle.Fill,
            TextAlign = SKTextAlign.Center,
            TextSize = 24
        };
        var coord = new SKPoint(e.Info.Width / 2, (e.Info.Height + paint.TextSize) / 2);
        canvas.DrawText("SkiaSharp", coord, paint);
    }
 
    public override NSObject RepresentedObject {
		get => base.RepresentedObject;
		set {
			base.RepresentedObject = value;
 
			// Update the view, if already loaded.
		}
	}
}

[charlesroddie]

@rolfbjarne @filipnavara

[charlesroddie]

It appears that MonoBundle is the right place for native libs on mac but the step to add it as an rpath (#4435 ) isn't happening on NativeAOT.

A workaround is to include <MonoBundlingExtraArgs>--link_flags="-rpath '@executable_path/../MonoBundle'</MonoBundlingExtraArgs>.

[rolfbjarne]

I was not able to reproduce using SkiaSharp, everything works as expected, and the native executable has an rpath:

$ otool -l ./bin/Debug/net8.0-macos/issue19520.app/Contents/MacOS/issue19520|g RPATH -A 2
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../MonoBundle/ (offset 12)

So could you please create a test project to reproduce this?

If you want to submit it privately, you can create a new ticket here: https://developercommunity.visualstudio.com/VisualStudioMac/report, and then you can add private attachments once the ticket has been created. Please add the link to the new ticket here if you do this, so that we can easily connect them.

[FoggyFinder]

$ otool -l ./bin/Debug/net8.0-macos/issue19520.app/Contents/MacOS/issue19520|g RPATH -A 2

@rolfbjarne Seems like it's Debug mode while bug is present only in Release mode with NativeAot enabled

So could you please create a test project to reproduce this?

https://github.com/FoggyFinder/MacOSApp1

tested with dotnet publish -r osx-x64 -c Release

[rolfbjarne]

OK, I figured it out, thanks for the test case. A fix is in progress.