System.AccessViolationException when running the age training

[zevele]

I've followed the instructions to perform age training, only to fail on the last step.
The last step ends after loading the test images, without any error:

D:\Projects\VS2019\FaceRecognitionDotNet-master\tools\AgeTraining>dotnet run -c Release -- train --dataset=AdienceDataset ^
More?                                --epoch=600 ^
More?                                --lr=0.001 ^
More?                                --min-lr=0.00001 ^
More?                                --min-batchsize=384 ^
More?                                --validation-interval=20
C:\Program Files\dotnet\sdk\6.0.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): war
ning NETSDK1138: The target framework 'netcoreapp2.0' is out of support and will not receive security updates in the fu
ture. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [D:\Projects\VS
2019\FaceRecognitionDotNet-master\tools\AgeTraining\AgeTraining.csproj]
C:\Program Files\dotnet\sdk\6.0.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): war
ning NETSDK1138: The target framework 'netcoreapp2.0' is out of support and will not receive security updates in the fu
ture. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [D:\Projects\VS
2019\FaceRecognitionDotNet-master\tools\AgeTraining\AgeTraining.csproj]
C:\Program Files\dotnet\sdk\6.0.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(28,5): war
ning NETSDK1138: The target framework 'netcoreapp2.0' is out of support and will not receive security updates in the fu
ture. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy. [D:\Projects\VS
2019\FaceRecognitionDotNet-master\tools\AgeTraining\AgeTraining.csproj]
            Dataset: AdienceDataset
              Epoch: 600
      Learning Rate: 0.001
  Min Learning Rate: 1E-05
     Min Batch Size: 384
Validation Interval: 20
           Use Mean: False

Start load train images
Load train images: 12188
Start load test images
Load test images: 3082

D:\Projects\VS2019\FaceRecognitionDotNet-master\tools\AgeTraining>

When I'm running the project inside VS in debug, a AccessViolationException is thrown on Line Program.cs:488
LossMulticlassLog.TrainOneStep(trainer, imageBatches[i], labelBatches[i]);
this is the first iteration (i=0).

Any ideas how I can solve this issue?

[takuya-takeuchi]

I think this isseu could be related to #205

[zevele]

I think this isseu could be related to #205

So how can I workaround this? do I need to recompile DlibDotNet with these changes reverted?

[MisterMcDuck]

Are you running this in Windows or Linux? Could you try with Debug to see if it emits any type of error message?

[zevele]

Are you running this in Windows or Linux? Could you try with Debug to see if it emits any type of error message?

I'm running windows 10 (DNN)

Regarding the debug message - Do you mean when running AgeTraining? here's the output I'm getting in VS:

'dotnet.exe' (CoreCLR: DefaultDomain): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Private.CoreLib.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'D:\Projects\VS2019\FaceRecognitionDotNet-master\tools\AgeTraining\bin\x64\Debug\netcoreapp2.0\AgeTraining.dll'. Symbols loaded.
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Runtime.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Users\******\.nuget\packages\microsoft.extensions.commandlineutils\1.1.1\lib\netstandard1.3\Microsoft.Extensions.CommandLineUtils.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.IO.FileSystem.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Runtime.Extensions.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Console.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.IO.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Collections.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Threading.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Linq.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Threading.Thread.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Users\******\.nuget\packages\dlibdotnet\19.21.0.20210302\lib\netstandard2.0\DlibDotNet.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\netstandard.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Text.RegularExpressions.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Runtime.InteropServices.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Runtime.InteropServices.RuntimeInformation.dll'. 
Exception thrown: 'System.Exception' in DlibDotNet.dll
Exception thrown: 'System.Exception' in DlibDotNet.dll
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.0.9\System.Collections.Concurrent.dll'. 
'dotnet.exe' (CoreCLR: clrhost): Loaded 'D:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\PrivateAssemblies\Runtime\Microsoft.VisualStudio.Debugger.Runtime.NetCoreApp.dll'. 
An unhandled exception of type 'System.AccessViolationException' occurred in DlibDotNet.dll
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

The exception itself does not contain any data except for the message (no stacktrace or inner exceptions).

An unhandled exception of type 'System.AccessViolationException' occurred in DlibDotNet.dll
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

I'm not sure how to perform the change in #205 myself (is it changing long to u_int32?). I've foung the files - but they are not included in the solution.

[takuya-takeuchi]

I have never seen this exception on windows. But there may be any issue.

[MisterMcDuck]

I've encountered this issue a few times, usually the culprit was either

• Multithreaded access to dlib resources
• Lack of sufficient memory available

I don't think #205 will help because you're running on windows, but I may be misunderstanding something. The issues I faced in #205 were because I was running on Linux which treats unsigned longs as a different data size than Windows.

[takuya-takeuchi]

Note
We have to fix the following codes after resolve issue on DlibDotNet.

• https://github.com/takuya-takeuchi/FaceRecognitionDotNet/blob/master/src/GenderClassification/dlib/dnn/loss/multiclass_log/gender/Gender.h
• https://github.com/takuya-takeuchi/FaceRecognitionDotNet/blob/master/src/AgeClassification/dlib/dnn/loss/multiclass_log/age/Age.h

[takuya-takeuchi]

Now, mnist example with fixed code works fine on osx and windows.

[takuya-takeuchi]

It should be resolved by 1.3.0.7