x64 vs ARM64 Microbenchmarks Performance Study Report

[adamsitnik]

Recently @kunalspathak asked me if I could produce a report similar to #66848 for x64 vs arm64 comparison.

I took .NET 7 Preview2 results provided by @AndyAyersMS, @kunalspathak and myself for #66848, hacked the tool a little bit (it was not designed to compare different architecture results) and compared x64 vs arm64 using the following configs:

• my old 4 year old macBook Pro x64: macOS Monterey 12.2.1, Intel Core i7-5557U CPU 3.10GHz (Broadwell), 1 CPU, 4 logical and 2 physical cores vs @AndyAyersMS M1 Max arm64: macOS Monterey 12.2.1, Apple M1 Max 2.40GHz, 1 CPU, 10 logical and 10 physical cores
• @kunalspathak Windows 10 (10.0.20348.587) Intel Xeon Platinum 8272CL CPU 2.60GHz, 2 CPU, 104 logical and 52 physical cores vs @kunalspathak Windows 11 (10.0.25058.1000) ARM64 machine with lots of cores

Of course it was not an apples-to-apples comparision, just the best thing we could do right now.

Full public results (without absolute values, as I don't have the permission to share them) can be found here.
Internal MS results (with absolute values) can be found here. If you don't have the access please ping me on Teams.

As usual, I've focused on the benchmarks that take longer to execute on arm64 compared to x64. If you are interested in benchmarks that take less to execute, you need to read the report linked above in the reverse order.

Benchmarks:

@kunalspathak

• System.Numerics.Tests.Perf_BitOperations.PopCount_ulong is 5-8 time slower (most likely due to lack of vectorization). PopCount_uint is slower only on Windows.

@tannergooding @GrabYourPitchforks

• lot of Base64Encode benchmarks like System.Buffers.Text.Tests.Base64Tests.Base64Encode(NumberOfBytes: 1000) are 6 up to 16 times slower Optimize System.Buffers for arm64 using cross-platform intrinsics #35033

@stephentoub @kouvel

• Some RentReturnArrayPoolTests benchmarks are up to few times slower, but these are multi-threaded and very often multimodal benchmarks. Faster thread local statics #63619
• System.Threading.Tests.Perf_Timer.AsynchronousContention is 2-3 times slower.

@wfurt @MihaZupan

• A lot of SocketSendReceivePerfTest benchmarks likeSystem.Net.WebSockets.Tests.SocketSendReceivePerfTest.ReceiveSend are 2 times slower.

@dotnet/area-system-drawing

• System.Drawing.Tests.Perf_Image_Load.Image_FromStream_NoValidation are few times slower on Windows. Only the NoValidation benchmarks seem to run slower.

@stephentoub

• Few RegularExpressions benchmarks like System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sher[a-z]+|Hol[a-z]+", Options: Compiled) are 40-50% slower. This pattern uses IndexOfAny("HOho") to find the next possible match location. It has a 256-bit vectorization path on x64 but only 128-bit on ARM64.

@jkotas @AndyAyersMS

• PerfLabTests.LowLevelPerf.GenericClassGenericStaticField benchmark can be from 16% to x3 times slower. Same goes for PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod.

@dotnet/jit-contrib

• System.Security.Cryptography.Tests.Perf_Hashing.Sha1 is 17-55% slower. (Potentially differences in the GDI+ code)
• System.IO.Tests.Perf_StreamWriter.WriteString(writeLength: 100) is 21-46% slower.
• System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream benchmark can be from 16% to x4 times slower. Optimize System.Buffers for arm64 using cross-platform intrinsics #35033
• SIMD.ConsoleMandel benchmarks are 40% slower . Double Vector128 for SpanHelpers.IndexOf(byte,byte,int) on ARM64 #66993
• Burgers.Test3 is 12-59% slower Double Vector128 for SpanHelpers.IndexOf(byte,byte,int) on ARM64 #66993
• A lot of System.Collections.Contains benchmarks are 2-3 times slower (most likely due to lack of vectorization). Same goes for System.Memory.Span<Char>.IndexOfValue, System.Memory.Span<Char>.Fill, System.Memory.Span<Int32>.StartsWith, System.Memory.Span<Byte>.IndexOfAnyTwoValues and System.Memory.ReadOnlySpan.IndexOfString(Ordinal). Double Vector128 for SpanHelpers.IndexOf(byte,byte,int) on ARM64 #66993
• A lot of SequenceCompareTo benchmarks are 30% up to 4 times slower Double Vector128 for SpanHelpers.IndexOf(byte,byte,int) on ARM64 #66993

@tannergooding

• System.MathBenchmarks.Double.Exp and System.MathBenchmarks.Single.Exp are 35% slower. Optimize jump stubs on arm64 #62302

@dotnet/area-system-globalization

• System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: ja) benchmark can be from 20% to x7 times slower (it's most likely an ICU problem). Initializing the "ja" culture takes 200ms when using ICU #31273

• Various Perf_Interlocked benchmarks are slower, but this is expected due to memory model differences.

• Various Perf_Process.Start benchmarks are slower, but only on macOS so it's most likely a macOS issue.

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Recently @kunalspathak asked me if I could produce a report similar to #66848 for x64 vs arm64 comparison.

I took .NET 7 Preview2 results provided by @AndyAyersMS, @kunalspathak and myself for #66848, hacked the tool a little bit (it was not designed to compare different architecture results) and compared x64 vs arm64 using the following configs:

• my old 4 year old macBook Pro x64: macOS Monterey 12.2.1, Intel Core i7-5557U CPU 3.10GHz (Broadwell), 1 CPU, 4 logical and 2 physical cores vs @AndyAyersMS M1 Max arm64: macOS Monterey 12.2.1, Apple M1 Max 2.40GHz, 1 CPU, 10 logical and 10 physical cores
• @kunalspathak Windows 10 (10.0.20348.587) Intel Xeon Platinum 8272CL CPU 2.60GHz, 2 CPU, 104 logical and 52 physical cores vs @kunalspathak Windows 11 (10.0.25058.1000) ARM64 machine with lots of cores

Of course it was not an apples-to-apples comparision, just the best thing we could do right now.

Full public results (without absolute values, as I don't have the permission to share them) can be found here.
Internal MS results (with absolute values) can be found here. If you don't have the access please ping me on Teams.

As usual, I've focused on the benchmarks that take longer to execute on arm64 compared to x64. If you are interested in benchmarks that take less to execute, you need to read the report linked above in the reverse order.

Benchmarks:

• A lot of Base64Encode benchmarks like System.Buffers.Text.Tests.Base64Tests.Base64Encode(NumberOfBytes: 1000) are 6 up to 16 times slower (most likely due to lack of vectorization). @tannergooding @GrabYourPitchforks is it expected?
• System.Numerics.Tests.Perf_BitOperations.PopCount_ulong is 5-8 time slower (most likely due to lack of vectorization). PopCount_uint is slower only on Windows. @kunalspathak is this expected?
• Some RentReturnArrayPoolTests benchmarks are up to few times slower, but these are multi-threaded and very often multimodal benchmarks. @stephentoub @kouvel is it expected?
• System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: ja) benchmark can be from 20% to x7 times slower (it's most likely an ICU problem). @dotnet/area-system-globalization is it expected?
• A lot of System.Collections.Contains benchmarks are 2-3 times slower (most likely due to lack of vectorization). Same goes for System.Memory.Span<Char>.IndexOfValue, System.Memory.Span<Char>.Fill, System.Memory.Span<Int32>.StartsWith, System.Memory.Span<Byte>.IndexOfAnyTwoValues and System.Memory.ReadOnlySpan.IndexOfString(Ordinal). @tannergooding @EgorBo is it expected?
• A lot of SequenceCompareTo benchmarks are 30% up to 4 times slower (most likely due to lack of vectorization). @tannergooding @EgorBo is it expected?
• System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeToStream benchmark can be from 16% to x4 times slower. @dotnet/jit-contrib is this expected?
• System.Threading.Tests.Perf_Timer.AsynchronousContention is 2-3 times slower. @stephentoub @kouvel is it expected?
• A lot of SocketSendReceivePerfTest benchmarks likeSystem.Net.WebSockets.Tests.SocketSendReceivePerfTest.ReceiveSend are 2 times slower. @wfurt @MihaZupan is it expected?
• System.Drawing.Tests.Perf_Image_Load.Image_FromStream_NoValidation are few times slower on Windows. @dotnet/area-system-drawing is it expected? Only the NoValidation benchmarks seem to run slower.
• PerfLabTests.LowLevelPerf.GenericClassGenericStaticField benchmark can be from 16% to x3 times slower. Same goes for PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod. @jkotas @AndyAyersMS is it expected?
• Few RegularExpressions benchmarks like System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sher[a-z]+|Hol[a-z]+", Options: Compiled) are 40-50% slower (most likely it's using a method that has not been vectorized). @stephentoub is it expected?
• Burgers.Test3 is 12-59% slower (most likely it's using a method that has not been vectorized). @dotnet/jit-contrib is it expected?
• System.Security.Cryptography.Tests.Perf_Hashing.Sha1 is 17-55% slower (most likely due to lack of vectorization). @dotnet/jit-contrib is it expected?
• SIMD.ConsoleMandel benchmarks are 40% slower (most likely due to lack of vectorization). @dotnet/jit-contrib is it expected?
• System.IO.Tests.Perf_StreamWriter.WriteString(writeLength: 100) is 21-46% slower (most likely due to lack of vectorization). @dotnet/jit-contrib is it expected?
• System.MathBenchmarks.Double.Exp and ystem.MathBenchmarks.Single.Exp are 35% slower. @tannergooding is it expected?
• Various Perf_Interlocked benchmarks are slower, but this is expected due to memory model differences.
• Various Perf_Process.Start benchmarks are slower, but only on macOS so it's most likely a macOS issue.

Author:	adamsitnik
Assignees:	-
Labels:	area-Meta, tenet-performance, tracking
Milestone:	-

[EgorBo]

Nice! I did a similar report last week and shared on our perf meeting last Monday

A lot of Base64Encode benchmarks like System.Buffers.Text.Tests.Base64Tests.Base64Encode(NumberOfBytes: 1000) are 6 up to 16 times slower (most likely due to lack of vectorization). @tannergooding @GrabYourPitchforks is it expected?

Base64 (for utf8) is only vectorized for x64, there is an issue for arm64 #35033 (I think we wanted to assign it to someone to ramp up)

---

System.Numerics.Tests.Perf_BitOperations.PopCount_ulong is 5-8 time slower (most likely due to lack of vectorization).

it is properly accelerated (I compared it with __builtin_popcnt in LLVM), the problem that popcnt is vector only on arm64 so we have some overhead on packing/extracting - 5 instructions vs 1 on x64

---

Some RentReturnArrayPoolTests benchmarks are up to few times slower

My guess that Rent-Return is most likely bottle-necked on TLS access speed, can be improved with #63619 if arm64 has special registers for that.

---

A lot of System.Collections.Contains benchmarks are 2-3 times slower (most likely due to lack of vectorization).

A lot of SequenceCompareTo benchmarks are 30% up to 4 times slower (most likely due to lack of vectorization

That is expected due to lack of Vector256 I believe, I proposed to add dual-vector128 for arm64 here #66993

Burgers.Test3 is 12-59% slower (most likely it's using a method that has not been vectorized)

SIMD.ConsoleMandel benchmarks are 40% slower

Same here, it uses Vector<T> so it's Vector256 on x64 vs Vector128 on arm64

---

Various Perf_Interlocked benchmarks are slower, but this is expected due to memory model differences.

Correct, the codegen for interlocked ops is completely fine on both arm64 8.0 and 8.1 (Atomics)

---

System.MathBenchmarks.Double.Exp and ystem.MathBenchmarks.Single.Exp are 35% slower.

If arm64 was M1 than it's the jump-stubs issue, see #62302 (comment)

---

PerfLabTests.LowLevelPerf.GenericClassGenericStaticField benchmark can be from 16% to x3 times slower. Same goes for PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod. @jkotas @AndyAyersMS is it expected?

My guess that it's because we don't use relocs on arm64 and have to compose full 64bit address using several instructions to access a static field. E.g.:

static int field;

void IncrementField() => field++;

X64:

       FF05C6CC4200         inc      dword ptr [(reloc 0x7ffeb73eac3c)]

arm64:

        D2958780          movz    x0, #0xac3c
        F2B6E760          movk    x0, #0xb73b LSL #16
        F2CFFFC0          movk    x0, #0x7ffe LSL #32
        B9400001          ldr     w1, [x0]
        11000421          add     w1, w1, #1
        B9000001          str     w1, [x0]

Overall, I have a feeling that we might get a very nice boost for many benchmarks/GC if we integrate PGO for native code (VM/GC)

[vcsjones]

System.Security.Cryptography.Tests.Perf_Hashing.Sha1 is 17-55% slower (most likely due to lack of vectorization). jit-contrib is it expected?

The SHA1.ComputeHash is going to be backed by the platform's SHA1 implementation (OpenSSL, CNG, SecurityTransforms) and doesn't do any vectorization itself anywhere. It's possible that the platform the tests were run under do not have optimized ARM64 implementations of SHA1.

[danmoseley]

Nice! I did a similar report last week and shared on our perf meeting last Monday

@EgorBo that data seems like something you could share on a gist for everyone? (Or perhaps just the scenarios with unusual ratios)

[danmoseley]

The System.Drawing ones may just be a difference in Windows GDI+ performance since it's largely a wrapper.

[AndyAyersMS]

PerfLabTests.LowLevelPerf.GenericClassGenericStaticField benchmark can be from 16% to x3 times slower. Same goes for PerfLabTests.LowLevelPerf.GenericClassGenericStaticMethod. @jkotas @AndyAyersMS is it expected?

My guess that it's because we don't use relocs on arm64 and have to compose full 64bit address using several instructions to access a static field.

https://github.com/dotnet/performance/blob/d7dac8a7ca12a28d099192f8a901cf8e30361384/src/benchmarks/micro/runtime/perflab/LowLevelPerf.cs#L320-L325

Access for generic statics (for shared generics at least, maybe for all?) can more complicated -- the address must be looked up in runtime data structures. Worth investigating.

[tarekgh]

System.Globalization.Tests.Perf_DateTimeCultureInfo.Parse(culturestring: ja) benchmark can be from 20% to x7 times slower (it's most likely an ICU problem).

Most likely it is because of ICU. We already having the issue #31273 tracking that. I don't know though why ARM64 runs make more slower.

[danmoseley]

Access for generic statics (for shared generics at least, maybe for all?) can more complicated -- the address must be looked up in runtime data structures. Worth investigating.

@EgorBo perhaps you could open an issue and update the top post?

[EgorBo]

@EgorBo perhaps you could open an issue and update the top post?

Access for generic statics (for shared generics at least, maybe for all?) can more complicated -- the address must be looked up in runtime data structures. Worth investigating.

right, but it doesn't look to be the case here since it's not shared

@EgorBo that data seems like something you could share on a gist for everyone?

Sure, let me see how to export an excel sheet to gist 😄

[ericstj]

The System.Drawing ones may just be a difference in Windows GDI+ performance since it's largely a wrapper.

There is a lot of interop in this scenario. Could be differences in interop or performance of this callback

runtime/src/libraries/System.Drawing.Common/src/System/Drawing/Internal/GPStream.COMWrappers.cs

Line 29 in 3ae8739

public unsafe Interop.HRESULT CopyTo(IntPtr pstm, ulong cb, ulong* pcbRead, ulong* pcbWritten)

Could compare to performance of a load that doesn't use stream, and thus would be more of a GDI+ baseline. cc @eerhardt