Add magic bytes validation for Mach-O binaries in DotnetHostHelper

[Copilot]

This PR ports the magic bytes validation fix for Mach-O binaries from TestFX PR #6033 to address the issue where GetMuxerArchitectureByMachoOnMac was reading magic bytes but never validating them against known Mach-O magic numbers.

Problem

The GetMuxerArchitectureByMachoOnMac method in DotnetHostHelper.cs was reading magic bytes from binary files but never validating them against known Mach-O magic numbers. This meant the method could attempt to process non-Mach-O binaries as if they were valid Mach-O files, potentially leading to incorrect architecture detection.

Solution

Added validation of magic bytes to ensure we're actually looking at a valid Mach-O binary before proceeding with CPU type detection.

Changes Made

1. Added Mach-O magic number constants based on the Wikipedia Mach-O specification:

   • MachOMagic32BigEndian = 0xfeedface (32-bit big-endian)
   • MachOMagic64BigEndian = 0xfeedfacf (64-bit big-endian)
   • MachOMagic32LittleEndian = 0xcefaedfe (32-bit little-endian)
   • MachOMagic64LittleEndian = 0xcffaedfe (64-bit little-endian)
   • MachOMagicFatBigEndian = 0xcafebabe (multi-architecture big-endian)

2. Added validation logic that checks the magic bytes against all valid Mach-O magic numbers before attempting architecture detection

3. Enhanced logging to report invalid magic bytes for debugging purposes

Implementation Details

• Endianness handling: Correctly handles both big-endian and little-endian Mach-O binaries by validating against all magic number variants
• Minimal changes: Only 13 lines added, no existing functionality removed or modified
• Backward compatibility: All existing behavior is preserved; this only adds missing validation
• Error handling: Invalid magic bytes are logged and the method returns null, allowing fallback to other detection methods

The implementation aligns with the existing pattern in the codebase where CPU types also have both "Magic" and "Cigam" (byte-swapped) variants to handle different endianness.

Fixes #15226.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• yttvsblobprodcus357.vsblob.vsassets.io
  • Triggering command: /home/REDACTED/work/vstest/vstest/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/vstest/vstest/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/vstest/vstest/artifacts/toolset/10.0.0-beta.25358.3.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.