Support modern AWS EC2 instances like T4g

[mbacarella]

Is Solo5 the right platform for this?

Perhaps this could be a meta-issue describing the work required. My goal is to get Mirage running on T4g instances at AWS.

[hannesm]

Hello @mbacarella, would you mind to extend a bit on what would be needed to run a VM on such a T4g instance?

[dinosaure]

Several questions are needed:

1. which assembler such processor uses? It seems a 64-bit ARM assembler (aarch64/arm64) which is good for us but experimental
2. how the AWS hypervizor talk to the operating system? We know that, on Google Cloud, we advises to use virtio but I never tried AWS, do you have some pointers for us?

[mbacarella]

Yes, it's ARM64. These run on AWS's Nitro virtualization platform, which is their replacement for Xen.

Poking around a Linux T4g instance:

# lscpu
Architecture:           aarch64
  CPU op-mode(s):       32-bit, 64-bit
  Byte Order:           Little Endian
CPU(s):                 2
  On-line CPU(s) list:  0,1
Vendor ID:              ARM
  Model name:           Neoverse-N1
    Model:              1
    Thread(s) per core: 1
    Core(s) per socket: 2
    Socket(s):          1
    Stepping:           r3p1
    BogoMIPS:           243.75
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdr
                        dm lrcpc dcpop asimddp ssbs
Caches (sum of all):
  L1d:                  128 KiB (2 instances)
  L1i:                  128 KiB (2 instances)
  L2:                   2 MiB (2 instances)
  L3:                   32 MiB (1 instance)
NUMA:
  NUMA node(s):         1
  NUMA node0 CPU(s):    0,1
Vulnerabilities:
  Itlb multihit:        Not affected
  L1tf:                 Not affected
  Mds:                  Not affected
  Meltdown:             Not affected
  Mmio stale data:      Not affected
  Retbleed:             Not affected
  Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:           Mitigation; __user pointer sanitization
  Spectre v2:           Mitigation; CSV2, BHB
  Srbds:                Not affected
  Tsx async abort:      Not affected

# lspci
00:00.0 Host bridge: Amazon.com, Inc. Device 0200
00:01.0 Serial controller: Amazon.com, Inc. Device 8250
00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)

According to this doc, a Linux instance appears to need to be installed/configured with some Amazon specific drivers.
https://aws.amazon.com/premiumsupport/knowledge-center/boot-error-linux-nitro-instance/

The drivers are here:
https://github.com/amzn/amzn-drivers

So, one way forward with the fewest variables changed is select an x86-64 instance type that runs on Nitro (t3.nano?), follow the Mirage AWS EC2 on Xen instructions (https://mirage.io/docs/xen-boot) and maybe hope for screen output.

From there, we can explore adding those drivers to Solo5. Somewhere in there, also try ARM64.