diff --git a/docs/design/coreclr/jit/hot-cold-splitting.md b/docs/design/coreclr/jit/hot-cold-splitting.md new file mode 100644 index 00000000000000..a68cda3d11e553 --- /dev/null +++ b/docs/design/coreclr/jit/hot-cold-splitting.md @@ -0,0 +1,173 @@ +# Hot/Cold Code Splitting + +This document describes the current state of hot/cold splitting in the JIT. + +Hot/Cold splitting is an optimization that splits code into frequently-executed ("hot") and rarely-executed ("cold") +parts, and places them in separate memory regions. Increased hot code density better leverages spatial locality, +improving application performance via fewer instruction cache misses, less OS paging, and fewer TLB misses. + +## Background + +The JIT previously supported hot/cold splitting for AOT-compiled NGEN images in .NET Framework. With Crossgen2 support +[in progress](https://github.com/dotnet/runtimelab/tree/feature/hot-cold-splitting) (and no existing support for +splitting dynamically-generated code), JIT support has not been tested since retiring .NET Framework -- thus, there +are likely regressions. Furthermore, the JIT never supported splitting functions with certain features, like exception +handling or switch tables. Finally, with ARM64 code generation being a newer addition to the JIT, hot/cold splitting +was never implemented for the architecture. These limitations significantly inhibit the applicability of hot/cold splitting. + +The below sections describe various improvements made to the JIT's hot/cold splitting support to remove such limitations. + +## Testing the JIT Without Runtime Support + +Without runtime support for hot/cold splitting in .NET as of summer 2022, testing the JIT's existing hot/cold splitting +support is not as simple as turning the feature on. A new "fake" splitting mode, enabled by the +`COMPlus_JitFakeProcedureSplitting` environment variable, removes this dependency on runtime support. This mode allows +the JIT to execute its hot/cold splitting workflow without changing the runtime's behavior. This workflow proceeds as +follows: + +* The JIT identifies where to split the function in `Compiler::fgDetermineFirstColdBlock`, as usual. +* In `Compiler::eeAllocMem`, the JIT requests one memory buffer from the host (either Crossgen2 or the VM) for the +entire function; this is unlike normal splitting, where separate buffers are allocating for the hot/cold sections. + * After the host has allocating the buffer, the JIT manually sets the cold code pointers to right after the hot section. + * Note there is no space between the hot/cold sections, unlike with normal splitting. The instructions are still contiguous. +* During code generation, the JIT emits instructions as if the hot/cold sections are arbitrarily far away: + * Jumps between hot/cold sections are long. + * The JIT reports jump target relocations to the host as necessary. + * On some platforms like ARM64 (see below), the JIT emits certain pseudo-instructions to handle the instruction section's +lack of contiguousness. +* For the sake of simplicity, the JIT generates unwind info as if it is not splitting. Because the hot/cold sections +are adjacent, the JIT generates unwind info once for the entire function. + +While enabling fake-splitting also enables `opts.compProcedureSplitting`, there is no guarantee the JIT will fake-split +a function unless `Compiler::fgDetermineFirstColdBlock` finds a splitting point; without PGO data, the JIT's heuristics +may be too conservative for extensive testing. To aid regression testing, the JIT also has a stress-splitting mode now, +under `COMPlus_JitStressProcedureSplitting`. When `opts.compProcedureSplitting` and stress-splitting are both enabled, +the JIT splits every function after its first basic block; in other words, `fgFirstColdBlock` is always +`fgFirstBB->bbNext`. The rest of the hot/cold splitting workflow is the same: The JIT emits instructions to handle the +split code sections and, if fake-splitting, utilizes only one memory buffer. + +When used in tandem, fake-splitting and stress-splitting have strong potential to reveal regressions in the JIT's +hot/cold splitting functionality without runtime support. As such, a new rolling test job in the +[runtime-jit-experimental](https://dev.azure.com/dnceng/public/_build?definitionId=793), `jit_stress_splitting`, runs +all `dotnet/runtime` tests with fake-splitting and stress-splitting enabled. + +### PRs + +* [runtime/69763](https://github.com/dotnet/runtime/pull/69763): Implement fake-splitting and stress-splitting modes +* [runtime/69922](https://github.com/dotnet/runtime/pull/69922): Add `jit_stress_splitting` to `runtime_jit_experimental` + +## ARM64 Support + +After devising strategies for testing the JIT independently of runtime support for splitting, achieving functional +parity for ARM64 became a priority. While initial splitting prototypes in Crossgen2 target x64, the JIT can +achieve some correctness with hot/cold splitting on ARM64 by leveraging fake-splitting alone. + +Most of the JIT's hot/cold splitting workflow is architecture-independent; only code generation is ARM64-specific. +The majority of implementation work here is thus related to emitting various long pseudo-instructions: + +* On both ARM32 and ARM64, conditional jumps have less range than unconditional jumps due to the architectures' fixed +instruction width. Normally, the conditional jump's range is large enough to cover any reasonably-sized function. +With splitting enabled, hot/cold sections can be arbitrarily far apart for dynamically-generated code, and up to +232 bits apart in AOT-compiled code (this is the maximum code size allowed in PE files). To avoid +arbitrarily limiting code sizes, conditional jumps must have the same range as unconditional jumps. "Jump stubs" solve +this by replacing each conditional jump with a negated conditional jump, followed by an unconditional jump to the +original target -- this pseudo-instruction's format is `IF_LARGEJMP`. For example, `branch condition, target` becomes +the following: + +``` +branch !condition, pc+1 +branch target +``` + +* Without splitting, the read-only data section is adjacent to the function's instruction section on ARM64. When +splitting, the data section is adjacent to the hot section; from the hot section, we can load constants with a single +`ldr` instruction. However, this is not possible from the cold section: Because it is arbitrarily far away, the target +address cannot be determined relative to the PC. Instead, the JIT emits a `IF_LARGELDC` pseudoinstruction with a +few different possibilities: + * First, compute the target page address with an `adrp` instruction. + * Case 1: Load the constant into a general register with a `ldr` instruction. (Final sequence: `adrp + ldr`) + * If the destination register is a vector register, move the value from the general register with a `fmov` +instruction. (Final sequence: `adrp + ldr + fmov`) + * Case 2: If the constant is 16 bytes in size, load it directly into a vector register. + * General registers are 8 bytes in width on ARM64. Thus, they cannot temporarily hold the constant. + * Compute the exact address with an `add` instruction, and load the constant with an `ld1` instruction. +(Final sequence: `adrp + add + ld1`) + +Aside from these pseudo-instructions, hot/cold splitting required a few other tweaks to ARM64 code generation: +* When emitting long jumps between hot/cold sections, the JIT reports the target's relocation to the host with the +relocation type `IMAGE_REL_ARM64_BRANCH26`. +* While enabling fake-splitting did not require changes here, it is worth noting an importance difference in unwind info +generation on x64 versus ARM64. On x64, the JIT emits the full unwind info for a hot function fragment, and emits +"chained" unwind info for the cold function fragment. This chained unwind info does not contain any unwind codes, but +instead points to the hot fragment's unwind info. When unwinding, the VM will use this chained info to find the relevant +unwind info. + +There is no concept of chained unwind info on ARM64; instead, the JIT generates unwind info for each function fragment, +regardless of its hot/cold status. While this should not have any immediate implications for JIT work around hot/cold +splitting, this does affect the feature's implementation in Crossgen2 and the VM. On x64, the Crossgen2 splitting +prototype uses chained unwind info to differentiate between cold main body fragments and cold EH funclets (see below for +details on EH splitting). This comparison is not possible on ARM64 -- the JIT may have to pass more information to the +host when generating unwind info on ARM64 to indicate if a cold fragment is a funclet. + +### PRs + +* [runtime/70708](https://github.com/dotnet/runtime/pull/70708): Enable fake-splitting on ARM64 + +## Splitting Functions with Exception Handling (EH) + +An EH funclet is a "mini-function" for handling or filtering exceptions; for example, for a conventional "try/catch" +expression, the catch block becomes a funclet (this is true for finally/fault/filter/etc. blocks as well). The +JIT places EH funclets contiguously in memory, adjacent to the main function body. Because of the prevalence of +exception handling in .NET programs, enabling splitting of EH funclets massively expands this optimization's +applicability. + +Because EH funclets immediately succeed the main function body, the JIT can easily split such functions without +breaking existing invariants: + +* If the JIT finds a split point in the main body, it splits there as usual. The latter part of the main body, +along with all of the function's EH funclets, is cold. +* If the JIT does not find a split point in the main body, and none of the funclets execute frequently, it splits +at the beginning of the funclet section. The main body is hot, and all EH funclets become cold. +* Else, no splitting occurs. + +This approach may not be the most performant implementation: Splitting funclets individually could yield better +spatial locality. However, this would require re-arranging the order of funclets (currently, there is no specific +order imposed), and significantly altering unwind info generation, thus breaking many invariants in the host. This +approach enables splitting in many more scenarios without breaking existing invariants or introducing +architecture-specific workarounds. However, if the JIT supports splitting functions multiple times in the future, we +should revisit this. + +In the absence of PGO data, the JIT assumes exceptions occur rarely; this justifies moving handlers to the cold section. +Because `finally` blocks execute regardless of an exception occurring, it may be detrimental to make these handlers +cold. Thus, [Compiler::fgCloneFinally](https://github.com/dotnet/runtime/blob/41419131095d36fb5b811600ad0dab3b0d804269/src/coreclr/jit/fgehopt.cpp#L617) +copies the `finally` block to the hot section, provided it is not too large. Once runtime support for splitting matures, +we should revisit this optimization to ensure the JIT is not too sparse or overzealous in its usage. + +### PRs + +* [runtime/71236](https://github.com/dotnet/runtime/pull/71236): Enable hot/cold splitting of EH funclets +* [runtime/71273](https://github.com/dotnet/runtime/pull/71273): Disable `HANDLER_ENTRY_MUST_BE_IN_HOT_SECTION` +* [runtimelab/1923](https://github.com/dotnet/runtimelab/pull/1923): Fix unwind info for cold EH funclets on x64 +* [runtimelab/1930](https://github.com/dotnet/runtimelab/pull/1930): Fix unwind info for cold EH funclets on ARM64 + +## Future Work + +As of writing, support for hot/cold splitting in Crossgen2 on x64 is in progress. While some future tasks are JIT-specific +and will not require runtime support to begin work, many will require close collaboration. See the `dotnet/runtimelab` +hot/cold splitting [prototype](https://github.com/dotnet/runtimelab/tree/feature/hot-cold-splitting) for +runtime-specific tasks. + +* Profile runtime effects of hot/cold splitting. Since we are largely interested in how this affects spatial locality, +key metrics could include number of hot/cold page touches, number of jumps to the cold section taken, number of +instruction cache misses, etc. It is important that such profiling utilizes PGO data, as the JIT's splitting heuristics +are quite sparse, and may not be useful for measuring performance. +* Enable hot/cold splitting of functions with switch tables. +* Work with Crossgen2 prototype to support hot/cold splitting on ARM64. + * This task will specifically require work in the JIT for differentiating cold funclets from regular cold code. +On x64, Crossgen2 and the VM use chained unwind info (or lack thereof) to differentiate the two. Since there is no +concept of chained unwind info on ARM64, the JIT may need to report more information to the host. +* Support hot/cold splitting of dynamically-compiled code. + * Since the JIT has historically never supported splitting jitted code, it may be interesting to measure the overhead +of performing hot/cold splitting during runtime. +* Support hot/cold splitting of NativeAOT code. + * Most of the work here will likely involve generating unwind info correctly.