Replace setjmp/longjmp usage in Wasmtime

Since Wasmtime's inception it's used the `setjmp` and `longjmp`
functions in C to implement handling of traps. While this solution was
easy to implement, relatively portable, and performant enough, there are
a number of downsides that have evolved over time to make this an
unattractive approach in the long run:

* Using `setjmp` fundamentally requires using C because Rust does not
  understand a function that returns twice. It's fundamentally unsound
  to invoke `setjmp` in Rust meaning that Wasmtime has forever needed a
  C compiler configured and set up to build. This notably means that
  `cargo check` cannot check other targets easily.

* Using `longjmp` means that Rust function frames are unwound on the
  stack without running destructors. This is a dangerous operation of
  which we get no protection from the compiler about. Both frames
  entering wasm and frames exiting wasm are all skipped. Absolutely
  minimizing this has been beneficial for portability to platforms such
  as Pulley.

* Currently the no_std implementation of Wasmtime requires embedders to
  provide `wasmtime_{setjmp,longjmp}` which is a thorn in the side of
  what is otherwise a mostly entirely independent implementation of
  Wasmtime.

* There is a performance floor to using `setjmp` and `longjmp`. Calling
  `setjmp` requires using C but Wasmtime is otherwise written in Rust
  meaning that there's a Rust->C->Rust->Wasm boundary which
  fundamentally can't be inlined without cross-language LTO which is
  difficult to configure.

* With the implementation of the WebAssembly exceptions proposal
  Wasmtime now has two means of unwinding the stack. Ideally Wasmtime
  would only have one, and the more general one is the method of
  exceptions.

* Jumping out of a signal handler on Unix is tricky business. While
  we've made it work it's generally most robust of the signal handler
  simply returns which it now does.

With all of that in mind the purpose of this commit is to replace the
setjmp/longjmp mechanism of handling traps with the recently implemented
support for exceptions in Cranelift. That is intended to resolve all of
the above points in one swoop.

One point in particular though that's nice about setjmp/longjmp is that
unwinding the stack on a trap is an O(1) operation. For situations such
as stack overflow that's a particularly nice property to have as we can
guarantee embedders that traps are a constant time (albeit somewhat
expensive with signals) operation. Exceptions naively require unwinding
the entire stack, and although frame pointers mean we're just traversing
a linked list I wanted to preserve the O(1) property here nonetheless.
To achieve this a solution is implemented where the array-to-wasm
(host-to-wasm) trampolines setup state in `VMStoreContext` so looking up
the current trap handler frame is an O(1) operation. Namely the sp/fp/pc
values for a `Handler` are stored inline.

Implementing this feature required supporting
relocations-to-offsets-in-functions which was not previously supported
by Wasmtime. This required Cranelift refactorings such as #11570, #11585,
and #11576. This then additionally required some more refactoring in
this commit which was difficult to split out as it otherwise wouldn't be
tested.

Apart from the relocation-related business much of this change is about
updating the platform signal handlers to use exceptions instead of
longjmp to return. For example on Unix this means updating the
`ucontext_t` with register values that the handler specifies. Windows
involves updating similar contexts, and macOS mach ports ended up not
needing too many changes.

In terms of overall performance the relevant benchmark from this
repository, compared to before this commit, is:

    sync/no-hook/core - host-to-wasm - typed - nop
			    time:   [10.552 ns 10.561 ns 10.571 ns]
			    change: [−7.5238% −7.4011% −7.2786%] (p = 0.00 < 0.05)
			    Performance has improved.

Closes #3927
cc #10923

prtest:full

Author: alexcrichton

.github/workflows/main.yml

@@ -318,7 +318,7 @@ jobs:
   micro_checks:
     name: Check ${{matrix.name}}
     strategy:
-      fail-fast: true
+      fail-fast: ${{ github.event_name != 'pull_request' }}
       matrix:
         include:
           - name: wasmtime
@@ -504,7 +504,7 @@ jobs:
     name: "Platform: ${{ matrix.target }}"
     runs-on: ${{ matrix.os }}
     strategy:
-      fail-fast: true
+      fail-fast: ${{ github.event_name != 'pull_request' }}
       matrix:
         include:
         - target: x86_64-unknown-freebsd
@@ -627,7 +627,7 @@ jobs:
     if: needs.determine.outputs.test-capi
 
     strategy:
-      fail-fast: true
+      fail-fast: ${{ github.event_name != 'pull_request' }}
       matrix:
         os: [ubuntu-24.04, macos-15, windows-2025]
 

cranelift/codegen/src/isa/aarch64/inst/mod.rs

@@ -2905,6 +2905,10 @@ pub enum LabelUse {
     /// 21-bit offset for ADR (get address of label). PC-rel, offset is not shifted. Immediate is
     /// 21 signed bits, with high 19 bits in bits 23:5 and low 2 bits in bits 30:29.
     Adr21,
+    /// Corresponds to `Reloc::Aarch64AdrPrelPgHi21`
+    AdrpHi21,
+    /// Corresponds to `Reloc::Aarch64AddAbsLo12Nc`
+    AddLo12,
     /// 32-bit PC relative constant offset (from address of constant itself),
     /// signed. Used in jump tables.
     PCRel32,
@@ -2928,6 +2932,10 @@ impl MachInstLabelUse for LabelUse {
             // range.
             LabelUse::Adr21 => (1 << 20) - 1,
             LabelUse::PCRel32 => 0x7fffffff,
+
+            // Similar to `PCRel32`
+            LabelUse::AdrpHi21 => 0x7fffffff,
+            LabelUse::AddLo12 => 0x7fffffff,
         }
     }
 
@@ -2958,6 +2966,8 @@ impl MachInstLabelUse for LabelUse {
             LabelUse::Ldr19 => 0x00ffffe0,    // bits 23..5 inclusive
             LabelUse::Adr21 => 0x60ffffe0,    // bits 30..29, 25..5 inclusive
             LabelUse::PCRel32 => 0xffffffff,
+            LabelUse::AdrpHi21 => (0x3 << 29) | (0x1ffffc << 3),
+            LabelUse::AddLo12 => 0xfff << 10,
         };
         let pc_rel_shifted = match self {
             LabelUse::Adr21 | LabelUse::PCRel32 => pc_rel,
@@ -2972,6 +2982,13 @@ impl MachInstLabelUse for LabelUse {
             LabelUse::Branch26 => pc_rel_shifted & 0x3ffffff,
             LabelUse::Adr21 => (pc_rel_shifted & 0x7ffff) << 5 | (pc_rel_shifted & 0x180000) << 10,
             LabelUse::PCRel32 => pc_rel_shifted,
+            LabelUse::AdrpHi21 => {
+                let page = |offset| (offset & !0xfff) as i64;
+                let pc_rel = page(label_offset) - page(use_offset);
+                let pc_rel_shifted = (pc_rel >> 12) as u32;
+                ((pc_rel_shifted & 0x3) << 29) | ((pc_rel_shifted & 0x1ffffc) << 3)
+            }
+            LabelUse::AddLo12 => (label_offset & 0xfff) << 10,
         };
         let is_add = match self {
             LabelUse::PCRel32 => true,
@@ -3059,9 +3076,11 @@ impl MachInstLabelUse for LabelUse {
         }
     }
 
-    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<LabelUse> {
-        match (reloc, addend) {
-            (Reloc::Arm64Call, 0) => Some(LabelUse::Branch26),
+    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<(LabelUse, Addend)> {
+        match reloc {
+            Reloc::Arm64Call => Some((LabelUse::Branch26, addend)),
+            Reloc::Aarch64AdrPrelPgHi21 => Some((LabelUse::AdrpHi21, addend)),
+            Reloc::Aarch64AddAbsLo12Nc => Some((LabelUse::AddLo12, addend)),
             _ => None,
         }
     }

cranelift/codegen/src/isa/call_conv.rs

@@ -84,7 +84,7 @@ impl CallConv {
     /// Does this calling convention support exceptions?
     pub fn supports_exceptions(&self) -> bool {
         match self {
-            CallConv::Tail | CallConv::SystemV => true,
+            CallConv::Tail | CallConv::SystemV | CallConv::Winch => true,
             _ => false,
         }
     }

cranelift/codegen/src/isa/pulley_shared/inst.isle

@@ -705,9 +705,9 @@
 
 ;; Helper for creating `MInst.LoadExtName*` instructions.
 (decl load_ext_name (BoxExternalName i64 RelocDistance) XReg)
-(rule 1 (load_ext_name name offset (RelocDistance.Near))
+(rule (load_ext_name name offset (RelocDistance.Near))
   (load_ext_name_near name offset))
-(rule 0 (load_ext_name name offset (RelocDistance.Far))
+(rule (load_ext_name name offset (RelocDistance.Far))
   (load_ext_name_far name offset))
 
 ;;;; Helpers for Emitting Calls ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

cranelift/codegen/src/isa/pulley_shared/inst/mod.rs

@@ -906,9 +906,9 @@ impl MachInstLabelUse for LabelUse {
         }
     }
 
-    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<LabelUse> {
-        match (reloc, addend) {
-            (Reloc::PulleyPcRel, 0) => Some(LabelUse::PcRel),
+    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<(LabelUse, Addend)> {
+        match reloc {
+            Reloc::PulleyPcRel => Some((LabelUse::PcRel, addend)),
             _ => None,
         }
     }

cranelift/codegen/src/isa/riscv64/inst/mod.rs

@@ -1796,9 +1796,11 @@ impl MachInstLabelUse for LabelUse {
         (veneer_offset, Self::PCRel32)
     }
 
-    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<LabelUse> {
-        match (reloc, addend) {
-            (Reloc::RiscvCallPlt, _) => Some(Self::PCRel32),
+    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<(LabelUse, Addend)> {
+        match reloc {
+            Reloc::RiscvCallPlt => Some((Self::PCRel32, addend)),
+            Reloc::RiscvPCRelHi20 => Some((Self::PCRelHi20, addend)),
+            Reloc::RiscvPCRelLo12I => Some((Self::PCRelLo12I, addend)),
             _ => None,
         }
     }

cranelift/codegen/src/isa/s390x/inst/mod.rs

@@ -3513,10 +3513,10 @@ impl MachInstLabelUse for LabelUse {
         unreachable!();
     }
 
-    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<Self> {
-        match (reloc, addend) {
-            (Reloc::S390xPCRel32Dbl, 2) => Some(LabelUse::PCRel32Dbl),
-            (Reloc::S390xPLTRel32Dbl, 2) => Some(LabelUse::PCRel32Dbl),
+    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<(Self, Addend)> {
+        match reloc {
+            Reloc::S390xPCRel32Dbl => Some((LabelUse::PCRel32Dbl, addend - 2)),
+            Reloc::S390xPLTRel32Dbl => Some((LabelUse::PCRel32Dbl, addend - 2)),
             _ => None,
         }
     }

cranelift/codegen/src/isa/x64/inst/mod.rs

@@ -1630,9 +1630,9 @@ impl MachInstLabelUse for LabelUse {
         }
     }
 
-    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<Self> {
-        match (reloc, addend) {
-            (Reloc::X86CallPCRel4, -4) => Some(LabelUse::JmpRel32),
+    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<(Self, Addend)> {
+        match reloc {
+            Reloc::X86CallPCRel4 => Some((LabelUse::JmpRel32, addend + 4)),
             _ => None,
         }
     }

cranelift/codegen/src/machinst/buffer.rs

@@ -658,15 +658,8 @@ impl<I: VCodeInst> MachBuffer<I> {
 
     /// Bind a label to the current offset. A label can only be bound once.
     pub fn bind_label(&mut self, label: MachLabel, ctrl_plane: &mut ControlPlane) {
-        trace!(
-            "MachBuffer: bind label {:?} at offset {}",
-            label,
-            self.cur_offset()
-        );
-        debug_assert_eq!(self.label_offsets[label.0 as usize], UNKNOWN_LABEL_OFFSET);
-        debug_assert_eq!(self.label_aliases[label.0 as usize], UNKNOWN_LABEL);
         let offset = self.cur_offset();
-        self.label_offsets[label.0 as usize] = offset;
+        self.bind_label_at_offset(label, offset);
         self.lazily_clear_labels_at_tail();
         self.labels_at_tail.push(label);
 
@@ -681,6 +674,14 @@ impl<I: VCodeInst> MachBuffer<I> {
         // Post-invariant: by `optimize_branches()` (see argument there).
     }
 
+    /// Bind a label to the specified offset. A label can only be bound once.
+    fn bind_label_at_offset(&mut self, label: MachLabel, offset: CodeOffset) {
+        trace!("MachBuffer: bind label {label:?} at offset {offset}");
+        debug_assert_eq!(self.label_offsets[label.0 as usize], UNKNOWN_LABEL_OFFSET);
+        debug_assert_eq!(self.label_aliases[label.0 as usize], UNKNOWN_LABEL);
+        self.label_offsets[label.0 as usize] = offset;
+    }
+
     /// Lazily clear `labels_at_tail` if the tail offset has moved beyond the
     /// offset that it applies to.
     fn lazily_clear_labels_at_tail(&mut self) {
@@ -2127,7 +2128,17 @@ impl MachBranch {
 pub struct MachTextSectionBuilder<I: VCodeInst> {
     buf: MachBuffer<I>,
     next_func: usize,
+    expected_funcs: usize,
     force_veneers: ForceVeneers,
+    /// A list of (new_label, label, offset_from_label) to bind once all
+    /// functions have been added.
+    ///
+    /// This list is pushed to in `resolve_reloc` below whenever the target of a
+    /// relocation is not a function itself but a relative location within a
+    /// function. When that happens `new_label` is created and binding it is
+    /// deferred until the end of the builder when all functions are known. This
+    /// is used to bind `new_label` to `offset(label) + offset_from_label`.
+    labels_to_bind: Vec<(MachLabel, MachLabel, i64)>,
 }
 
 impl<I: VCodeInst> MachTextSectionBuilder<I> {
@@ -2139,7 +2150,9 @@ impl<I: VCodeInst> MachTextSectionBuilder<I> {
         MachTextSectionBuilder {
             buf,
             next_func: 0,
+            expected_funcs: num_funcs,
             force_veneers: ForceVeneers::No,
+            labels_to_bind: Vec::new(),
         }
     }
 }
@@ -2151,7 +2164,7 @@ impl<I: VCodeInst> TextSectionBuilder for MachTextSectionBuilder<I> {
         func: &[u8],
         align: u32,
         ctrl_plane: &mut ControlPlane,
-    ) -> u64 {
+    ) -> (u64, Option<usize>) {
         // Conditionally emit an island if it's necessary to resolve jumps
         // between functions which are too far away.
         let size = func.len() as u32;
@@ -2162,15 +2175,18 @@ impl<I: VCodeInst> TextSectionBuilder for MachTextSectionBuilder<I> {
 
         self.buf.align_to(align);
         let pos = self.buf.cur_offset();
-        if labeled {
-            self.buf.bind_label(
-                MachLabel::from_block(BlockIndex::new(self.next_func)),
-                ctrl_plane,
-            );
+        let label = if labeled {
             self.next_func += 1;
+            Some(MachLabel::from_block(BlockIndex::new(self.next_func - 1)))
+        } else {
+            None
+        };
+
+        if let Some(label) = label {
+            self.buf.bind_label(label, ctrl_plane);
         }
         self.buf.put_data(func);
-        u64::from(pos)
+        (u64::from(pos), label.map(|l| l.0 as usize))
     }
 
     fn resolve_reloc(&mut self, offset: u64, reloc: Reloc, addend: Addend, target: usize) -> bool {
@@ -2180,10 +2196,21 @@ impl<I: VCodeInst> TextSectionBuilder for MachTextSectionBuilder<I> {
         let label = MachLabel::from_block(BlockIndex::new(target));
         let offset = u32::try_from(offset).unwrap();
         match I::LabelUse::from_reloc(reloc, addend) {
-            Some(label_use) => {
+            // If the offset from `label` is exactly 0, we can bind to the
+            // existing label.
+            Some((label_use, 0)) => {
                 self.buf.use_label_at_offset(offset, label, label_use);
                 true
             }
+
+            // If `addend` is nonzero, however, make a new label and defer its
+            // resolution while binding `label_use` to the new label.
+            Some((label_use, addend)) => {
+                let new_label = self.buf.get_label();
+                self.buf.use_label_at_offset(offset, new_label, label_use);
+                self.labels_to_bind.push((new_label, label, addend));
+                true
+            }
             None => false,
         }
     }
@@ -2198,7 +2225,16 @@ impl<I: VCodeInst> TextSectionBuilder for MachTextSectionBuilder<I> {
 
     fn finish(&mut self, ctrl_plane: &mut ControlPlane) -> Vec<u8> {
         // Double-check all functions were pushed.
-        assert_eq!(self.next_func, self.buf.label_offsets.len());
+        assert_eq!(self.next_func, self.expected_funcs);
+
+        // If any new labels were introduced relative to where functions ended
+        // up then bind those all here.
+        for (new_label, label, addend) in self.labels_to_bind.drain(..) {
+            let offset = self.buf.resolve_label_offset(label);
+            assert!(offset != UNKNOWN_LABEL_OFFSET);
+            let new_offset = u32::try_from(i64::from(offset) + addend).unwrap();
+            self.buf.bind_label_at_offset(new_label, new_offset);
+        }
 
         // Finish up any veneers, if necessary.
         self.buf

cranelift/codegen/src/machinst/mod.rs

@@ -260,9 +260,12 @@ pub trait MachInstLabelUse: Clone + Copy + Debug + Eq {
 
     /// Returns the corresponding label-use for the relocation specified.
     ///
-    /// This returns `None` if the relocation doesn't have a corresponding
-    /// representation for the target architecture.
-    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<Self>;
+    /// The `reloc` and `addend` specified are the kind of relocation and static
+    /// offset to the target. This function returns `None` if the relocation
+    /// isn't modeled by `LabelUse`. When this function returns `Some` it can
+    /// return a new `Addend` as some relocations modify the addend that they
+    /// are applied to.
+    fn from_reloc(reloc: Reloc, addend: Addend) -> Option<(Self, Addend)>;
 }
 
 /// Describes a block terminator (not call) in the VCode.
@@ -527,14 +530,15 @@ pub trait TextSectionBuilder {
     /// `resolve_reloc`.
     ///
     /// This function returns the offset at which the data was placed in the
-    /// text section.
+    /// text section along with the optional label that was created for this
+    /// function (suitable to pass to `resolve_reloc`).
     fn append(
         &mut self,
         labeled: bool,
         data: &[u8],
         align: u32,
         ctrl_plane: &mut ControlPlane,
-    ) -> u64;
+    ) -> (u64, Option<usize>);
 
     /// Attempts to resolve a relocation for this function.
     ///