CABI: assert basic reentrancy rules the host must obey (clarification)

Author: lukewagner

design/mvp/Concurrency.md

@@ -16,6 +16,7 @@ emojis. For an even higher-level introduction, see [these][wasmio-2024]
   * [Thread Built-ins](#thread-built-ins)
   * [Thread-Local Storage](#thread-local-storage)
   * [Blocking](#blocking)
+  * [Reentrance](#reentrance)
   * [Waitables and Waitable Sets](#waitables-and-waitable-sets)
   * [Streams and Futures](#streams-and-futures)
   * [Stream Readiness](#stream-readiness)
@@ -313,6 +314,7 @@ of the new subtask created for the import call. Thus, one reason for
 associating every thread with a "containing task" is to ensure that there is
 always a well-defined async call stack.
 
+TODO: maybe move this to 'Reentrance' section
 A semantically-observable use of the async call stack is to distinguish between
 hazardous **recursive reentrance**, in which a component instance is reentered
 when one of its tasks is already on the callstack, from business-as-usual
@@ -503,6 +505,10 @@ once the reason for blocking is addressed.
 The [Canonical ABI explainer] defines the above behavior more precisely; search
 for `may_block` to see all the relevant points.
 
+### Reentrance
+
+TODO
+
 ### Waitables and Waitable Sets
 
 When an `async`-typed function is called with the async ABI and the call
@@ -1278,7 +1284,7 @@ comes after:
 * add an `async` effect on `resource` type definitions allowing a resource
   type to block during its destructor
 * `recursive` function type attribute: allow a function to opt in to
-  recursive [reentrance], extending the ABI to link the inner and
+  recursive [#reentrance], extending the ABI to link the inner and
   outer activations
 * add a `strict-callback` option that adds extra trapping conditions to
   provide the semantic guarantees needed for engines to statically avoid
@@ -1356,7 +1362,6 @@ comes after:
 [Binary Format]: Binary.md
 [WIT]: WIT.md
 [Blast Zone]: FutureFeatures.md#blast-zones
-[Reentrance]: Explainer.md#component-invariants
 [`start`]: Explainer.md#start-definitions
 
 [Store]: https://webassembly.github.io/spec/core/exec/runtime.html#syntax-store

design/mvp/Explainer.md

@@ -1735,10 +1735,10 @@ propagated once received.
 If `waitable-set.wait` is called from a synchronous- or `async callback`-lifted
 export, no other threads that were implicitly created by a separate
 synchronous- or `async callback`-lifted export call can start or progress in
-the current component instance until `waitable-set.wait` returns (thereby
-ensuring non-reentrance of the core wasm code). However, explicitly-created
-threads and threads implicitly created by non-`callback` `async`-lifted
-("stackful async") exports may start or progress at any time.
+the current component instance until `waitable-set.wait` returns (preserving
+[component invariant] #2). However, explicitly-created threads and threads
+implicitly created by non-`callback` `async`-lifted ("stackful async") exports
+may start or progress at any time.
 
 A `subtask` event notifies the supertask that its subtask is now in the given
 state (the meanings of which are described by the [concurrency explainer]).
@@ -2201,12 +2201,12 @@ thread to be resumed (as with `thread.yield-to`). If `cancellable` is set,
 `thread.yield` returns whether the current task was [cancelled] by the caller;
 otherwise, `thread.yield` always returns `false`.
 
-If `thread.yield` is called from a synchronous- or `async callback`-lifted
-export, it returns immediately without blocking (instead of trapping, as with
-other possibly-blocking operations like `waitable-set.wait`). This is because,
-unlike other built-ins, `thread.yield` may be scattered liberally throughout
-code that might show up in the transitive call tree of a synchronous function
-call.
+If `thread.yield` is called from a non-`async`-typed function that has not yet
+returned a value, it returns immediately without blocking (instead of trapping,
+as with other possibly-blocking operations like `waitable-set.wait`). This is
+because, unlike other built-ins, `thread.yield` may be scattered liberally
+throughout code that might show up in the transitive call tree of a synchronous
+function call.
 
 For details, see [Thread Built-ins] in the concurrency explainer and
 [`canon_thread_yield`] in the Canonical ABI explainer.
@@ -2908,8 +2908,8 @@ definition. Thus, component functions form a "membrane" around the collection
 of core module instances contained by a component instance, allowing the
 Component Model to establish invariants that increase optimizability and
 composability in ways not otherwise possible in the shared-everything setting
-of Core WebAssembly. The Component Model proposes establishing the following
-two runtime invariants:
+of Core WebAssembly. The Component Model establishes the following runtime
+invariants:
 1. Components define a "lockdown" state that prevents continued execution
    after a trap. This both prevents continued execution with corrupt state and
    also allows more-aggressive compiler optimizations (e.g., store reordering).
@@ -2919,13 +2919,16 @@ two runtime invariants:
    implicitly checked at every execution step by component functions. Thus,
    after a trap, it's no longer possible to observe the internal state of a
    component instance.
-2. The Component Model disallows reentrance by trapping if a callee's
-   component-instance is already on the stack when the call starts.
-   (For details, see [`call_might_be_recursive`](CanonicalABI.md#component-instance-state)
-   in the Canonical ABI explainer.) This default prevents obscure
-   composition-time bugs and also enables more-efficient non-reentrant
-   runtime glue code. This rule will be relaxed by an opt-in
-   function type attribute in the [future](Concurrency.md#todo).
+2. The Component Model's [reentrance] rules allow a producer toolchain to
+   implement both synchronous and `async` (🔀) exported functions using a
+   single, fixed-size shadow stack in linear memory that is pushed and popped in
+   LIFO order. (Internal use of cooperative threads (🧵) or the "stackful" ABI
+   option (🚟) may however require the use of multiple shadow stacks.)
+3. Until the Component Model provides a mechanism for guest code to reliably
+   avoid recursive deadlocks (in particular, due to async backpressure),
+   recursive [reentrance] rules trap in cases which might otherwise lead to
+   recursive deadlock. (This limitation is intended to be relaxed in the
+   [future](Concurrency.md#todo).)
 
 
 ## JavaScript Embedding
@@ -3327,6 +3330,7 @@ For some use-case-focused, worked examples, see:
 [Resolved]: Concurrency.md#cancellation
 [Cancellation]: Concurrency.md#cancellation
 [Cancelled]: Concurrency.md#cancellation
+[Reentrance]: Concurrency.md#reentrance
 
 [Component Model Documentation]: https://component-model.bytecodealliance.org
 [`wizer`]: https://github.com/bytecodealliance/wizer

design/mvp/canonical-abi/definitions.py

@@ -197,7 +197,7 @@ class ComponentInstance:
   num_waiting_to_enter: int
   exclusive: Optional[Thread]
 
-  def __init__(self, store, parent = None):
+  def __init__(self, store, parent):
     assert(parent is None or parent.store is store)
     self.store = store
     self.parent = parent
@@ -208,13 +208,11 @@ def __init__(self, store, parent = None):
     self.exclusive = None
     self.num_waiting_to_enter = 0
 
-  def reflexive_ancestors(self) -> set[ComponentInstance]:
-    s = set()
+  def root(self) -> RootComponentInstance:
     inst = self
-    while inst is not None:
-      s.add(inst)
+    while inst.parent is not None:
       inst = inst.parent
-    return s
+    return inst
 
   def is_ancestor_of(self, other):
     ancestor = other.parent
@@ -224,17 +222,26 @@ def is_ancestor_of(self, other):
       ancestor = ancestor.parent
     return False
 
-  def call_might_be_recursive(self, caller: Supertask):
-    if caller.inst is None:
-      while caller is not None:
-        if caller.inst and caller.inst.reflexive_ancestors() & self.reflexive_ancestors():
-          return True
-        caller = caller.supertask
-      return False
-    else:
-      return (caller.inst is self or
-              caller.inst.is_ancestor_of(self) or
-              self.is_ancestor_of(caller.inst))
+  def host_call_is_recursive(self, caller: Supertask):
+    assert(caller.inst is None or caller.inst.root() is not self.root())
+    while caller is not None:
+      if caller.inst is not None and caller.inst.root() is self.root():
+        return True
+      caller = caller.supertask
+    return False
+
+  def intra_component_call_might_be_recursive(self, caller: Supertask):
+    assert(caller.inst.root() is self.root())
+    return (caller.inst is self or
+            caller.inst.is_ancestor_of(self) or
+            self.is_ancestor_of(caller.inst))
+
+class RootComponentInstance(ComponentInstance):
+  may_enter: bool
+
+  def __init__(self, store):
+    ComponentInstance.__init__(self, store, parent = None)
+    self.may_enter = True
 
 ## Concurrency Primitives
 
@@ -546,15 +553,27 @@ def current_instance() -> ComponentInstance:
 
 class Store:
   waiting: list[Thread]
+  entered: int
 
   def __init__(self):
     self.waiting = []
+    self.entered = 0
 
   def lift(self, callee, opts: CanonicalOptions, ft: FuncType, inst: ComponentInstance) -> FuncInst:
     def func_inst(caller: Supertask, on_start, on_resolve) -> OnCancel:
-      trap_if(inst.call_might_be_recursive(caller))
+      self.entered += 1
+      intra_component = caller.inst is not None and caller.inst.root() is inst.root()
+      if intra_component:
+        trap_if(inst.intra_component_call_might_be_recursive(caller))
+      else:
+        trap_if(inst.host_call_is_recursive(caller))
+        trap_if(not inst.root().may_enter)
+        inst.root().may_enter = False
       task = Task(caller, inst, ft, opts, on_start, on_resolve)
       Thread(task, lambda: canon_lift(callee)).resume()
+      if not intra_component:
+        inst.root().may_enter = True
+      self.entered -= 1
       return task.request_cancellation
     return func_inst
 
@@ -565,10 +584,14 @@ def invoke(self, f: FuncInst, caller: Optional[Supertask], on_start, on_resolve)
     return f(host_caller, on_start, on_resolve)
 
   def tick(self):
+    assert(self.entered == 0)
     random.shuffle(self.waiting)
     for thread in self.waiting:
       if thread.ready():
+        assert(thread.task.inst.root().may_enter)
+        thread.task.inst.root().may_enter = False
         thread.resume()
+        thread.task.inst.root().may_enter = True
         return
 
 ## Lifting and Lowering Context