Skip to content

Mv core mods to runtime/, spawn/, discovery/ subpkgs#427

Merged
goodboy merged 14 commits intomainfrom
subsys_reorg
Apr 2, 2026
Merged

Mv core mods to runtime/, spawn/, discovery/ subpkgs#427
goodboy merged 14 commits intomainfrom
subsys_reorg

Conversation

@goodboy
Copy link
Copy Markdown
Owner

@goodboy goodboy commented Mar 25, 2026

Mv core mods to runtime/, spawn/, discovery/ subpkgs

Motivation

The top-level tractor/ package had grown to ~20 single-file modules
all sitting flat in one dir, making it increasingly hard to grok
which modules belong to which subsystem (actor runtime vs process
spawning vs service discovery). This branch reorganizes the internals
into three coherent subpackages — runtime/, spawn/, discovery/
— each with its own __init__.py re-exporting the public-facing
names so downstream code sees no breakage.

While in there, the Arbiter actor-type gets its proper name
(Registrar) and moves to the discovery/ subpkg where it
semantically belongs. A handful of quality-of-life improvements ride
along: a RuntimeVars struct for typed access to actor runtime
state, reg_err_types() for custom remote exception registration,
wrapt-based decorator preservation for tractor_test(), and
CPU-freq-scaling latency headroom for tests that were flaking under
auto-cpufreq.


Summary of changes

By chronological commit

  • (ef6e2807) Add get_runtime_vars() copy-accessor to
    ._state.

  • (a3b72866) Add reg_err_types() for custom remote exc
    type registration/lookup.

  • (af11e473) Proto a ._state.RuntimeVars struct.

  • (8cc2d3aa) Expose RuntimeVars + get_runtime_vars()
    from pkg __init__.

  • (99053194) Use wrapt for tractor_test() decorator
    to preserve fn sigs.

  • (0e9114c6) Move various mods to new subpkgs,

    • _runtime, _state, _supervise, _portal, _rpc into
      runtime/.
    • _spawn, _entry, _forkserver_override,
      _mp_fixup_main into spawn/.
    • _addr, _discovery, _multiaddr into discovery/.
  • (2730d2ba) Update all test and example imports for the
    new subpkg paths.

  • (df543456) Add more Bash allow-rules to claude
    local settings.

  • (92cce5a2) Rename Arbiter -> Registrar, mv impl
    from _runtime to discovery._registry with back-compat alias.

  • (af1ca890, 23a9c8c9) Add /run-tests
    claude skill for pytest suite runs with dev-workflow helpers.

  • (e2d71fb2) Filter __pycache__ from example
    discovery in test_docs_examples.

  • (03941f18) Simplify Arbiter compat alias to a
    plain assignment.

  • (8de010c9) (79b0833a) Add
    cpu_scaling_factor() latency-headroom helper to conftest
    and apply to flaky-timeout tests.

  • (6727c317) Fix tractor_test kwarg check and
    Windows start_method default.


Scopes changed

  • tractor.runtime

    • new subpkg housing _runtime, _state, _supervise, _portal,
      _rpc.
    • __init__.py re-exports Actor, Arbiter, async_main(), etc.
  • tractor.spawn

    • new subpkg housing _spawn, _entry,
      _forkserver_override, _mp_fixup_main.
    • __init__.py re-exports new_proc(), trio_proc(), etc.
  • tractor.discovery

    • new subpkg housing _addr, _discovery, _multiaddr.
    • __init__.py re-exports address + registry helpers.
  • tractor.discovery._registry

    • Registrar — renamed from Arbiter, extracted from
      _runtime.py (~250 lines).
  • tractor.__init__

    • alias Arbiter = Registrar for back-compat.
  • tractor.runtime._state

    • add RuntimeVars struct for typed runtime-vars access.
    • add get_runtime_vars() copy-accessor.
  • tractor._exceptions

    • add reg_err_types() public API for custom remote error type
      registration.
  • tractor._testing.pytest

    • .tractor_test() — use wrapt.decorator for sig preservation.
    • fix kwarg check and Windows start_method default.
  • tests.*

    • update all imports across test suite for new subpkg paths.
  • tests.test_docs_examples

    • filter __pycache__ dirs from example script discovery.
  • tests.conftest

    • add get_cpu_state() + cpu_scaling_factor() latency-headroom
      helpers.
    • apply to test_cancellation, test_docs_examples,
      test_legacy_one_way_streaming.
  • .claude.skills.run-tests

    • new /run-tests skill for test suite invocation.
  • .claude/settings.local.json

    • broader Bash allow-rules.

Future follow up

(Tracked for later in #432.)

The diff surfaces two items worth tracking:

  • test_clustering.py has a # TODO noting that
    open_actor_cluster() teardown hangs — this needs investigation
    and likely a fix in the cluster shutdown path.

  • discovery._registry carries an # XXX NOTE about Registrar
    registry values needing to be hashable and msgspec-serializable
    — the current approach works but may need revisiting when the
    Address types land from Add Address types to the built-in msgspec, fix non-registar root addr setup bug #410.


(this pr content was generated in some part by claude-code)

Copilot AI review requested due to automatic review settings March 25, 2026 21:10
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reorganizes tractor subsystems into clearer packages (runtime/, spawn/, discovery/) and updates internal/public imports accordingly, including terminology changes from “arbiter” → “registrar”.

Changes:

  • Move/alias the registry actor implementation into tractor.discovery._registry and rename root-registry concept to “registrar”.
  • Introduce tractor.runtime and tractor.spawn packages (plus discovery/) and rewrite imports across runtime, IPC, dev tooling, tests, and examples.
  • Add new helpers/utilities: multiprocessing __main__ fixup module; multiaddr parser; test-suite CPU scaling headroom + updates to tractor_test decorator.

Reviewed changes

Copilot reviewed 59 out of 61 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tractor/trionics/_mngrs.py Update runtime state import path.
tractor/to_asyncio.py Update runtime state import path.
tractor/spawn/_spawn.py Repoint imports to new runtime/ + discovery/ module locations.
tractor/spawn/_mp_fixup_main.py New multiprocessing __main__ fixup helpers.
tractor/spawn/_forkserver_override.py Update forkserver bootstrap import path.
tractor/spawn/_entry.py Update imports to new package layout.
tractor/spawn/init.py New spawn package initializer (no eager imports).
tractor/runtime/_supervise.py Update imports to new package layout (spawn/, discovery/, etc.).
tractor/runtime/_state.py Add RuntimeVars struct + get_runtime_vars; update exception imports.
tractor/runtime/_runtime.py Update imports to new package layout; registrar terminology; Arbiter compat alias.
tractor/runtime/_rpc.py Update imports to new package layout.
tractor/runtime/_portal.py Update imports to new package layout.
tractor/runtime/init.py New runtime package initializer (no eager imports).
tractor/msg/_ops.py Update runtime state import path.
tractor/log.py Update runtime state import path.
tractor/ipc/_uds.py Update runtime state + runtime actor type import paths.
tractor/ipc/_transport.py Update Address type import path; registrar wording in comment.
tractor/ipc/_server.py Update imports to new runtime/discovery locations; comment wording updates.
tractor/ipc/_chan.py Update address import path.
tractor/experimental/_pubsub.py Small typing/formatting adjustments.
tractor/discovery/_registry.py New Registrar actor (registry) module; keeps Arbiter alias.
tractor/discovery/_multiaddr.py New libp2p-style multiaddress parsing utilities.
tractor/discovery/_discovery.py Update imports to new runtime/discovery locations; registrar wording.
tractor/discovery/_addr.py Update imports to new runtime/ipc locations.
tractor/discovery/init.py New discovery package initializer (no eager imports).
tractor/devx/debug/_tty_lock.py Update imports to new runtime/discovery locations.
tractor/devx/debug/_trace.py Update imports to new runtime locations.
tractor/devx/debug/_sync.py Update runtime state import path.
tractor/devx/debug/_sigint.py Update runtime state/runtime actor import paths.
tractor/devx/debug/_repl.py Update runtime state import path.
tractor/devx/debug/_post_mortem.py Update imports to new runtime locations.
tractor/devx/_stackscope.py Update runtime/spawn import paths.
tractor/_testing/pytest.py Refactor tractor_test decorator to support optional args; update runtime/spawn/discovery references.
tractor/_testing/addr.py Update address import path.
tractor/_streaming.py Update runtime actor type import path.
tractor/_root.py Update imports to new runtime/spawn/discovery locations; root actor is now Registrar.
tractor/_exceptions.py Update runtime state import; add reg_err_types helper.
tractor/_context.py Update runtime state/portal/runtime actor import paths.
tractor/_child.py Update runtime actor + spawn entry import paths.
tractor/init.py Re-export from new module locations; export RuntimeVars/Registrar/Arbiter.
tests/test_spawning.py Update assertions + wording for registrar; adjust parameters.
tests/test_rpc.py Update root-actor assertion to is_registrar.
tests/test_root_runtime.py Update runtime state access paths.
tests/test_multi_program.py Update imports + test naming to registrar terminology.
tests/test_local.py Update docs/strings/assertions to registrar terminology + new portal path.
tests/test_legacy_one_way_streaming.py Add timeout override; add CPU scaling headroom + logging fixture usage.
tests/test_inter_peer_cancellation.py Use public tractor.debug_mode() instead of internal state access.
tests/test_infected_asyncio.py Update runtime state import path.
tests/test_docs_examples.py Ignore __pycache__; add CPU scaling headroom for example timeouts.
tests/test_discovery.py Update registrar terminology and is_registrar checks.
tests/test_context_stream_semantics.py Update runtime state/runtime actor import paths.
tests/test_cancellation.py Add CPU scaling headroom; minor typing annotation.
tests/msg/test_pldrx_limiting.py Use public tractor.debug_mode() instead of internal state access.
tests/ipc/test_multi_tpt.py Update runtime/discovery imports.
tests/ipc/test_each_tpt.py Update runtime/discovery imports; use runtime get_rt_dir.
tests/devx/test_tooling.py Update _shutdown_msg import path.
tests/conftest.py Add CPU scaling inspection helpers for latency headroom.
examples/service_discovery.py Update printed wording to registrar terminology.
examples/debugging/fast_error_in_root_after_spawn.py Update ActorNursery type import path.
.claude/skills/run-tests/SKILL.md New documented procedure for running test suite/subsets.
.claude/settings.local.json Expand allowed local commands for Claude tooling.
Comments suppressed due to low confidence (6)

tractor/runtime/_state.py:106

  • RuntimeVars.__setattr__ unconditionally calls breakpoint(), which will pause execution any time runtime vars are mutated (including in production) and will hang test runs/CI. Remove the breakpoint() (or gate it behind an explicit debug flag) and keep __setattr__ behavior side-effect free.
    tractor/runtime/_state.py:158
  • get_runtime_vars() is annotated to return dict, but when as_dict=False it returns a RuntimeVars instance. Since this is exported from tractor.__init__, update the return type to dict[str, Any] | RuntimeVars (and/or overloads) so callers and type checkers aren’t misled.
    tractor/runtime/_runtime.py:1894
  • The module-level import at the bottom (from ..discovery._registry import Registrar) introduces a circular import if a user imports tractor.discovery._registry directly (it imports runtime._runtime.Actor, which then imports discovery._registry before Registrar is defined). Consider removing this eager import and providing backward compat via a lazy __getattr__/local import, or by keeping the alias only in tractor.discovery._registry.
    tractor/runtime/_state.py:81
  • RuntimeVars field types/defaults don’t match the actual values being stored in _runtime_vars (e.g. _root_mailbox defaults to (None, None) but is annotated as tuple[str, str|int]). Constructing RuntimeVars(**_runtime_vars) will raise msgspec.ValidationError. Update the annotations to include None where applicable and align defaults accordingly.
    tractor/runtime/_state.py:90
  • RuntimeVars uses mutable literals ([]) as defaults for _root_addrs and _registry_addrs. With msgspec.Struct, these defaults can be shared across instances, leading to cross-test/process state leakage. Use field(default_factory=list) for list defaults.
    tractor/runtime/_state.py:117
  • RuntimeVars.update() iterates from_dict.items(), but msgspec.Struct instances don’t provide .items() like a dict. This will fail if callers pass another RuntimeVars (or any Struct) as the input. Consider accepting only Mapping (dict-like) or using msgspec.structs.asdict() / __struct_fields__ to iterate fields safely.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +111 to +122
# NOTE, ensure we inject any test-fn declared fixture names.
for kw in [
'reg_addr',
'loglevel',
'start_method',
'debug_mode',
'tpt_proto',
'timeout',
]:
if kw in inspect.signature(wrapped).parameters:
assert kwargs[kw]

Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixture injection check assert kwargs[kw] is incorrect for valid falsey values (e.g. debug_mode=False, reg_addr=None) and will also KeyError if pytest provided the arg positionally. Instead, check for key presence (or bind args/kwargs to the wrapped signature) and avoid asserting truthiness of fixture values.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Owner Author

@goodboy goodboy Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 response authored by claude-code

Valid catch. assert kwargs[kw] fails for falsey fixture values like debug_mode=False (raises AssertionError) and would KeyError if pytest provided the arg positionally. Changed to assert kw in kwargs which checks presence without requiring truthiness.

📎 fixed in 6727c317

Comment on lines +123 to +132
if (
(start_method := kwargs.get('start_method'))
is
None
):
if (
platform.system() == "Windows"
and
start_method != 'trio'
):
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows start-method validation is inverted: when start_method is unset (None), this branch will still raise because None != 'trio'. This makes @tractor_test unusable on Windows unless start_method is explicitly provided. Set the default start method to 'trio' on Windows (or only enforce the restriction when a non-None start_method is requested).

Suggested change
if (
(start_method := kwargs.get('start_method'))
is
None
):
if (
platform.system() == "Windows"
and
start_method != 'trio'
):
start_method = kwargs.get('start_method')
if platform.system() == "Windows":
if start_method is None:
# Default to the only supported start method on Windows.
kwargs['start_method'] = 'trio'
elif start_method != 'trio':

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Owner Author

@goodboy goodboy Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 response authored by claude-code

Valid catch. The walrus operator assigns start_method = None in the outer if, then the inner if tests None != 'trio' which is always True — making @tractor_test unusable on Windows without explicit start_method. Restructured to match the pre-wrapt logic: default to 'trio' when unset, only raise when explicitly set to a non-'trio' value.

📎 fixed in 6727c317

goodboy added a commit that referenced this pull request Mar 29, 2026
- Use `kw in kwargs` membership test instead of
  `kwargs[kw]` to avoid `KeyError` on missing params.
- Restructure Windows `start_method` logic to properly
  default to `'trio'` when unset; only raise on an
  explicit non-trio value.

Review: PR #427 (Copilot)
#427 (review)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
@goodboy goodboy requested a review from guilledk March 30, 2026 01:22
@goodboy goodboy changed the title Subsys reorg Mv core mods to runtime/, spawn/, discovery/ subpkgs Mar 31, 2026
goodboy added 14 commits April 2, 2026 17:59
Expose a copy of the current actor's `_runtime_vars` dict
via a public fn; TODO to convert to `RuntimeVars` struct.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Allow external app code to register custom exception types
on `._exceptions` so they can be re-raised on the receiver
side of an IPC dialog via `get_err_type()`.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
So we can start transition from runtime-vars `dict` to a typed struct
for better clarity and wire-ready monitoring potential, as well as
better traceability when .

Deats,
- add a new `RuntimeVars(Struct)` with all fields from `_runtime_vars`
  dict typed out
- include `__setattr__()` with `breakpoint()` for debugging
  any unexpected mutations.
- add `.update()` method for batch-updating compat with `dict`.
- keep old `_runtime_vars: dict` in place (we need to port a ton of
  stuff to adjust..).

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Export the new `RuntimeVars` struct and `get_runtime_vars()`
from `tractor.__init__` and improve the accessor to
optionally return the struct form.

Deats,
- add `RuntimeVars` and `get_runtime_vars` to
  `__init__.py` exports; alphabetize `_state` imports.
- move `get_runtime_vars()` up in `_state.py` to sit
  right below `_runtime_vars` dict definition.
- add `as_dict: bool = True` param so callers can get
  either the legacy `dict` or the new `RuntimeVars`
  struct.
- drop the old stub fn at bottom of `_state.py`.
- rm stale `from .msg.pretty_struct import Struct` comment.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Refactor the test-fn deco to use `wrapt.decorator`
instead of `functools.wraps` for better fn-sig
preservation and optional-args support via
`PartialCallableObjectProxy`.

Deats,
- add `timeout` and `hide_tb` deco params
- wrap test-fn body with `trio.fail_after(timeout)`
- consolidate per-fixture `if` checks into a loop
- add `iscoroutinefunction()` type-check on wrapped fn
- set `__tracebackhide__` at each wrapper level

Also,
- update imports for new subpkg paths:
  `tractor.spawn._spawn`, `tractor.discovery._addr`,
  `tractor.runtime._state`
  (see upcoming, likely large patch commit ;)

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Restructure the flat `tractor/` top-level private mods
into (more nested) subpackages:

- `runtime/`: `_runtime`, `_portal`, `_rpc`, `_state`,
  `_supervise`
- `spawn/`: `_spawn`, `_entry`, `_forkserver_override`,
  `_mp_fixup_main`
- `discovery/`: `_addr`, `_discovery`, `_multiaddr`

Each subpkg `__init__.py` is kept lazy (no eager
imports) to avoid circular import issues.

Also,
- update all intra-pkg imports across ~35 mods to use
  the new subpkg paths (e.g. `from .runtime._state`
  instead of `from ._state`)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Adjust all `tractor._state`, `tractor._addr`,
`tractor._supervise`, etc. refs in tests and examples
to use the new `runtime/`, `discovery/`, `spawn/` paths.

Also,
- use `tractor.debug_mode()` pub API instead of
  `tractor._state.debug_mode()` in a few test mods
- add explicit `timeout=20` to `test_respawn_consumer_task`
  `@tractor_test` deco call

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Move the `Arbiter` class out of `runtime._runtime` into its
logical home at `discovery._registry` as `Registrar(Actor)`.
This completes the long-standing terminology migration from
"arbiter" to "registrar/registry" throughout the codebase.

Deats,
- add new `discovery/_registry.py` mod with `Registrar`
  class + backward-compat `Arbiter = Registrar` alias.
- rename `Actor.is_arbiter` attr -> `.is_registrar`;
  old attr now a `@property` with `DeprecationWarning`.
- `_root.py` imports `Registrar` directly for
  root-actor instantiation.
- export `Registrar` + `Arbiter` from `tractor.__init__`.
- `_runtime.py` re-imports from `discovery._registry`
  for backward compat.

Also,
- update all test files to use `.is_registrar`
  (`test_local`, `test_rpc`, `test_spawning`,
  `test_discovery`, `test_multi_program`).
- update "arbiter" -> "registrar" in comments/docstrings
  across `_discovery.py`, `_server.py`, `_transport.py`,
  `_testing/pytest.py`, and examples.
- drop resolved TODOs from `_runtime.py` and `_root.py`.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Add `get_cpu_state()` helper to read CPU freq settings
from `/sys/devices/system/cpu/` and use it to compensate
the perf time-limit when `auto-cpufreq` (or similar)
scales down the max frequency.

Deats,
- read `*_pstate_max_freq` and `scaling_max_freq`
  to compute a `cpu_scaled` ratio.
- when `cpu_scaled != 1.`, increase `this_fast` limit
  proportionally (factoring dual-threaded cores).
- log a warning via `test_log` when compensating.

(this commit msg was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
Factor the CPU-freq-scaling helper out of
`test_legacy_one_way_streaming` into `conftest.py`
alongside a new `cpu_scaling_factor()` convenience fn
that returns a latency-headroom multiplier (>= 1.0).

Apply it to the two other flaky-timeout tests,
- `test_cancel_via_SIGINT_other_task`: 2s -> scaled
- `test_example[we_are_processes.py]`: 16s -> scaled

Deats,
- add `get_cpu_state()` + `cpu_scaling_factor()` to
  `conftest.py` so all test mods can share the logic.
- catch `IndexError` (empty glob) in addition to
  `FileNotFoundError`.
- rename `factor` var -> `headroom` at call sites for
  clarity on intent.

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
- Use `kw in kwargs` membership test instead of
  `kwargs[kw]` to avoid `KeyError` on missing params.
- Restructure Windows `start_method` logic to properly
  default to `'trio'` when unset; only raise on an
  explicit non-trio value.

Review: PR #427 (Copilot)
#427 (review)

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
The `open_actor_cluster()` teardown hangs
intermittently on UDS when `gather_contexts(mngrs=())`
raises `ValueError` mid-setup; likely a race in the
actor-nursery cleanup vs UDS socket shutdown. TCP
passes reliably (5/5 runs).

- Add `tpt_proto` fixture param to the test
- `pytest.skip()` on UDS with a TODO for deeper
  investigation of `._clustering`/`._supervise`
  teardown paths

(this patch was generated in some part by [`claude-code`][claude-code-gh])
[claude-code-gh]: https://github.com/anthropics/claude-code
@goodboy goodboy merged commit 8f6bc56 into main Apr 2, 2026
7 of 8 checks passed
@goodboy goodboy deleted the subsys_reorg branch April 2, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants