Skip to content

Conversation

@dinana
Copy link
Contributor

@dinana dinana commented Oct 12, 2025

Problem

The execution reconciliation crashes with AttributeError: 'CancelledError' object has no attribute 'client_id' when generate_mass_status() is cancelled during startup. This prevents the trading node from starting when reconciliation is enabled.

Root Cause

In Python 3.8+, asyncio.CancelledError inherits from BaseException instead of Exception (PEP 3151). The exception check at line 1292 only catches Exception, so CancelledError passes through uncaught and crashes when the code attempts to access mass_status.client_id.

Exception hierarchy:

  • BaseException
    • asyncio.CancelledError ← NOT caught by isinstance(x, Exception)
    • Exception
      • RuntimeError, ValueError, etc.

Solution

Changed the isinstance check from Exception to BaseException to catch all exception types that can be returned by asyncio.gather(..., return_exceptions=True), including CancelledError.

Before:

if isinstance(mass_status_or_exception, Exception):

After:

if isinstance(mass_status_or_exception, BaseException):

Testing

Verified the fix works with:

  • Reconciliation enabled with 7 existing positions
  • StreamingConfig enabled (which previously triggered the crash)
  • All engines connected successfully
  • All actors reached RUNNING state
  • No CancelledError occurred

Impact

  • Severity: High - prevented node startup when reconciliation enabled
  • Affects: All adapters using async operations in generate_mass_status()
  • Fix: Future-proof - catches any exception type from asyncio.gather()

Fixes crash during execution reconciliation startup.

## Problem

The execution reconciliation crashes with `AttributeError: 'CancelledError' object has no attribute 'client_id'` when `generate_mass_status()` is cancelled during startup. This prevents the trading node from starting when reconciliation is enabled.

## Root Cause

In Python 3.8+, `asyncio.CancelledError` inherits from `BaseException` instead of `Exception` (PEP 3151). The exception check at line 1292 only catches `Exception`, so `CancelledError` passes through uncaught and crashes when the code attempts to access `mass_status.client_id`.

**Exception hierarchy:**
- `BaseException`
  - `asyncio.CancelledError` ← NOT caught by `isinstance(x, Exception)`
  - `Exception`
    - `RuntimeError`, `ValueError`, etc.

## Solution

Changed the isinstance check from `Exception` to `BaseException` to catch all exception types that can be returned by `asyncio.gather(..., return_exceptions=True)`, including `CancelledError`.

**Before:**
\`\`\`python
if isinstance(mass_status_or_exception, Exception):
\`\`\`

**After:**
\`\`\`python
if isinstance(mass_status_or_exception, BaseException):
\`\`\`

## Testing

Verified the fix works with:
- Reconciliation enabled with 7 existing positions
- StreamingConfig enabled (which previously triggered the crash)
- All engines connected successfully
- All actors reached RUNNING state
- No CancelledError occurred

## Impact

- **Severity:** High - prevented node startup when reconciliation enabled
- **Affects:** All adapters using async operations in `generate_mass_status()`
- **Fix:** Future-proof - catches any exception type from `asyncio.gather()`

Fixes crash during execution reconciliation startup.
@cjdsellers cjdsellers changed the title fix: Handle asyncio.CancelledError in execution reconciliation Fix handling of asyncio.CancelledError in execution reconciliation Oct 12, 2025
@cjdsellers
Copy link
Member

Thanks for the fix @dinana 🙏

@cjdsellers cjdsellers merged commit adb7ab2 into nautechsystems:develop Oct 12, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants