Skip to content

Enhancement: Add exception chaining (raise ... from) throughout codebase to preserve error context #37422

@shaheeramjad

Description

@shaheeramjad

What would you like to happen?

Currently, many exception handlers throughout the Python SDK re-raise exceptions without using raise ... from, which loses the original traceback context. This makes debugging significantly harder because developers can't see the root cause when exceptions are wrapped.

Example Problem:

# Current code (loses context)
try:
    connection = self._engine.connect()
    result = connection.execute(text(query))
except Exception as e:
    raise RuntimeError(f"Database operation failed: {e}")

When this fails, you only see:

RuntimeError: Database operation failed: connection timeout

But you don't see the original ConnectionTimeoutError with its full traceback.

Desired Solution:

# With exception chaining (preserves context)
try:
    connection = self._engine.connect()
    result = connection.execute(text(query))
except Exception as e:
    raise RuntimeError(f"Database operation failed: {e}") from e

Now you see:

RuntimeError: Database operation failed: connection timeout
The above exception was the direct cause of the following exception:
ConnectionTimeoutError: Failed to connect within 30 seconds
  [full original traceback here]

I would like to systematically add exception chaining throughout the codebase. This involves:

  1. Identifying all exception re-raises - Find all places where exceptions are caught and re-raised
  2. Adding from e clause - Update raise NewException(...) to raise NewException(...) from e
  3. Enabling pylint warning - Enable the raise-missing-from warning in .pylintrc (currently disabled with TODO)
  4. Adding tests - Ensure exception chains are preserved in error scenarios
  5. Incremental approach - Start with high-traffic modules (transforms, runners, IO)

This enhancement will significantly improve debuggability by preserving the full exception context, making it easier to identify root causes of failures in production pipelines.

Note: This is already tracked in .pylintrc line 139:

raise-missing-from, #TODO(https://github.com/apache/beam/issues/21169) Enable and fix warnings

Issue Priority

Priority: 3 (P3 - Code quality improvement)

Issue Components

  • Component: Python SDK

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions