-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Description
What would you like to happen?
Currently, many exception handlers throughout the Python SDK re-raise exceptions without using raise ... from, which loses the original traceback context. This makes debugging significantly harder because developers can't see the root cause when exceptions are wrapped.
Example Problem:
# Current code (loses context)
try:
connection = self._engine.connect()
result = connection.execute(text(query))
except Exception as e:
raise RuntimeError(f"Database operation failed: {e}")When this fails, you only see:
RuntimeError: Database operation failed: connection timeout
But you don't see the original ConnectionTimeoutError with its full traceback.
Desired Solution:
# With exception chaining (preserves context)
try:
connection = self._engine.connect()
result = connection.execute(text(query))
except Exception as e:
raise RuntimeError(f"Database operation failed: {e}") from eNow you see:
RuntimeError: Database operation failed: connection timeout
The above exception was the direct cause of the following exception:
ConnectionTimeoutError: Failed to connect within 30 seconds
[full original traceback here]
I would like to systematically add exception chaining throughout the codebase. This involves:
- Identifying all exception re-raises - Find all places where exceptions are caught and re-raised
- Adding
from eclause - Updateraise NewException(...)toraise NewException(...) from e - Enabling pylint warning - Enable the
raise-missing-fromwarning in.pylintrc(currently disabled with TODO) - Adding tests - Ensure exception chains are preserved in error scenarios
- Incremental approach - Start with high-traffic modules (transforms, runners, IO)
This enhancement will significantly improve debuggability by preserving the full exception context, making it easier to identify root causes of failures in production pipelines.
Note: This is already tracked in .pylintrc line 139:
raise-missing-from, #TODO(https://github.com/apache/beam/issues/21169) Enable and fix warningsIssue Priority
Priority: 3 (P3 - Code quality improvement)
Issue Components
- Component: Python SDK