Fix Connection or Variable access in Server context #56602
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes apache#56120 Fixes apache#56583
kaxil
force-pushed
the
fix-conn-var-retrieval
branch
3 times, most recently
from
8d21aaf to
0cfdaa5
Compare
kaxil
requested review from
XD-DENG,
amoghrajesh,
ashb,
bugraoz93,
dstandish,
ephraimbuddy,
jason810496,
pierrejeambrun,
potiuk,
rawwar and
shubhamraj-git
as code owners
2 tasks
kaxil
force-pushed
the
fix-conn-var-retrieval
branch
from
0cfdaa5 to
ade61f4
Compare
ashb
reviewed
Oct 14, 2025
Member
ashb
left a comment
ashb
left a comment
There was a problem hiding this comment.
Messier then i'd like but i can't think of another option right now
ashb
approved these changes
Oct 14, 2025
Member
Member
Author
Thanks @gopidesupavan , Please do let us know what you find! |
Member
just in time: works fine with this changes :) 
|
kaxil
added a commit
that referenced
this pull request
Oct 14, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes #56120 Fixes #56583 (cherry picked from commit ae0a330)
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 15, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes apache#56120 Fixes apache#56583
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 17, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes apache#56120 Fixes apache#56583
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 19, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes apache#56120 Fixes apache#56583
2 tasks
TyrellHaywood
pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes apache#56120 Fixes apache#56583
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568 Fixes apache#57145
kaxil
added a commit
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568 Fixes #57145
kaxil
added a commit
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568 Fixes #57145 (cherry picked from commit da32b68)
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with
ImportErrorforSUPERVISOR_COMMSbecause it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval.Implemented automatic context detection using separate secrets backend chains for client and server processes:
SUPERVISOR_COMMSpresence and useExecutionAPISecretsBackendto route through the Execution APISUPERVISOR_COMMSis unavailable and useMetastoreBackendfor direct database accessFixes #56120
Fixes #56583
PS: I do not love the way we deal with secrets backends in airflow/configuration, even before this PR -- but not going to handle it in this PR.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.