Description / Context
Airflow 3.x uses fastapi as its api-server backend, which uses uvicorn (>=0.30.0). Uvicorn switched parts of its internal architecture between 0.29 and 0.30 so that the multi-process manager uses the more resource-intensive spawning instead of forking for child processes.
See:
Problem
Using multiple workers with the default version of unicorn results in child processes being terminated as the spawn does not complete within the hard-coded timeout (5s). Possible workarounds are:
- Set
AIRFLOW__API__WORKERS=1, this prevents uvicorn launching child processes avoiding the timeout issue
- current implementation in our operator
- Downgrade to
uvicorn==0.29.0, which is the last version to use forking
- the current build references 0.34, so that is quite a downgrade
Proposal
Test that keeping the management of child processes at the api-server level (i.e. multiple api-server replicas, each with AIRFLOW__API__WORKERS=1, rather than AIRFLOW__API__WORKERS set to greater than 1 in a single api-server instance) is a feasible approach, and document in detail as such.
Description / Context
Airflow 3.x uses fastapi as its api-server backend, which uses uvicorn (>=0.30.0). Uvicorn switched parts of its internal architecture between 0.29 and 0.30 so that the multi-process manager uses the more resource-intensive spawning instead of forking for child processes.
See:
Problem
Using multiple workers with the default version of unicorn results in child processes being terminated as the spawn does not complete within the hard-coded timeout (5s). Possible workarounds are:
AIRFLOW__API__WORKERS=1, this prevents uvicorn launching child processes avoiding the timeout issueuvicorn==0.29.0, which is the last version to use forkingProposal
Test that keeping the management of child processes at the api-server level (i.e. multiple api-server replicas, each with
AIRFLOW__API__WORKERS=1, rather thanAIRFLOW__API__WORKERSset to greater than 1 in a single api-server instance) is a feasible approach, and document in detail as such.