Add Python 3.12 support (#36755)

Finally after a number of dependency upgrades we seem to be able to
upgrade to Python 3.12 (pending universal_pathlib 0.2.0 conversion)

Several providers are excluded from being installed and wait for
Python 3.12, but it should not block Airlfow's general 3.12 support.

Co-authored-by: dirrao <[email protected]>

Author: potiuk

.github/workflows/ci.yml

@@ -2298,13 +2298,15 @@ jobs:
       - name: >
           Build CI ARM images ${{ needs.build-info.outputs.image-tag }}
           ${{needs.build-info.outputs.all-python-versions-list-as-string}}:${{env.IMAGE_TAG}}
-        run: >
-          breeze ci-image build --run-in-parallel --builder airflow_cache --platform "linux/arm64"
+        run: |
+          # Do not run parallel builds here as they often fail due to github token expiry issues similar to
+          # those described in https://github.com/moby/buildkit/issues/2367
+          for python in ${{needs.build-info.outputs.python-versions-list-as-string}}; do
+            breeze ci-image build --builder airflow_cache --platform "linux/arm64" --python ${python}
+          done
         env:
           UPGRADE_TO_NEWER_DEPENDENCIES: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }}
           DOCKER_CACHE: ${{ needs.build-info.outputs.cache-directive }}
-          PYTHON_VERSIONS: ${{needs.build-info.outputs.all-python-versions-list-as-string}}
-          DEBUG_RESOURCES: ${{needs.build-info.outputs.debug-resources}}
           COMMIT_SHA: ${{ github.sha }}
       - name: "Stop ARM instance"
         run: ./scripts/ci/images/ci_stop_arm_instance.sh

Dockerfile

@@ -1528,7 +1528,7 @@ ARG USE_CONSTRAINTS_FOR_CONTEXT_PACKAGES="false"
 
 # By changing the epoch we can force reinstalling Airflow and pip all dependencies
 # It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH environment variable.
-ARG AIRFLOW_CI_BUILD_EPOCH="10"
+ARG AIRFLOW_CI_BUILD_EPOCH="11"
 ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
 
 

Dockerfile.ci

@@ -1108,7 +1108,7 @@ ARG PYTHON_BASE_IMAGE
 ARG AIRFLOW_IMAGE_REPOSITORY="https://github.com/apache/airflow"
 
 # By increasing this number we can do force build of all dependencies
-ARG DEPENDENCIES_EPOCH_NUMBER="10"
+ARG DEPENDENCIES_EPOCH_NUMBER="11"
 
 # Make sure noninteractive debian install is used and language variables set
 ENV PYTHON_BASE_IMAGE=${PYTHON_BASE_IMAGE} \

README.md

@@ -100,7 +100,7 @@ Apache Airflow is tested with:
 
 |             | Main version (dev)           | Stable version (2.8.2) |
 |-------------|------------------------------|------------------------|
-| Python      | 3.8, 3.9, 3.10, 3.11         | 3.8, 3.9, 3.10, 3.11   |
+| Python      | 3.8, 3.9, 3.10, 3.11, 3.12   | 3.8, 3.9, 3.10, 3.11   |
 | Platform    | AMD64/ARM64(\*)              | AMD64/ARM64(\*)        |
 | Kubernetes  | 1.25, 1.26, 1.27, 1.28, 1.29 | 1.25, 1.26, 1.27, 1.28 |
 | PostgreSQL  | 12, 13, 14, 15, 16           | 12, 13, 14, 15, 16     |

airflow/__init__.py

@@ -39,7 +39,12 @@
 # configuration is therefore initted early here, simply by importing it.
 from airflow import configuration, settings
 
-__all__ = ["__version__", "DAG", "Dataset", "XComArg"]
+__all__ = [
+    "__version__",
+    "DAG",
+    "Dataset",
+    "XComArg",
+]
 
 # Make `airflow` a namespace package, supporting installing
 # airflow.providers.* in different locations (i.e. one in site, and one in user

airflow/providers/apache/cassandra/hooks/cassandra.py

@@ -19,10 +19,22 @@
 from __future__ import annotations
 
 import re
+import sys
 from typing import Any, Union
 
+import cassandra
 from cassandra.auth import PlainTextAuthProvider
-from cassandra.cluster import Cluster, Session
+
+from airflow.exceptions import AirflowOptionalProviderFeatureException
+
+try:
+    from cassandra.cluster import Cluster, Session
+except cassandra.DependencyException:
+    if sys.version_info >= (3, 12):
+        raise AirflowOptionalProviderFeatureException(
+            "You need to have cassandra driver compiled with libev to get it working."
+        )
+    raise
 from cassandra.policies import (
     DCAwareRoundRobinPolicy,
     RoundRobinPolicy,

airflow/providers/apache/cassandra/provider.yaml

@@ -46,6 +46,26 @@ dependencies:
   - apache-airflow>=2.6.0
   - cassandra-driver>=3.13.0
 
+# Cassandra provider is not yet compatible with Python 3.12
+# The main issue is that python cassandra driver by default uses asyncore which has been deprecated since
+# Python 3.6 and removed in Python 3.12 (https://docs.python.org/3.11/library/asyncore.html)
+#
+# Currently the "wheel" package 3.29.0 distributed for manylinux platform is build without libev support (
+# which could be a viable asyncore replacement), and cassandra driver works in Python 3.12 if it is built
+# with libev support (using sdist, having gcc, libev4 and libev-dev installed). But it would be too much to
+# expect our users to build the driver from sources to use it with Python 3.12, so we are waiting for the
+# next release of cassandra-driver which will have libev support in the wheel package.
+# The issue is tracked here https://datastax-oss.atlassian.net/browse/PYTHON-1378
+#
+# Another option to get cassandra drive back is to have asyncio support in the driver instead of asyncore:
+# The issue is tracked here: https://datastax-oss.atlassian.net/browse/PYTHON-1375 and is scheduled
+# to be fixed in cassandra-driver 3.30.0.
+#
+# All Cassandra tests are automatically skipped if cassandra package is not present, so once you remove the
+# exclusion, they will start running for Python 3.12.
+#
+excluded-python-versions: ['3.12']
+
 integrations:
   - integration-name: Apache Cassandra
     external-doc-url: https://cassandra.apache.org/

contributing-docs/07_local_virtualenv.rst

@@ -37,7 +37,7 @@ Required Software Packages
 Use system-level package managers like yum, apt-get for Linux, or
 Homebrew for macOS to install required software packages:
 
-* Python (One of: 3.8, 3.9, 3.10, 3.11)
+* Python (One of: 3.8, 3.9, 3.10, 3.11, 3.12)
 * MySQL 5.7+
 * libxml
 * helm (only for helm chart tests)
@@ -186,6 +186,8 @@ This is what it shows currently:
 +-------------+---------+----------+---------------------------------------------------------------+
 | airflow-311 | virtual | devel    | Environment with Python 3.11                                  |
 +-------------+---------+----------+---------------------------------------------------------------+
+| airflow-312 | virtual | devel    | Environment with Python 3.12                                  |
++-------------+---------+----------+---------------------------------------------------------------+
 
 The default env (if you have not used one explicitly) is ``default`` and it is a Python 3.8
 virtualenv for maximum compatibility with ``devel`` extra installed - this devel extra contains the minimum set

dev/README_RELEASE_AIRFLOW.md

@@ -822,7 +822,7 @@ the older branches, you should set the "skip" field to true.
 ## Verify production images
 
 ```shell script
-for PYTHON in 3.8 3.9 3.10 3.11
+for PYTHON in 3.8 3.9 3.10 3.11 3.12
 do
     docker pull apache/airflow:${VERSION}-python${PYTHON}
     breeze prod-image verify --image-name apache/airflow:${VERSION}-python${PYTHON}

dev/breeze/README.md

@@ -66,6 +66,6 @@ PLEASE DO NOT MODIFY THE HASH BELOW! IT IS AUTOMATICALLY UPDATED BY PRE-COMMIT.
 
 ---------------------------------------------------------------------------------------------------------
 
-Package config hash: e2db123fd25e40b515520fb9f6ed32a601fe85e6a22b9845343bc5c81e5785b472527ed3bf6d07521c512d2222f99877a1ce1911ac504a3a6af7b7866aa674cc
+Package config hash: 0c6255210f3c20a29aa24405412399cf3f6ff897658f13a0fb7628b4888d6bfe99f49bdb2aa68d9045e14179d4be33a02543c12e394395931024522431bf5dec
 
 ---------------------------------------------------------------------------------------------------------