Skip to content

[CI] Support multi-Python version build (3.10/3.11/3.12) #7230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
EmmonsCurse wants to merge 6 commits into
base: develop
Choose a base branch
from
Open

[CI] Support multi-Python version build (3.10/3.11/3.12) #7230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4dfdeef
[CI] Support multi-Python version build (3.10/3.11/3.12)
EmmonsCurse Apr 7, 2026
ec4c9fa
remove cp310-cp310 and install jinja2
EmmonsCurse Apr 7, 2026
107dccc
[CI] fix
EmmonsCurse May 12, 2026
fdc7cf5
[CI] fix Invalid input error
EmmonsCurse May 12, 2026
79e2857
[CI] fix github-token error
EmmonsCurse May 12, 2026
7d80249
Merge branch 'develop' into add_multi_python
EmmonsCurse May 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 30 additions & 3 deletions .github/workflows/_build_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
description: "Build Images"
required: true
type: string
default: "ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddleqa:cuda126-py310"
default: "ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddleqa:fastdeploy-build-cuda126-manylinux"
FASTDEPLOY_ARCHIVE_URL:
description: "URL of the compressed FastDeploy code archive."
required: true
Expand Down Expand Up @@ -53,6 +53,11 @@ on:
required: false
type: string
default: ""
PYTHON_VERSION:
description: "Python version to use (3.10, 3.11, 3.12)"
required: false
type: string
default: "3.10"
outputs:
wheel_path:
description: "Output path of the generated wheel"
Expand Down Expand Up @@ -138,6 +143,7 @@ jobs:
PADDLE_WHL_URL: ${{ inputs.PADDLE_WHL_URL }}
WITH_NIGHTLY_BUILD: ${{ inputs.WITH_NIGHTLY_BUILD }}
FD_UNIFY_BUILD: ${{ inputs.FD_UNIFY_BUILD }}
PYTHON_VERSION: ${{ inputs.PYTHON_VERSION }}
run: |
set -x
runner_name="${{ runner.name }}"
Expand Down Expand Up @@ -173,8 +179,22 @@ jobs:
-e "PADDLE_WHL_URL=${PADDLE_WHL_URL}" \
-e "BRANCH_REF=${BRANCH_REF}" \
-e "CCACHE_MAXSIZE=50G" \
-e "PYTHON_VERSION=${PYTHON_VERSION}" \
-e "no_proxy=localhost,127.0.0.1,0.0.0.0,bcebos.com,.bcebos.com,bj.bcebos.com,su.bcebos.com,paddle-ci.gz.bcebos.com,apiin.im.baidu.com,baidu-int.com,.baidu.com,aliyun.com,gitee.com,pypi.tuna.tsinghua.edu.cn,.tuna.tsinghua.edu.cn" \
--gpus "\"device=${gpu_id}\"" ${docker_image} /bin/bash -c '
# Switch Python environment based on PYTHON_VERSION
if [[ "${PYTHON_VERSION}" == "3.11" ]]; then
Copy link
Copy Markdown

@PaddlePaddle-bot PaddlePaddle-bot May 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ 疑问 路径 /opt/_internal/cpython-3.11.0/ 硬编码了具体补丁版本号。若镜像实际安装的是 3.11.5 等更高补丁版本，export 会指向不存在的路径，PATH 前缀静默失效后回落到镜像默认 Python，导致实际版本与预期不符且无任何报错。建议切换后立即校验：python3 --version | grep -q "3.11" || { echo "ERROR: wrong python version"; exit 1; }。同样问题存在于 cu129/cu130 变体。

export LD_LIBRARY_PATH=/opt/_internal/cpython-3.11.0/lib/:${LD_LIBRARY_PATH}

This comment was marked as outdated.

Sign in to view

export PATH=/opt/_internal/cpython-3.11.0/bin/:${PATH}
elif [[ "${PYTHON_VERSION}" == "3.12" ]]; then
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.12.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.12.0/bin/:${PATH}
else
# Default to Python 3.10
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.10.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.10.0/bin/:${PATH}
fi

if [[ -n "${FD_VERSION}" ]]; then
export FASTDEPLOY_VERSION=${FD_VERSION}
echo "Custom FastDeploy version: ${FASTDEPLOY_VERSION}"
Expand Down Expand Up @@ -202,8 +222,15 @@ jobs:
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install wheel
# Install jinja2 for custom ops code generation (required for 3.11/3.12)
python -m pip install jinja2 wheel
# Skip cp310-specific wheels for non-3.10 Python versions

This comment was marked as outdated.

Sign in to view

if [[ "${PYTHON_VERSION}" != "3.10" ]]; then
grep -v "cp310-cp310" requirements.txt > requirements_filtered.txt
python -m pip install -r requirements_filtered.txt || true

This comment was marked as outdated.

Sign in to view

Copy link
Copy Markdown

@PaddlePaddle-bot PaddlePaddle-bot May 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 建议 || true 会吞掉 pip install 的失败退出码，依赖未安装时构建静默继续，最终以难以定位的 ImportError 告终。建议去掉 || true 让安装失败直接终止构建；如果部分包在 3.11/3.12 下确实不可用，应将它们明确加入 grep 过滤条件。同样问题存在于 _build_linux_cu129.yml 和 _build_linux_cu130.yml，需一并修复。

else
python -m pip install -r requirements.txt || true
fi
# 编译RDMA
export FD_ENABLE_RDMA_COMPILE=1
export FD_UNIFY_BUILD="${FD_UNIFY_BUILD}"
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/_build_linux_cu129.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ on:
required: false
type: string
default: ""
PYTHON_VERSION:
description: "Python version to use (3.10, 3.11, 3.12)"
required: false
type: string
default: "3.10"
outputs:
wheel_path_cu129:
description: "Output path of the generated wheel"
Expand Down Expand Up @@ -125,6 +130,7 @@ jobs:
PADDLE_WHL_URL: ${{ inputs.PADDLE_WHL_URL }}
WITH_NIGHTLY_BUILD: ${{ inputs.WITH_NIGHTLY_BUILD }}
FD_UNIFY_BUILD: ${{ inputs.FD_UNIFY_BUILD }}
PYTHON_VERSION: ${{ inputs.PYTHON_VERSION }}
run: |
set -x
runner_name="${{ runner.name }}"
Expand Down Expand Up @@ -160,8 +166,22 @@ jobs:
-e "PADDLE_WHL_URL=${PADDLE_WHL_URL}" \
-e "BRANCH_REF=${BRANCH_REF}" \
-e "CCACHE_MAXSIZE=50G" \
-e "PYTHON_VERSION=${PYTHON_VERSION}" \
-e "no_proxy=localhost,127.0.0.1,0.0.0.0,bcebos.com,.bcebos.com,bj.bcebos.com,su.bcebos.com,paddle-ci.gz.bcebos.com,apiin.im.baidu.com,baidu-int.com,.baidu.com,aliyun.com,gitee.com,pypi.tuna.tsinghua.edu.cn,.tuna.tsinghua.edu.cn" \
--gpus "\"device=${gpu_id}\"" ${docker_image} /bin/bash -c '
# Switch Python environment based on PYTHON_VERSION
if [[ "${PYTHON_VERSION}" == "3.11" ]]; then
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.11.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.11.0/bin/:${PATH}
elif [[ "${PYTHON_VERSION}" == "3.12" ]]; then
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.12.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.12.0/bin/:${PATH}
else
# Default to Python 3.10
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.10.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.10.0/bin/:${PATH}
fi

if [[ -n "${FD_VERSION}" ]]; then
export FASTDEPLOY_VERSION=${FD_VERSION}
echo "Custom FastDeploy version: ${FASTDEPLOY_VERSION}"
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/_build_linux_cu130.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ on:
required: false
type: string
default: ""
PYTHON_VERSION:
description: "Python version to use (3.10, 3.11, 3.12)"
required: false
type: string
default: "3.10"
outputs:
wheel_path_cu130:
description: "Output path of the generated wheel"
Expand Down Expand Up @@ -125,6 +130,7 @@ jobs:
PADDLE_WHL_URL: ${{ inputs.PADDLE_WHL_URL }}
WITH_NIGHTLY_BUILD: ${{ inputs.WITH_NIGHTLY_BUILD }}
FD_UNIFY_BUILD: ${{ inputs.FD_UNIFY_BUILD }}
PYTHON_VERSION: ${{ inputs.PYTHON_VERSION }}
run: |
set -x
runner_name="${{ runner.name }}"
Expand Down Expand Up @@ -160,8 +166,22 @@ jobs:
-e "PADDLE_WHL_URL=${PADDLE_WHL_URL}" \
-e "BRANCH_REF=${BRANCH_REF}" \
-e "CCACHE_MAXSIZE=50G" \
-e "PYTHON_VERSION=${PYTHON_VERSION}" \
-e "no_proxy=localhost,127.0.0.1,0.0.0.0,bcebos.com,.bcebos.com,bj.bcebos.com,su.bcebos.com,paddle-ci.gz.bcebos.com,apiin.im.baidu.com,baidu-int.com,.baidu.com,aliyun.com,gitee.com,pypi.tuna.tsinghua.edu.cn,.tuna.tsinghua.edu.cn" \
--gpus "\"device=${gpu_id}\"" ${docker_image} /bin/bash -c '
# Switch Python environment based on PYTHON_VERSION
if [[ "${PYTHON_VERSION}" == "3.11" ]]; then
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.11.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.11.0/bin/:${PATH}
elif [[ "${PYTHON_VERSION}" == "3.12" ]]; then
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.12.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.12.0/bin/:${PATH}
else
# Default to Python 3.10
export LD_LIBRARY_PATH=/opt/_internal/cpython-3.10.0/lib/:${LD_LIBRARY_PATH}
export PATH=/opt/_internal/cpython-3.10.0/bin/:${PATH}
fi

if [[ -n "${FD_VERSION}" ]]; then
export FASTDEPLOY_VERSION=${FD_VERSION}
echo "Custom FastDeploy version: ${FASTDEPLOY_VERSION}"
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/ci_hpu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
name: CI_HPU

on:
pull_request:
branches:
- develop
- 'release/*'
# pull_request:
# branches:
# - develop
# - 'release/*'
workflow_dispatch:

concurrency:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/ci_iluvatar.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
name: ILUVATAR-CI
on:
pull_request:
types: [opened, synchronize]
branches: [develop, release/**]
# pull_request:
# types: [opened, synchronize]
# branches: [develop, release/**]
permissions: read-all

concurrency:
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/ci_metax.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
name: CI_METAX

on:
pull_request_target:
types:
- opened
- synchronize
branches:
- develop
- release/**
#on:
Copy link
Copy Markdown

@PaddlePaddle-bot PaddlePaddle-bot May 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ 疑问 on: 关键字本身被注释掉（#on:），而 ci_hpu/iluvatar/xpu 的做法是保留 on: 关键字只注释 pull_request 子项并保留 workflow_dispatch。ci_metax.yml 目前无任何 on: 顶层键，GitHub Actions 将其视为无效工作流，既无法自动触发也无法手动触发。如果只是临时禁用 PR 触发，建议改为：on: workflow_dispatch:

# pull_request_target:
# types:
# - opened
# - synchronize
# branches:
# - develop
# - release/**

permissions:
contents: read
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/ci_xpu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
name: CI_XPU

on:
pull_request:
types: [opened, synchronize]
branches: [develop, release/**]
# pull_request:
# types: [opened, synchronize]
# branches: [develop, release/**]
permissions: read-all

concurrency:
Expand Down
Loading