|
1 | 1 | # Manylinux Build Strategy |
2 | 2 |
|
3 | | -## 问题概述 |
4 | | - |
5 | | -Colab和许多Linux环境需要`manylinux`兼容的wheels，而不是特定于某个Linux发行版的wheels。我们之前的尝试过于复杂，试图手动处理所有的依赖关系。 |
6 | | - |
7 | | -## 新策略 |
8 | | - |
9 | | -基于社区最佳实践和其他成功项目的经验，我们采用了以下简化方案： |
10 | | - |
11 | | -### 1. 使用 cibuildwheel |
12 | | - |
13 | | -`cibuildwheel`是PyPA推荐的构建多平台wheels的工具，它： |
14 | | -- 自动处理manylinux环境设置 |
15 | | -- 管理不同Python版本的构建 |
16 | | -- 提供标准化的测试框架 |
17 | | - |
18 | | -### 2. 使用 manylinux_2_28 |
19 | | - |
20 | | -- 从`manylinux2014`升级到`manylinux_2_28` |
21 | | -- 提供更现代的基础库 |
22 | | -- 使用`dnf`而不是`yum` |
23 | | -- 更好地支持现代Python版本 |
24 | | - |
25 | | -### 3. CMake配置 |
26 | | - |
27 | | -在`pyproject.toml`中添加： |
28 | | -```toml |
29 | | -[tool.scikit-build.cmake.define] |
30 | | -Python_FIND_VIRTUALENV = "ONLY" |
31 | | -Python3_FIND_VIRTUALENV = "ONLY" |
32 | | -``` |
33 | | - |
34 | | -这帮助CMake在cibuildwheel的虚拟环境中正确找到Python。 |
35 | | - |
36 | | -### 4. 简化的依赖安装 |
37 | | - |
38 | | -只安装必要的系统依赖： |
39 | | -- `gcc-c++` |
40 | | -- `boost-devel` |
41 | | -- `zeromq-devel` |
42 | | -- `openblas-devel` |
43 | | -- `cmake` |
44 | | - |
45 | | -## 测试策略 |
46 | | - |
47 | | -1. **手动触发**: 使用`workflow_dispatch`在GitHub Actions上测试 |
48 | | -2. **自动测试**: 在`fix/manylinux-*`分支上推送时自动运行 |
49 | | -3. **PR测试**: 当PR修改相关文件时自动测试 |
50 | | - |
51 | | -## 如何使用 |
52 | | - |
53 | | -### 在本地测试 |
54 | | -```bash |
55 | | -# 安装cibuildwheel |
56 | | -pip install cibuildwheel |
57 | | - |
58 | | -# 构建wheels |
59 | | -cibuildwheel --platform linux packages/leann-backend-hnsw |
60 | | -``` |
61 | | - |
62 | | -### 在GitHub Actions测试 |
63 | | -1. 推送到`fix/manylinux-compatibility`分支 |
64 | | -2. 或手动触发"Test Manylinux Build"工作流 |
65 | | - |
66 | | -## 下一步 |
67 | | - |
68 | | -1. **监控CI运行结果** |
69 | | -2. **根据错误调整依赖** |
70 | | -3. **测试生成的wheels在Colab的兼容性** |
71 | | -4. **如果成功，将更改合并到main** |
72 | | - |
73 | | -## 参考 |
74 | | - |
75 | | -- [cibuildwheel文档](https://cibuildwheel.readthedocs.io/) |
76 | | -- [manylinux规范](https://github.com/pypa/manylinux) |
77 | | -- [scikit-build-core文档](https://scikit-build-core.readthedocs.io/) |
| 3 | +## Problem |
| 4 | +Google Colab requires wheels compatible with `manylinux_2_35_x86_64` or earlier. Our previous builds were producing `manylinux_2_39_x86_64` wheels, which are incompatible. |
| 5 | + |
| 6 | +## Solution |
| 7 | +We're using `cibuildwheel` with `manylinux2014` images to build wheels that are compatible with a wide range of Linux distributions, including Google Colab. |
| 8 | + |
| 9 | +### Key Changes |
| 10 | + |
| 11 | +1. **cibuildwheel Configuration** |
| 12 | + - Using `manylinux2014` images (provides `manylinux_2_17` compatibility) |
| 13 | + - Using `yum` package manager (CentOS 7 based) |
| 14 | + - Installing `cmake3` and creating symlink for compatibility |
| 15 | + |
| 16 | +2. **Build Matrix** |
| 17 | + - Python versions: 3.9, 3.10, 3.11, 3.12, 3.13 |
| 18 | + - Platforms: Linux (x86_64), macOS |
| 19 | + - No Windows support (not required) |
| 20 | + |
| 21 | +3. **Dependencies** |
| 22 | + - Linux: gcc-c++, boost-devel, zeromq-devel, openblas-devel, cmake3 |
| 23 | + - macOS: boost, zeromq, openblas, cmake (via Homebrew) |
| 24 | + |
| 25 | +4. **Environment Variables** |
| 26 | + - `CMAKE_BUILD_PARALLEL_LEVEL=8`: Speed up builds |
| 27 | + - `Python_FIND_VIRTUALENV=ONLY`: Help CMake find Python in cibuildwheel env |
| 28 | + - `Python3_FIND_VIRTUALENV=ONLY`: Alternative variable for compatibility |
| 29 | + |
| 30 | +## Testing Strategy |
| 31 | + |
| 32 | +1. **CI Pipeline**: `test-manylinux.yml` |
| 33 | + - Triggers on PR to main, manual dispatch, or push to `fix/manylinux-*` branches |
| 34 | + - Builds wheels using cibuildwheel |
| 35 | + - Tests installation on Ubuntu 22.04 (simulating Colab) |
| 36 | + |
| 37 | +2. **Local Testing** |
| 38 | + ```bash |
| 39 | + # Download built wheels |
| 40 | + # Test in fresh environment |
| 41 | + python -m venv test_env |
| 42 | + source test_env/bin/activate |
| 43 | + pip install leann_core-*.whl leann_backend_hnsw-*manylinux*.whl leann-*.whl |
| 44 | + python -c "from leann import LeannBuilder; print('Success!')" |
| 45 | + ``` |
| 46 | + |
| 47 | +## References |
| 48 | +- [cibuildwheel documentation](https://cibuildwheel.readthedocs.io/) |
| 49 | +- [manylinux standards](https://github.com/pypa/manylinux) |
| 50 | +- [PEP 599 - manylinux2014](https://peps.python.org/pep-0599/) |
0 commit comments