-
Notifications
You must be signed in to change notification settings - Fork 1.8k
pyemscripten/pyodide wheel builds #14787
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 27 commits
22f963e
f2d01c6
993c844
57c8ea3
fcf84c8
96cd750
be30149
9946a08
661e5f5
80168d0
ebbc395
969392d
094f18e
7f9a394
c8e69a6
5a8bea7
2d47164
b25e36e
fa544e6
a2169d1
a236786
5e8ca1d
5688b7e
411b2a1
bb4e8e6
783278d
089badf
dfb4d79
1af2b7c
041c097
fe6daeb
f3ff55f
ce45141
a79f81b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -366,3 +366,98 @@ jobs: | |
| with: | ||
| name: "cryptography-${{ github.event.inputs.version }}-${{ matrix.WINDOWS.WINDOWS }}-${{ matrix.PYTHON.VERSION }}-${{ matrix.PYTHON.ABI_VERSION }}" | ||
| path: wheelhouse\ | ||
|
|
||
| pyemscripten: | ||
| needs: [sdist] | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| PYODIDE: | ||
| - VERSION: "314.0.0a1" | ||
| EMSDK: "5.0.3" | ||
| PYTHON: "3.14" | ||
| CIBW_BUILD: "cp314-pyodide_wasm32" | ||
| CIBW_ENABLE: "pyodide-prerelease" | ||
| WHEEL_TAG: "cp314-pyemscripten_2026_0_wasm32" | ||
| name: "${{ matrix.PYODIDE.WHEEL_TAG }}" | ||
| env: | ||
| # Pyodide pins emscripten in xbuildenv/Makefile.envs. | ||
| PYODIDE_VERSION: ${{ matrix.PYODIDE.VERSION }} | ||
| EMSDK_VERSION: ${{ matrix.PYODIDE.EMSDK }} | ||
| # Must stay in lockstep with ci.yml::pyemscripten. | ||
| OPENSSL_VERSION: "3.6.2" | ||
| steps: | ||
| - name: Get OpenSSL build script from repository | ||
| uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | ||
| with: | ||
| ref: ${{ github.event.inputs.version || github.ref }} | ||
| persist-credentials: false | ||
| sparse-checkout: | | ||
| .github/bin/build_openssl.sh | ||
| sparse-checkout-cone-mode: false | ||
|
|
||
| - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 | ||
| with: | ||
| python-version: ${{ matrix.PYODIDE.PYTHON }} | ||
| timeout-minutes: 3 | ||
|
|
||
| - uses: dtolnay/rust-toolchain@e97e2d8cc328f1b50210efc529dca0028893a2d9 | ||
| with: | ||
| # Pyodide's emscripten build passes nightly-only `-Z` flags | ||
| # (link-native-libraries, emscripten-wasm-eh). | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It should be possible to build on stable soon, since neither of these flags are needed any longer. It seems like it might still be passing link-native-libraries=yes somewhere which is the default but it's still nightly only.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Wrote a TODO near this line and in ci.yml to make sure we validate if we still need nightly by the time pyodide 314.0 is released. Also updated the issue task list to make sure we don't forget about this. |
||
| # TODO: This should be switched to stable by the time pyodide 314.0 is released. | ||
| toolchain: nightly | ||
| target: wasm32-unknown-emscripten | ||
|
|
||
| - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 | ||
| with: | ||
| name: cryptography-sdist | ||
|
|
||
| # TODO: switch to released cibuildwheel once pypa/cibuildwheel#2812 | ||
| # (which adds cp314-pyodide_wasm32 / pyemscripten_2026_0_wasm32) ships | ||
| # in a tagged release. | ||
| - run: pip install "cibuildwheel @ git+https://github.com/pypa/cibuildwheel.git@main" | ||
| - run: mkdir wheelhouse | ||
|
|
||
| - name: Cache cross-compiled OpenSSL | ||
| # Same key shape as ci.yml::pyemscripten so the cache is shared. | ||
| uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 | ||
| timeout-minutes: 2 | ||
| with: | ||
| path: /tmp/openssl-pyodide | ||
| key: openssl-pyodide-${{ env.PYODIDE_VERSION }}-${{ env.OPENSSL_VERSION }}-emscripten-${{ env.EMSDK_VERSION }}-${{ hashFiles('.github/bin/build_openssl.sh') }}-${{ runner.os }}-${{ runner.arch }}-0 | ||
|
|
||
| - name: Build the wheel | ||
| # Pyodide does not expose a downstream-linkable OpenSSL (its `_ssl` | ||
| # module is baked into pyodide.asm.wasm), so we cross-compile our | ||
| # own static libssl/libcrypto via .github/bin/build_openssl.sh | ||
| # (TYPE=pyemscripten branch). | ||
| run: cibuildwheel --platform pyodide --output-dir wheelhouse/ cryptography*.tar.gz | ||
| env: | ||
| CIBW_BUILD: ${{ matrix.PYODIDE.CIBW_BUILD }} | ||
| CIBW_ENABLE: ${{ matrix.PYODIDE.CIBW_ENABLE }} | ||
| CIBW_BEFORE_BUILD_PYODIDE: | | ||
| TYPE=pyemscripten \ | ||
| VERSION="$OPENSSL_VERSION" \ | ||
| OSSL_PATH=/tmp/openssl-pyodide \ | ||
| bash ${{ github.workspace }}/.github/bin/build_openssl.sh | ||
| CIBW_ENVIRONMENT_PYODIDE: >- | ||
| OPENSSL_VERSION=$OPENSSL_VERSION | ||
| OPENSSL_DIR=/tmp/openssl-pyodide | ||
| OPENSSL_STATIC=1 | ||
| OPENSSL_NO_VENDOR=1 | ||
| # Smoketest only; the full pytest suite runs in ci.yml::pyemscripten. | ||
| CIBW_TEST_COMMAND_PYODIDE: | | ||
| python -c " | ||
| from cryptography.hazmat.backends.openssl.backend import backend | ||
| print(f'Loaded: {backend.openssl_version_text()}') | ||
| print(f'Linked Against: {backend._ffi.string(backend._lib.OPENSSL_VERSION_TEXT).decode(\"ascii\")}') | ||
| " | ||
|
|
||
| - run: | | ||
| echo "CRYPTOGRAPHY_WHEEL_NAME=$(basename $(ls wheelhouse/cryptography*.whl))" >> "$GITHUB_ENV" | ||
| - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 | ||
| with: | ||
| name: "${{ env.CRYPTOGRAPHY_WHEEL_NAME }}" | ||
| path: wheelhouse/ | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've taken a read through of this, and here is my concern: this is basically a significant amount of configuration, that none of hte pyca/cryptography knows anything about. We don't know when versions should be bumped, don't know what compatibility looks like, don't know how to debug. It duplicates a ton of our other CI code as well.
For this to be a realistic thing we were willing to maintain, we need to get this down much closer to "this is a new entry in the existing linux matrix that shares the vast majority of the code with the other CI workflows"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the read. Maintainability concerns are fair. Two paths I'd like your call on before cutting code:
Option A — Fold into the linux matrix
New row:
Pros: checkout/setup-python/rust-toolchain/cache/upload-coverage reuse linux's steps; YAML shrinks; pyemscripten appears as a row in the linux matrix.
Cons: linux job grows 4-5
if: matrix.PYTHON.PYODIDEbranches (install + Tests must diverge — nox can't drive a pyodide venv since uv has no wasm build).Option B — Standalone job, shrunk via shared building blocks
.github/actions/setup-pyodide-build/for thepyodide-build[resolve]+xbuildenv install+install-emscriptentrio.build_openssl.sh(TYPE=pyemscripten already supported)..github/bin/test_pyemscripten.shforpyodide build .→pyodide venv→ install wheel →pytest --cov. Same script runs locally.Pros: YAML shrinks; no conditional proliferation; unique logic in one action + one script, both grep-able and locally runnable.
Cons: still its own job rather than a row in the linux matrix.
I lean B — fewer conditionals, better local-debug story — but it's your call.
Independent of A vs B, on the maintainability gap: I'd add
docs/development/pyemscripten.rstcovering the Pyodide ↔ Emscripten ↔ Python compatibility triple (and where Pyodide pins it inMakefile.envs), bump cadence, and 4-5 failure-triage entries. I'll do this regardless of which option we go with.What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think one option could be to just use cibuildwheel in all workflow files, as we ensure in our Pyodide platform code that the right Emscripten toolchain is picked up, and we won't need to worry about the pyodide-build version, etc. All dependencies are pinned/constrained, too, making it more reliable and reducing the bespoke configuration involved. The only thing to pick up and update would be based on changes between cibuildwheel releases, i.e., which Pyodide ABIs are available to build and whether to enable or disable them as needed. That said, I'd recommend building only for the 2026 ABI, as the 314.0.0 stable release is coming soon (issue, milestone). The second alpha is out as of two days ago.