-
Notifications
You must be signed in to change notification settings - Fork 3.9k
GH-47329: [Packaging][Python][C++] Build and test PyArrow against the pyodide_2025_0 ABI
#47330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
GH-47329: [Packaging][Python][C++] Build and test PyArrow against the pyodide_2025_0 ABI
#47330
Conversation
|
|
|
@github-actions crossbow submit test-conda-python-emscripten |
|
|
Hello, could the maintainers with the necessary permissions please trigger an Emscripten build on this PR? Thank you! I thought I could do this, as I previously submitted a PR that was accepted and merged: #45072, so I'm not sure if I understand the bot's message above. Is it that the word "contributors" here actually means those with a commit bit? 🤔 |
|
It looks like the actual error is something else, though, which gets masked by the error message: https://github.com/apache/arrow/actions/runs/16950058312/job/48040535515?pr=47330#step:5:16 |
|
@github-actions crossbow submit test-conda-python-emscripten |
|
Revision: 9d4bf5a Submitted crossbow builds: ursacomputing/crossbow @ actions-e6f3074ad3
|
|
Thanks. We'll need a |
|
Could you please submit another build? Thanks! |
|
@github-actions crossbow submit test-conda-python-emscripten |
|
Revision: 16f90d9 Submitted crossbow builds: ursacomputing/crossbow @ actions-c232acf2ff
|
|
I am trying to find out the correct place to install |
pyodide_2025_0 ABIpyodide_2025_0 ABI
I added this in af2e18d, but I'm not sure if this is the right approach. Could you please verify this commit, and also submit another build? Thanks! |
|
@github-actions crossbow submit test-conda-python-emscripten |
|
FYI: You can use See https://arrow.apache.org/docs/developers/continuous_integration/archery.html how to setup Archery. |
|
Revision: af2e18d Submitted crossbow builds: ursacomputing/crossbow @ actions-5d3d1c591a
|
I am not sure. I, too, think it is needed. Here is the relevant CI run: https://github.com/ursacomputing/crossbow/actions/runs/16956640331/job/48060086633#step:6:2545 |
|
Hmm, now the C++ build and Python compilation are succeeding, but we have a failure at link time. We have:
This indicates that we either intended to build |
|
Could you please submit another build? Thanks! |
|
@github-actions crossbow submit test-conda-python-emscripten |
|
Revision: 20de7ad Submitted crossbow builds: ursacomputing/crossbow @ actions-1b4dc9adab
|
|
Thanks! It's a bit strange that Node.js just became unresponsive in the CI run despite receiving an error. Nevertheless, the failure shows up as:
which subsequently seeps into the |
|
I'm unsure if 781b54a is the right approach, but it doesn't hurt to try – I suppose that if it works, it would work with |
|
@github-actions crossbow submit test-conda-python-emscripten |
|
Revision: f73c495 Submitted crossbow builds: ursacomputing/crossbow @ actions-f4846c031b
|
|
Yup, this isn't right. Sorry! I'll try to find out the cause of the |
|
I've isolated the problem to the Cython extension modules indeed; all of them fail to load (just that Node.js stops at the first one in the Crossbow runs): Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_acero.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_acero.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_csv.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_csv.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_orc.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_orc.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_parquet.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_parquet.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_feather.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_feather.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_fs.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_fs.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_json.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_json.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_orc.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_orc.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_parquet.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_parquet.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_pyarrow_cpp_tests.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_pyarrow_cpp_tests.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_substrait.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_substrait.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/lib.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/lib.cpython-313-wasm32-emscripten.so: 0
at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)and comparing with older built, working wheels: all of these shared objects should depend on |
...which is because I changed to building |
This reverts commit 20de7ad.
|
There is also the fact that the warning as follows: appears with Pyodide 0.28, but not with Pyodide 0.26. |
|
@github-actions crossbow submit test-conda-python-emscripten |
|
Revision: eae25be Submitted crossbow builds: ursacomputing/crossbow @ actions-49d30e1fce
|
Rationale for this change
This PR allows PyArrow to be built and tested against the new
pyodide_2025_0ABI introduced with the Pyodide 0.28 release.Towards #47329
What changes are included in this PR?
This PR updates the Pyodide version in the build scripts to 0.28.1, and the Emscripten version to 4.0.9.
Are these changes tested?
Are there any user-facing changes?
Yes, downstream packagers trying to build a WASM wheel for Pyodide are now requested to use newer Pyodide and Emscripten versions, and the latest stable version of pyodide-build available.
pyodide_2025_0ABI) #47329cc: @hoodmane @ryanking13