Skip to content

Conversation

@agriyakhetarpal
Copy link
Contributor

@agriyakhetarpal agriyakhetarpal commented Aug 13, 2025

Rationale for this change

This PR allows PyArrow to be built and tested against the new pyodide_2025_0 ABI introduced with the Pyodide 0.28 release.

Towards #47329

What changes are included in this PR?

This PR updates the Pyodide version in the build scripts to 0.28.1, and the Emscripten version to 4.0.9.

Are these changes tested?

Are there any user-facing changes?

Yes, downstream packagers trying to build a WASM wheel for Pyodide are now requested to use newer Pyodide and Emscripten versions, and the latest stable version of pyodide-build available.

cc: @hoodmane @ryanking13

@github-actions
Copy link

⚠️ GitHub issue #47329 has been automatically assigned in GitHub to PR creator.

@agriyakhetarpal
Copy link
Contributor Author

@github-actions crossbow submit test-conda-python-emscripten

@github-actions
Copy link

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/16950058312

@agriyakhetarpal
Copy link
Contributor Author

Hello, could the maintainers with the necessary permissions please trigger an Emscripten build on this PR? Thank you!

I thought I could do this, as I previously submitted a PR that was accepted and merged: #45072, so I'm not sure if I understand the bot's message above. Is it that the word "contributors" here actually means those with a commit bit? 🤔

@agriyakhetarpal
Copy link
Contributor Author

It looks like the actual error is something else, though, which gets masked by the error message: https://github.com/apache/arrow/actions/runs/16950058312/job/48040535515?pr=47330#step:5:16

fatal: not a git repository (or any of the parent directories): .git

@kou
Copy link
Member

kou commented Aug 14, 2025

@github-actions crossbow submit test-conda-python-emscripten

@github-actions
Copy link

Revision: 9d4bf5a

Submitted crossbow builds: ursacomputing/crossbow @ actions-e6f3074ad3

Task Status
test-conda-python-emscripten GitHub Actions

@agriyakhetarpal
Copy link
Contributor Author

agriyakhetarpal commented Aug 14, 2025

Thanks. We'll need a docker.io/apache/arrow-dev:amd64-conda-python-3.13 image for this, as the required Python version for Pyodide 0.28 builds is 3.13.

@agriyakhetarpal
Copy link
Contributor Author

Could you please submit another build? Thanks!

@kou
Copy link
Member

kou commented Aug 14, 2025

@github-actions crossbow submit test-conda-python-emscripten

@github-actions
Copy link

Revision: 16f90d9

Submitted crossbow builds: ursacomputing/crossbow @ actions-c232acf2ff

Task Status
test-conda-python-emscripten GitHub Actions

@agriyakhetarpal
Copy link
Contributor Author

I am trying to find out the correct place to install libatomic1; it's required by the Brotli and Protocol Buffers Emscripten builds. In the meantime; I changed -fexceptions to -fwasm-exceptions in eebb7a8.

@agriyakhetarpal agriyakhetarpal changed the title GH-47329: [Packaging][Python] Build and test PyArrow against the pyodide_2025_0 ABI GH-47329: [Packaging][Python][C++] Build and test PyArrow against the pyodide_2025_0 ABI Aug 14, 2025
@agriyakhetarpal
Copy link
Contributor Author

agriyakhetarpal commented Aug 14, 2025

I am trying to find out the correct place to install libatomic1; it's required by the Brotli and Protocol Buffers Emscripten builds.

I added this in af2e18d, but I'm not sure if this is the right approach. Could you please verify this commit, and also submit another build? Thanks!

@kou
Copy link
Member

kou commented Aug 14, 2025

@github-actions crossbow submit test-conda-python-emscripten

@kou
Copy link
Member

kou commented Aug 14, 2025

FYI: You can use UBUNTU=22.04 archery docker run ubuntu-cpp-emscripten on local.

See https://arrow.apache.org/docs/developers/continuous_integration/archery.html how to setup Archery.

@github-actions
Copy link

Revision: af2e18d

Submitted crossbow builds: ursacomputing/crossbow @ actions-5d3d1c591a

Task Status
test-conda-python-emscripten GitHub Actions

@agriyakhetarpal
Copy link
Contributor Author

The -fwasm-exceptions flag was apparently not needed, so I've reverted it.

You sure about this? I think it's definitely wrong to pass -fexceptions....

I am not sure. I, too, think it is needed. Here is the relevant CI run: https://github.com/ursacomputing/crossbow/actions/runs/16956640331/job/48060086633#step:6:2545

@agriyakhetarpal
Copy link
Contributor Author

Hmm, now the C++ build and Python compilation are succeeding, but we have a failure at link time. We have:

wasm-ld: error: attempted static link of dynamic object release/libarrow_python.so

em++: error: '/root/emsdk/upstream/bin/wasm-ld -o release/lib.cpython-313-wasm32-emscripten.so CMakeFiles/lib.dir/lib.cpp.o release/libarrow_python.so -L/root/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten -L/root/emsdk/upstream/emscripten/src/lib --relocatable -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr' failed (returned 1)

This indicates that we either intended to build libarrow_python.a instead of libarrow_python.so, or that we are incorrectly passing -static somewhere at link time and one of the SHARED/STATIC build flags was incorrectly turned ON/OFF. I suspect it is the former.

@agriyakhetarpal
Copy link
Contributor Author

Could you please submit another build? Thanks!

@AlenkaF
Copy link
Member

AlenkaF commented Aug 14, 2025

@github-actions crossbow submit test-conda-python-emscripten

@github-actions
Copy link

Revision: 20de7ad

Submitted crossbow builds: ursacomputing/crossbow @ actions-1b4dc9adab

Task Status
test-conda-python-emscripten GitHub Actions

@agriyakhetarpal
Copy link
Contributor Author

Thanks!

It's a bit strange that Node.js just became unresponsive in the CI run despite receiving an error. Nevertheless, the failure shows up as:

Failed to load dynamic library /lib/python3.13/site-packages/pyarrow/_acero.cpython-313-wasm32-emscripten.so: 0

which subsequently seeps into the import pyarrow.lib as _lib statement.

@agriyakhetarpal
Copy link
Contributor Author

I'm unsure if 781b54a is the right approach, but it doesn't hurt to try – I suppose that if it works, it would work with _csv, _compute, and all other Cython modules. Could someone please submit another build for me? Thank you!

@kou
Copy link
Member

kou commented Aug 14, 2025

@github-actions crossbow submit test-conda-python-emscripten

@github-actions
Copy link

Revision: f73c495

Submitted crossbow builds: ursacomputing/crossbow @ actions-f4846c031b

Task Status
test-conda-python-emscripten GitHub Actions

@agriyakhetarpal
Copy link
Contributor Author

agriyakhetarpal commented Aug 14, 2025

Yup, this isn't right. Sorry! I'll try to find out the cause of the acero load error locally.

@agriyakhetarpal agriyakhetarpal marked this pull request as draft August 14, 2025 23:41
@agriyakhetarpal
Copy link
Contributor Author

I've isolated the problem to the Cython extension modules indeed; all of them fail to load (just that Node.js stops at the first one in the Crossbow runs):

Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_acero.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_acero.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_compute.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_csv.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_csv.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_orc.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_orc.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_parquet.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_dataset_parquet.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_feather.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_feather.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_fs.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_fs.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_json.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_json.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_orc.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_orc.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_parquet.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_parquet.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_pyarrow_cpp_tests.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_pyarrow_cpp_tests.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_substrait.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/_substrait.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)
Failed to load lib  /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/lib.cpython-313-wasm32-emscripten.so
Error: Failed to load dynamic library /Users/agriyakhetarpal/Desktop/arrow/python/.venv-pyodide/lib/python3.13/site-packages/pyarrow/lib.cpython-313-wasm32-emscripten.so: 0
    at X.loadDynlib (/Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/pyodide.asm.js:8:659221)
    at async main (file:///Users/agriyakhetarpal/Library/Caches/.pyodide-xbuildenv-0.30.5/0.28.1/xbuildenv/pyodide-root/dist/python_cli_entry.mjs:91:7)

and comparing with older built, working wheels: all of these shared objects should depend on libarrow_python.so, but they no longer do.

@agriyakhetarpal
Copy link
Contributor Author

all of these shared objects should depend on libarrow_python.so, but they no longer do.

...which is because I changed to building libarrow_python.a in 20de7ad to fix the linkage, but that means this is not the right approach, and #47330 (comment) has to be solved differently. The Cython modules must link to libarrow_python.so.

@agriyakhetarpal
Copy link
Contributor Author

There is also the fact that the warning as follows:

em++: warning: linking a library with `-shared` will emit a static object file.  This is a form of emulation to support existing build systems.  If you want to build a runtime shared library use the SIDE_MODULE setting. [-Wemcc]

appears with Pyodide 0.28, but not with Pyodide 0.26.

@raulcd
Copy link
Member

raulcd commented Aug 18, 2025

@github-actions crossbow submit test-conda-python-emscripten

@github-actions
Copy link

Revision: eae25be

Submitted crossbow builds: ursacomputing/crossbow @ actions-49d30e1fce

Task Status
test-conda-python-emscripten GitHub Actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants