Skip to content

feat: Reinterpret binary data to fixed size numerical array#22840

Merged
nameexhaustion merged 84 commits intopola-rs:mainfrom
itamarst:add-binary-as-numerical-array
Aug 1, 2025
Merged

feat: Reinterpret binary data to fixed size numerical array#22840
nameexhaustion merged 84 commits intopola-rs:mainfrom
itamarst:add-binary-as-numerical-array

Conversation

@itamarst
Copy link
Copy Markdown
Contributor

@itamarst itamarst commented May 20, 2025

Fixes #22126

An updated version on code in #20456, by @balbok0, who did the initial implementation that got this working, so the bulk of the work.

@github-actions github-actions Bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels May 20, 2025
Comment thread crates/polars-core/src/datatypes/dtype.rs Outdated
@itamarst
Copy link
Copy Markdown
Contributor Author

itamarst commented May 20, 2025

Before I submit this for review I am going to:

  • Review this code myself, address whatever I see.
  • Address failing CI tests.

Comment thread crates/polars-ops/src/chunked_array/binary/cast_binary_to_numerical.rs Outdated
Comment thread crates/polars-ops/src/chunked_array/binary/cast_binary_to_numerical.rs Outdated
Comment thread crates/polars-ops/src/chunked_array/binary/cast_binary_to_numerical.rs Outdated
Comment thread crates/polars-ops/src/chunked_array/binary/cast_binary_to_numerical.rs Outdated
Comment thread crates/polars-ops/src/chunked_array/binary/cast_binary_to_numerical.rs Outdated
Comment thread crates/polars-ops/src/chunked_array/binary/namespace.rs Outdated
Comment thread crates/polars-ops/src/chunked_array/binary/namespace.rs Outdated
Comment thread crates/polars-plan/src/dsl/binary.rs Outdated
Comment thread crates/polars-plan/src/dsl/binary.rs Outdated
@itamarst itamarst closed this May 20, 2025
@codecov
Copy link
Copy Markdown

codecov Bot commented May 20, 2025

Codecov Report

❌ Patch coverage is 90.83333% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.42%. Comparing base (727dd4c) to head (8377805).
⚠️ Report is 185 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-compute/src/cast/binview_to.rs 87.83% 9 Missing ⚠️
...s/polars-ops/src/chunked_array/binary/namespace.rs 94.44% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #22840      +/-   ##
==========================================
- Coverage   81.57%   81.42%   -0.16%     
==========================================
  Files        1634     1641       +7     
  Lines      221194   222455    +1261     
  Branches     2813     2835      +22     
==========================================
+ Hits       180443   181137     +694     
- Misses      40060    40619     +559     
- Partials      691      699       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread py-polars/tests/unit/operations/namespaces/test_binary.py
@itamarst itamarst changed the title feat: Reinterpret binary data as numerical array feat: Reinterpret binary data as numerical array (WIP) May 20, 2025
@itamarst
Copy link
Copy Markdown
Contributor Author

Back to you @nameexhaustion

@itamarst itamarst requested a review from nameexhaustion July 11, 2025 13:24
Comment thread crates/polars-compute/src/cast/binview_to.rs Outdated
@itamarst itamarst requested a review from nameexhaustion July 14, 2025 15:01
@itamarst
Copy link
Copy Markdown
Contributor Author

@nameexhaustion done

@nameexhaustion
Copy link
Copy Markdown
Collaborator

It looks good, but let's take some steps to ensure safety.

Comment thread crates/polars-compute/src/cast/binview_to.rs Outdated
Comment thread crates/polars-compute/src/cast/binview_to.rs Outdated
Comment thread crates/polars-compute/src/cast/binview_to.rs Outdated
@itamarst
Copy link
Copy Markdown
Contributor Author

@nameexhaustion back to you. Also added a couple of extra tests.

@itamarst itamarst requested a review from nameexhaustion July 14, 2025 19:38
Comment thread crates/polars-compute/src/cast/binview_to.rs Outdated
Comment thread crates/polars-compute/src/cast/binview_to.rs Outdated
@itamarst
Copy link
Copy Markdown
Contributor Author

@nameexhaustion Done.

@itamarst itamarst requested a review from nameexhaustion July 15, 2025 12:54
@nameexhaustion nameexhaustion added the do not merge This pull requests should not be merged right now label Jul 25, 2025
@nameexhaustion nameexhaustion merged commit 5d429ea into pola-rs:main Aug 1, 2025
30 checks passed
@nameexhaustion nameexhaustion removed the do not merge This pull requests should not be merged right now label Aug 1, 2025
Washiil pushed a commit to Washiil/polars that referenced this pull request Aug 2, 2025
…22840)

Co-authored-by: Jakub Filipek <[email protected]>
Co-authored-by: Itamar Turner-Trauring <[email protected]>
Co-authored-by: nameexhaustion <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reinterpret a binary column as a fixed shape array

5 participants