chore: update kernel to 0.18, speculative fix for python test #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: DrakeLin <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
# Description
As of today, the Rust meta-crate re-exports the storage crates but never
calls their register_handlers helpers. That means every Rust binary
still has to remember to call deltalake::gcp::register_handlers(None)
(and the equivalents for S3/Azure/etc.) before using cloud URIs, even
though the Python bindings auto-register. This PR brings the meta-crate
to parity so DeltaOps::try_from_uri("gs://…") works out of the box when
the gcs feature is enabled.
# Problem
- Users of the deltalake crate must manually register each storage
backend before working with gs://, s3://, abfss://, etc.
- Forgetting the call leads to
DeltaTableError::InvalidTableLocation("Unknown scheme: gs"), which
blocks workflows like DataFusion writers on GCS.
- Docs/examples didn’t make it obvious when manual registration was
still required.
# Solution
- Add feature-gated ctor hooks in crates/deltalake/src/lib.rs that call
register_handlers(None) for AWS, Azure, GCS, HDFS, LakeFS, and Unity as
soon as their features are enabled.
- Pull in the lightweight ctor = "0.2" dependency so the hooks run at
startup.
- Add a small regression test that exercises
DeltaTableBuilder::from_uri("gs://…") with the gcs feature to guard
against regressions.
- Update the GCS integration docs and changelog to explain that the
meta-crate now auto-registers backends while deltalake-core users still
need to call the storage crates explicitly.
# Changes
- crates/deltalake/src/lib.rs: new #[ctor::ctor] modules for s3, azure,
gcs, hdfs, lakefs, and unity.
- crates/deltalake/Cargo.toml: add ctor dependency.
- crates/deltalake/tests/gcs_auto_registration.rs: new smoke tests for
gs:// URI recognition when the gcs feature is enabled.
- docs/integrations/object-storage/gcs.md & CHANGELOG.md: document the
auto-registration behavior.
# Testing
- cargo check -p deltalake --all-features
- cargo test -p deltalake --features gcs
- cargo test --test gcs_auto_registration --features gcs
- cargo build --example pharma_pipeline_gcs --features gcs,datafusion
# Documentation
- docs/integrations/object-storage/gcs.md
- CHANGELOG.md
---------
Signed-off-by: Ethan Urbanski <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
…n and Rust Profiles for different use cases: - dev: Fast local development with good debug experience (Cargo defaults) - test: test builds with minimal debug info to save disk/RAM (custom) - release: Production Rust crates - maximum performance (Cargo defaults) - bench: For benchmarking with flamegraphs (cargo bench) - profiling: For performance profiling with release opts + debug info - ci: CI/CD optimized - fast builds, release-like performance - python-release: Python wheel builds - portable, reproducible (PyPI releases) Signed-off-by: Florian Valeye <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Florian Valeye <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
…om_uri Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
--- updated-dependencies: - dependency-name: ctor dependency-version: 0.6.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
This work was really cool when @houqp explored it. In 2025 we're not really reliant on dynamodb locking and at some point in the distant future maybe we'll not need a DynamoDBLogStore either. Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
--- updated-dependencies: - dependency-name: convert_case dependency-version: 0.9.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
- Fix panic when get_add_actions() is called on tables with no add actions - Return empty RecordBatch with correct schema instead of panicking - Add unit test to verify get_add_actions() works after delete and vacuum Fixes delta-io#3918 Signed-off-by: Manish Sogiyawar <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Manish Sogiyawar <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
See delta-io#3918 Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
… overwrite (delta-io#3912) # Description Remove references in docs that suggest using `partition_filters` for selectively overwriting partitions, which has been removed from the `write_deltalake` API. # Related Issue(s) fixes delta-io#3904 Signed-off-by: zyd14 <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Fixes delta-io#3886 Signed-off-by: R. Tyler Croy <[email protected]> Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
# Conflicts: # crates/core/src/kernel/models/actions.rs
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: Stephen Carman <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This should resolve the failed test (maybe)