Skip to content
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions parquet/src/errors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ pub enum ParquetError {
/// Returned when a function needs more data to complete properly. The `usize` field indicates
/// the total number of bytes required, not the number of additional bytes.
NeedMoreData(usize),
/// Returned when a function needs more data to complete properly.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one part of this PR I am not sure about

Since the ParquetError is (now) marked as #[non_exhaustive] I don't think this is technically a breaking change. However, it would be really nice to avoid a new enum -- I will comment about this later in the PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I had this error in my original draft of ParquetMetaDataReader (3c340b7) but got talked out of it 😅 (#6392 (comment))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have known you were right and gone with your instinct!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW I could remove ParquetError::NeedMoreDataRange if I reworked the decoder to have a proper state machine internally (rather than calling into the existing ParquetMetaDataDecoder).

That is certainly my long term plan. If we like this API I will invest some time into seeing if I can sketch out what it would look like

/// The `Range<u64>` indicates the range of bytes that are needed.
NeedMoreDataRange(std::ops::Range<u64>),
}

impl std::fmt::Display for ParquetError {
Expand All @@ -69,6 +72,9 @@ impl std::fmt::Display for ParquetError {
}
ParquetError::External(e) => write!(fmt, "External: {e}"),
ParquetError::NeedMoreData(needed) => write!(fmt, "NeedMoreData: {needed}"),
ParquetError::NeedMoreDataRange(range) => {
write!(fmt, "NeedMoreDataRange: {}..{}", range.start, range.end)
}
}
}
}
Expand Down
7 changes: 4 additions & 3 deletions parquet/src/file/metadata/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,10 @@
//! metadata into parquet files. To work with metadata directly,
//! the following APIs are available:
//!
//! * [`ParquetMetaDataReader`] for reading
//! * [`ParquetMetaDataReader`] for reading from a reader for I/O
//! * [`ParquetMetaDataPushDecoder`] for decoding from bytes without I/O
//! * [`ParquetMetaDataWriter`] for writing.
//!
//! [`ParquetMetaDataReader`]: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataReader.html
//! [`ParquetMetaDataWriter`]: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataWriter.html
//!
//! # Examples
//!
Expand Down Expand Up @@ -92,6 +91,7 @@
//! * Same name, different struct
//! ```
mod memory;
mod push_decoder;
pub(crate) mod reader;
mod writer;

Expand Down Expand Up @@ -120,6 +120,7 @@ use crate::schema::types::{
};
#[cfg(feature = "encryption")]
use crate::thrift::{TCompactSliceInputProtocol, TSerializable};
pub use push_decoder::ParquetMetaDataPushDecoder;
pub use reader::{FooterTail, PageIndexPolicy, ParquetMetaDataReader};
use std::ops::Range;
use std::sync::Arc;
Expand Down
Loading
Loading