Skip to content

PARQUET-686: Add optional {signed,unsigned}_{min,max} fields to Statistics #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
a10y wants to merge 8 commits into from
Closed

PARQUET-686: Add optional {signed,unsigned}_{min,max} fields to Statistics #42

Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c06e197
Add optional {signed,unsigned}_{min,max} fields to Statistics
Aug 25, 2016
f310c5a
Remove parquet-mr specific reference
Aug 25, 2016
8f33c3a
removal of explicit signed_{min,max} in Statistics
Aug 28, 2016
35632de
Revert "removal of explicit signed_{min,max} in Statistics"
Aug 31, 2016
fc79696
Improved comment
Aug 31, 2016
eaf8c54
fix inline comments
Aug 31, 2016
b9fddb2
Update comment and add README section
Sep 1, 2016
474ece8
Add comment about Binary sorting
Sep 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions src/main/thrift/parquet.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,15 @@ enum FieldRepetitionType {
/**
* Statistics per row group and per page
* All fields are optional.
*
* For BinaryStatistics in Parquet, we want to distinguish between the
* statistics derived from comparisons of signed or unsigned bytes. The min
* and max fields are deprecated for BinaryStatistics, instead relying on
* specification of {unsigned,signed}_{min,max}. The filter API should allow
* clients to specify which statistics and method of comparison should be used
* for filtering. To maintain backward format compatibility, when filtering
* based on signed statistics the signed_min and signed_max are checked first,
* and if they are unset it falls back to using the values in min and max.
Copy link
Contributor

@isnotinvain isnotinvain Aug 31, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets clarify that min and max are always signed if present.

I wonder if any of this belongs in the README too?

*/
struct Statistics {
/** min and max value of the column, encoded in PLAIN encoding */
Expand All @@ -203,6 +212,12 @@ struct Statistics {
3: optional i64 null_count;
/** count of distinct values occurring */
4: optional i64 distinct_count;
/* Signed min and max for binary fields */
5: optional binary signed_max;
6: optional binary signed_min;
/* Unsigned min and max for binary fields */
7: optional binary unsigned_max;
8: optional binary unsigned_min;
}

/**
Expand Down