Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions proposals/1_BASE64_ENCODING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
Author: Julien Le Dem
Created: 2025-Aug-7
Name: add BASE64 compression
Issue: https://github.com/apache/parquet-format/issues/NNN
Status: ARCHIVED
Reason: Did not compress
---

# Proposal

## Description
Add Base64 to compression algorithms.
This is not backwards compatible as a new compression alg.

## Spec

See [BASE64 spec].

## Evaluation

After trying out in the java implementation, file size doubled on average.
See prototype [here](github.com/julienledem/mypoc)

50 changes: 50 additions & 0 deletions proposals/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Proposals

## Requirements

See the [requirements document](https://docs.google.com/document/d/1qGDnOyoNyPvcN4FCRhbZGAvp0SfewlWo-WVsai5IKUo/edit?tab=t.0#heading=h.v4emiipkghrx) (Note: this doc would become a markdown page in the repo)

## Proposal lifecycle

Discuss -> Draft -> POC -> Approved -> Implementation -> Release
Copy link
Contributor

@alamb alamb Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest reducing the number of stages here to make the process seem less daunting and more accurately describe the community aspect

I also think this is not quite how I have seen the current proposals work.

Specifically, I suggest NOT saying the PMC has a gatekeeping role for deciding if the community can try to implement proposals -- as this doesn't reflect my experience. Instead I think the the PMC should keep the formal gatekeeping activity restricted to the final approved modification to the spec. This will reduce the administrative burden on both proposals and the PMC.

Suggested change
Discuss -> Draft -> POC -> Approved -> Implementation -> Release
Discuss -> Draft / POC -> Implementations -> Approval

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done
I also updated the wording and clarified transition to remove any hint of gatekeeping (except for final approval for release)


### Discuss
Start a [DISCUSS] thread on the mailing list ([email protected]) with your idea.
Once you have a better idea of the direction, open a github issue using the proposal template.
You can attach a google doc to collaborate on the general idea with the community.

### Draft
Once the discussion has stabilized and you are ready to start a POC, open a PR to add a new Markdown file in the proposals folder and give more visibility to the work in progress.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps you can link to some examples, such as

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What purpose does the separate mark-down doc intended to serve? Per @alamb 's point above we haven't had ones for recent changes?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general this seems fine to me as it probably helps developers do better archeology.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was my idea.
We can also just make this a table that points to github issues given a pre-defined template?
That might be easier.
What do you think?


### POC
The proposal document can evolve along the course of the POC. In particular to add more links to findings and performance evaluations. Collaboration is encouraged. More validation on the POC increases the chances of success.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry a little bit about the friction of contributors keeping the markdown up-to-date? Maybe keeping this in the google doc? But we can see how it works in practice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have changed the wording to remove this as a requirement. I have left it as an option if people want to have a perennial place to save documents.


Make sure you consider the [#Requirements] to ensure the success of the POC.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend combining the Draft / POC stage -- it seems like any change also has a POC that shows what the proposal looks like in code so it makes more sense to me if they are combined


### Approved for Implementation
Once the POC has concluded, we should have a clear idea of whether we want to pursue the implementation accross the ecosystem. A PMC vote will formalize that stage

### Implementation
At this stage we need to meet the contribution guidelines to confsider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend combining "Approved for Implementation" and "Implementation" -- and highlighting that at this stage you will have convinced someone to implement the proposal in at least one other open source implementation and that showing at least those two implementations are interoperable

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
At this stage we need to meet the contribution guidelines to confsider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)
At this stage we need to meet the contribution guidelines to consider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)


### Release
Once the implementation phase is finished, we can include the contribution in the next release.

## Active Proposals
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a nice way to list / find the active proposal


| ID | Description | Status |
|-----|--------------|---------|
| [github issue] | adding this new encoding | POC |
| [github issue] | add Variant type | Implementation |

## Implemented
| ID | Description | Status | release it was added |
|-----|--------------|---------|-----------------------|
| [gihub issue] | encryption | Completed | x.y.z |

## Archived

| ID | Description | Status | reason for archiving |
|-----|--------------|---------|-----------------------|
| [github issue] | [adding base64 compression](1_BASE64_ENCODING.md) | Archived | POC showed that compression ratio was not practical |

22 changes: 22 additions & 0 deletions proposals/_PROPOSAL_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Proposal

---
Author: ~your name~
Created: ~date~
Name: *short sentence describing the proposal*
Issue: https://github.com/apache/parquet-format/issues/NNN
Status: DRAFT|POC|ACCEPTED|COMPLETED
---

## Description
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend also adding a "## Rationale" section:

Describe why this is a feature that will improve the parquet format and what alternatives currently exist for the usecase (e.g. must use a different format, or "must build additional infrastructure to avoid re-parsing footer on each query", or "must use a general purpose compression algorithm to achieve the same space, thus slowing down query performance)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added!

*Short description of the proposal. Is it a new encoding? Is it backwards compatible (old readers will just ignore it)? Is it additional metadata?*

## Spec

At the proposal stage you don't need a fully fleshed out spec yet.
Please add any link to relevant documentation, papers, etc.
at the implementation stage, the details will need to be all clarified.

## Evaluation
What datasets is it tested on and what is a success criteria
Please add any link to the relevant codebase.
Loading