Add a proposal process #513

julienledem · 2025-08-08T00:02:38Z

Rationale for this change

We need a better way to propose changes

What changes are included in this PR?

adding a process to the repo.

Do these changes have PoC implementations?

yes

alamb · 2025-08-11T13:38:46Z

Related mailing list thread: https://lists.apache.org/thread/hms38c0807r3qpst7ykf4b42xwpb23g7
Related google doc: https://docs.google.com/document/d/1qGDnOyoNyPvcN4FCRhbZGAvp0SfewlWo-WVsai5IKUo/edit?tab=t.0

alamb

this looks like a great start to me

alamb · 2025-08-11T13:43:58Z

proposals/README.md

+
+## Proposal lifecycle
+
+Discuss -> Draft -> POC -> Approved -> Implementation -> Release


I suggest reducing the number of stages here to make the process seem less daunting and more accurately describe the community aspect

I also think this is not quite how I have seen the current proposals work.

Specifically, I suggest NOT saying the PMC has a gatekeeping role for deciding if the community can try to implement proposals -- as this doesn't reflect my experience. Instead I think the the PMC should keep the formal gatekeeping activity restricted to the final approved modification to the spec. This will reduce the administrative burden on both proposals and the PMC.

Suggested change

Discuss -> Draft -> POC -> Approved -> Implementation -> Release

Discuss -> Draft / POC -> Implementations -> Approval

Done
I also updated the wording and clarified transition to remove any hint of gatekeeping (except for final approval for release)

alamb · 2025-08-11T13:44:56Z

proposals/README.md

+You can attach a google doc to collaborate on the general idea with the community.
+
+### Draft
+Once the discussion has stabilized and you are ready to start a POC, open a PR to add a new Markdown file in the proposals folder and give more visibility to the work in progress.


Perhaps you can link to some examples, such as

PARQUET-2249: Introduce IEEE 754 total order #221

What purpose does the separate mark-down doc intended to serve? Per @alamb 's point above we haven't had ones for recent changes?

In general this seems fine to me as it probably helps developers do better archeology.

That was my idea.
We can also just make this a table that points to github issues given a pre-defined template?
That might be easier.
What do you think?

alamb · 2025-08-11T13:45:47Z

proposals/README.md

+### POC
+The proposal document can evolve along the course of the POC. In particular to add more links to findings and performance evaluations. Collaboration is encouraged. More validation on the POC increases the chances of success.
+
+Make sure you consider the [#Requirements] to ensure the success of the POC.


I recommend combining the Draft / POC stage -- it seems like any change also has a POC that shows what the proposal looks like in code so it makes more sense to me if they are combined

alamb · 2025-08-11T13:47:46Z

proposals/README.md

+### Approved for Implementation
+Once the POC has concluded, we should have a clear idea of whether we want to pursue the implementation accross the ecosystem. A PMC vote will formalize that stage
+
+### Implementation
+At this stage we need to meet the contribution guidelines to confsider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)


I recommend combining "Approved for Implementation" and "Implementation" -- and highlighting that at this stage you will have convinced someone to implement the proposal in at least one other open source implementation and that showing at least those two implementations are interoperable

alamb · 2025-08-11T13:48:24Z

proposals/README.md

+### Release
+Once the implementation phase is finished, we can include the contribution in the next release. 
+
+## Active Proposals 


this is a nice way to list / find the active proposal

alamb · 2025-08-11T13:50:07Z

proposals/_PROPOSAL_TEMPLATE.md

+Status: DRAFT|POC|ACCEPTED|COMPLETED
+---
+
+## Description


I recommend also adding a "## Rationale" section:

Describe why this is a feature that will improve the parquet format and what alternatives currently exist for the usecase (e.g. must use a different format, or "must build additional infrastructure to avoid re-parsing footer on each query", or "must use a general purpose compression algorithm to achieve the same space, thus slowing down query performance)

emkornfield · 2025-08-18T04:32:01Z

proposals/README.md

+Once the POC has concluded, we should have a clear idea of whether we want to pursue the implementation accross the ecosystem. A PMC vote will formalize that stage
+
+### Implementation
+At this stage we need to meet the contribution guidelines to confsider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)


Suggested change

At this stage we need to meet the contribution guidelines to confsider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)

At this stage we need to meet the contribution guidelines to consider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)

integrated review feedback

add review feedback

julienledem · 2025-08-29T22:50:47Z

Thank you @alamb and @emkornfield for the feedback. I have integrated your excellent points in the proposal.
One question on whether the additional markdown doc is worth it or simply having a template for a github issue would be sufficient. And then listing the issues in the table bellow.
#513 (comment)

emkornfield · 2025-09-03T00:50:26Z

proposals/README.md

+# Proposals
+
+This proposal process is intended for significantly impactfull additions to the Parquet spec. The goal is to facilitate those projects and help them being contributed to Parquet.
+For example, changes that are not backward compatible like adding a new encoding or a new compression algorithm (older implementations can not read new files).


Nit, we refer to this elsewhere as forward compatible: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#general-guidelinespreferences-on-additions

emkornfield · 2025-09-03T00:53:09Z

proposals/README.md

+
+### Draft/POC
+Once you feel the discussion has stabilized and you are ready to start a POC, open a PR to add a new Markdown file in the proposals folder and give more visibility to the work in progress.
+The proposal document can evolve along the course of the POC. In particular to add more links to findings and performance evaluations. Collaboration is encouraged. More validation on the POC increases the chances of success.


I worry a little bit about the friction of contributors keeping the markdown up-to-date? Maybe keeping this in the google doc? But we can see how it works in practice.

I have changed the wording to remove this as a requirement. I have left it as an option if people want to have a perennial place to save documents.

emkornfield

The proposal looks reasonable, I think keeping things in a good doc until we are ready to finalize one way or another might be less friction but we can assess on the next project.

proposals/README.md

proposals/_PROPOSAL_TEMPLATE.md

Co-authored-by: Marc Cenac <[email protected]>

Signed-off-by: Julien Le Dem <[email protected]>

alamb

Looks good to me -- it is nicely described and reflects what I understand to be the reality of how this works

proposals/1_BASE64_ENCODING.md

proposals/README.md

Update proposals/README.md Co-authored-by: Andrew Lamb <[email protected]>

Signed-off-by: Julien Le Dem <[email protected]>

* start with a simple process and an example --------- Signed-off-by: Julien Le Dem <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Micah Kornfield Co-authored-by: Marc Cenac <[email protected]>

julienledem added 3 commits August 7, 2025 17:00

start with an example

e0cc257

Update 1_BASE64_ENCODING.md

faf6b23

Update README.md

ae98d5d

alamb reviewed Aug 11, 2025

View reviewed changes

emkornfield reviewed Aug 18, 2025

View reviewed changes

julienledem added 3 commits August 29, 2025 15:40

Update README.md

3d0eb6f

integrated review feedback

Update _PROPOSAL_TEMPLATE.md

edb8e82

integrated review feedback

Update README.md

51a4a3f

add review feedback

julienledem requested a review from rdblue August 29, 2025 22:55

julienledem self-assigned this Aug 29, 2025

julienledem marked this pull request as ready for review August 29, 2025 22:55

emkornfield reviewed Sep 3, 2025

View reviewed changes

emkornfield approved these changes Sep 3, 2025

View reviewed changes

mrcnc reviewed Sep 3, 2025

View reviewed changes

proposals/README.md Outdated Show resolved Hide resolved

mrcnc reviewed Sep 3, 2025

View reviewed changes

proposals/README.md Outdated Show resolved Hide resolved

mrcnc reviewed Sep 3, 2025

View reviewed changes

proposals/README.md Outdated Show resolved Hide resolved

mrcnc reviewed Sep 3, 2025

View reviewed changes

proposals/_PROPOSAL_TEMPLATE.md Outdated Show resolved Hide resolved

julienledem added 2 commits September 23, 2025 11:02

fix spelling and address feedback.

9cdce64

Co-authored-by: Marc Cenac <[email protected]>

add license header

ef310c3

Signed-off-by: Julien Le Dem <[email protected]>

julienledem force-pushed the proposals branch from 0a736d0 to ef310c3 Compare September 23, 2025 18:06

alamb approved these changes Sep 24, 2025

View reviewed changes

proposals/1_BASE64_ENCODING.md Show resolved Hide resolved

proposals/README.md Outdated Show resolved Hide resolved

proposals/README.md Outdated Show resolved Hide resolved

julienledem and others added 2 commits September 25, 2025 11:32

Update proposals/1_BASE64_ENCODING.md

39c6fd9

Update proposals/README.md Co-authored-by: Andrew Lamb <[email protected]>

fix readme

7e3cb6a

Signed-off-by: Julien Le Dem <[email protected]>

julienledem force-pushed the proposals branch from 9d0cbee to 7e3cb6a Compare September 25, 2025 18:45

julienledem merged commit 9fd57b5 into master Sep 25, 2025
8 checks passed


		## Proposal lifecycle

		Discuss -> Draft -> POC -> Approved -> Implementation -> Release

	Discuss -> Draft -> POC -> Approved -> Implementation -> Release
	Discuss -> Draft / POC -> Implementations -> Approval

	At this stage we need to meet the contribution guidelines to confsider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)
	At this stage we need to meet the contribution guidelines to consider the implementation finished (ex: two independent implementations with cross compatibility tests, spec updated, ...)

Add a proposal process #513

Add a proposal process #513

Uh oh!

Conversation

julienledem commented Aug 8, 2025

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

alamb commented Aug 11, 2025

Uh oh!

alamb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julienledem commented Aug 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emkornfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb left a comment •

edited

Loading

alamb Aug 11, 2025 •

edited

Loading