[FEA] Parquet writer to include Column Index feature

[revans2]

Is your feature request related to a problem? Please describe.
In Parquet 1.11 a new feature was added for column indexes/page indexes.

https://github.com/apache/parquet-format/blob/master/PageIndex.md

When I grep through the code I do not see support for this feature, but I could have missed it. Spark supports using this feature on the CPU to reduce the total amount of data read from disk, and it would be great to be able to write parquet files that support this too. This is so our customers who need to read data using the CPU too can read the data written by the GPU and fast as data written by the CPU.

Describe the solution you'd like
Insert in the ColumnIndex and OffsetIndex automatically for each parquet file we write.

Describe alternatives you've considered
We cannot do this without the help of cudf, so there really are no other alternatives.

Additional context
None

[devavret]

IIUC this one is a lot simpler than the reader. Is there any benefit to implementing this before we do it for the reader?

[revans2]

The benefit would be that Spark CPU and other CPU software that is compatible with the feature will be able to read the same data much faster, because they didn't have to read in all of the pages. It would not provide a direct benefit to GPU processing without the read side changes.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[etseidl]

Hi all. This issue is a pretty important one to me. I agree with @revans2 that implementing this in the writer has value to other projects even without reader support. I've been working on this lately, and once I get approval will be submitting a draft PR.

[devavret]

Hi, @etseidl, It would be great if you could submit a PR. One blocker we had due to which we didn't implement this yet is that without reader support, we wouldn't have any unit tests for this. And even if we did have reader support, I couldn't figure out how to properly test it. Any suggestions on that front are welcome.