-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Follow up task to #6637, which adds initial support for reading files that use Parquet modular encryption.
The Parquet format allows encrypting some or all column data while keeping footers in plaintext for compatibility with readers that don't support encryption. Readers that support encryption can still verify the integrity of the footer though, as a 28 byte nonce and GCM tag are written after the plaintext footer metadata (see https://github.com/apache/parquet-format/blob/master/Encryption.md#55-plaintext-footer-mode).
This should be supported in arrow-rs to allow readers to verify the integrity of plaintext footers.
This should probably be optional, eg. in C++ Parquet there's a FileDecryptionProperties::Builder::disable_footer_signature_verification method to allow disabling this.