feat(python): Add extra_columns parameter to scan_parquet

[nameexhaustion]

Supercedes #22695

• ref Tracking for Iceberg cloud enablement (scans only) #22450

Changes:

• Adds an extra_columns to scan_parquet
• (Rust) extra_columns_policy is now added under UnifiedScanArgs
• Introduces an internal ScanOptions Python class to consolidate input parsing of shared scan options (i.e. those in UnifiedScanArgs).

Fixes:

• Fixes Error not raised for extra columns outside schema with combination of select(all()) and allow_missing_columns=True #22218
  • This wasn't fixed until now, as users would not have been able to scan this case without the extra_columns parameter

[braaannigan]

@nameexhaustion @ion-elgreco Should this extra_columns parameter for scan_parquet be surfaced in scan_delta? I get exceptions at the moment where I try to load a multifile delta table where columns have been added over time. I'd like to be able to enable the extra_columns=True argument in scan_delta.

[ion-elgreco]

@nameexhaustion @ion-elgreco Should this extra_columns parameter for scan_parquet be surfaced in scan_delta? I get exceptions at the moment where I try to load a multifile delta table where columns have been added over time. I'd like to be able to enable the extra_columns=True argument in scan_delta.

Looks like a bug to me, providing the schema before-hand should be enough information for the reader to figure out what to do with missing columns or extra columns.

[nameexhaustion]

I get exceptions at the moment where I try to load a multifile delta table where columns have been added over time.

This PR addresses the case where columns are removed rather than added. The case when columns are added are already handled by allow_missing_columns.

I'd like to be able to enable the extra_columns=True argument in scan_delta.

This will be internally enabled by default in the future for scan_delta(), the user will not need to configure these manually.

[codecov]

Codecov Report

Attention: Patch coverage is 80.11050% with 36 lines in your changes missing coverage. Please review.

Project coverage is 80.97%. Comparing base (b9dd8cd) to head (aea4dfc).
Report is 27 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-python/src/io/mod.rs	73.43%	34 Missing ⚠️
...polars-plan/src/plans/optimizer/expand_datasets.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #22699      +/-   ##
==========================================
- Coverage   81.01%   80.97%   -0.05%     
==========================================
  Files        1671     1675       +4     
  Lines      236925   237101     +176     
  Branches     2792     2792              
==========================================
+ Hits       191956   191998      +42     
- Misses      44299    44433     +134     
  Partials      670      670              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.