Add source-layer shuffle to iceberg-source for correct and scalable C…#6682

Open

lawofcycles wants to merge 20 commits intoopensearch-project:mainfrom

lawofcycles:feature/iceberg-source-shuffle

Commits on Apr 2, 2026

Add source-layer shuffle to iceberg-source for correct and scalable CDC processing
lawofcycles
committed
skip updating lastsnapshotId when shuffle failed
lawofcycles
committed
spotless apply
lawofcycles
committed
Add S3 certificate path support and ssl_insecure_disable_verification to shuffle server
lawofcycles
committed
Fix coalesce to collect index from all nodes and extract ShuffleNodeClient utility
lawofcycles
committed
Add remote shuffle file cleanup via HTTP DELETE endpoint on all nodes
lawofcycles
committed
Fix shuffle write completion key race condition by creating GlobalState before partitions
lawofcycles
committed
Fix shuffle Avro serialization to use Iceberg DataWriter/PlannedDataReader for correct type handling
lawofcycles
committed
Use common HTTP server infrastructure for shuffle
lawofcycles
committed

Commits on Apr 10, 2026

Replace fully qualified ByteBuffer with import in ShuffleHttpService
lawofcycles
committed

Commits on Apr 11, 2026