Is your feature request related to a problem? Please describe.
The iceberg-source shuffle storage backend (local disk) is currently hardcoded via LocalDiskShuffleStorage. Users cannot switch to alternative storage backends (e.g. S3) through pipeline configuration.
Describe the solution you'd like
Expose the shuffle storage backend as a plugin extension point, consistent with how Data Prepper handles source coordination mechanisms and authentication providers. Users would configure the storage backend in the shuffle section of the pipeline configuration.
Describe alternatives you've considered (Optional)
The current ShuffleStorage interface already abstracts the storage backend, so the main work is wiring it into the plugin framework rather than a design change.
Additional context
Raised during review of #6682: #6682 (comment)
Is your feature request related to a problem? Please describe.
The iceberg-source shuffle storage backend (local disk) is currently hardcoded via
LocalDiskShuffleStorage. Users cannot switch to alternative storage backends (e.g. S3) through pipeline configuration.Describe the solution you'd like
Expose the shuffle storage backend as a plugin extension point, consistent with how Data Prepper handles source coordination mechanisms and authentication providers. Users would configure the storage backend in the
shufflesection of the pipeline configuration.Describe alternatives you've considered (Optional)
The current
ShuffleStorageinterface already abstracts the storage backend, so the main work is wiring it into the plugin framework rather than a design change.Additional context
Raised during review of #6682: #6682 (comment)