Skip to content

Commit 44c7320

Browse files
HeartSaVioRRaphaël Luta
authored andcommitted
[MINOR][DOC][SS] Correct description of minPartitions in Kafka option
## What changes were proposed in this pull request? `minPartitions` has been used as a hint and relevant method (KafkaOffsetRangeCalculator.getRanges) doesn't guarantee the behavior that partitions will be equal or more than given value. https://github.com/apache/spark/blob/d67b98ea016e9b714bef68feaac108edd08159c9/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetRangeCalculator.scala#L32-L46 This patch makes clear the configuration is a hint, and actual partitions could be less or more. ## How was this patch tested? Just a documentation change. Closes apache#25332 from HeartSaVioR/MINOR-correct-kafka-structured-streaming-doc-minpartition. Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 7ffc00c) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
1 parent 2582207 commit 44c7320

1 file changed

Lines changed: 4 additions & 2 deletions

File tree

docs/structured-streaming-kafka-integration.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -379,10 +379,12 @@ The following configurations are optional:
379379
<td>int</td>
380380
<td>none</td>
381381
<td>streaming and batch</td>
382-
<td>Minimum number of partitions to read from Kafka.
382+
<td>Desired minimum number of partitions to read from Kafka.
383383
By default, Spark has a 1-1 mapping of topicPartitions to Spark partitions consuming from Kafka.
384384
If you set this option to a value greater than your topicPartitions, Spark will divvy up large
385-
Kafka partitions to smaller pieces.</td>
385+
Kafka partitions to smaller pieces. Please note that this configuration is like a `hint`: the
386+
number of Spark tasks will be **approximately** `minPartitions`. It can be less or more depending on
387+
rounding errors or Kafka partitions that didn't receive any new data.</td>
386388
</tr>
387389
</table>
388390

0 commit comments

Comments
 (0)