File tree Expand file tree Collapse file tree
core/src/main/scala/org/apache/spark/scheduler Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -127,8 +127,8 @@ private[spark] class CompressedMapStatus(
127127
128128/**
129129 * A [[MapStatus ]] implementation that stores the accurate size of huge blocks, which are larger
130- * than both spark.shuffle.accurateBlockThreshold. It stores the average size of other non-empty
131- * blocks, plus a bitmap for tracking which blocks are empty.
130+ * than spark.shuffle.accurateBlockThreshold. It stores the average size of other non-empty blocks,
131+ * plus a bitmap for tracking which blocks are empty.
132132 *
133133 * @param loc location where the task is being executed
134134 * @param numNonEmptyBlocks the number of non-empty blocks
@@ -211,8 +211,8 @@ private[spark] object HighlyCompressedMapStatus {
211211 val size = uncompressedSizes(i)
212212 if (size > 0 ) {
213213 numNonEmptyBlocks += 1
214- // Remove the huge blocks from the calculation for average size and have accurate size for
215- // smaller blocks.
214+ // Huge blocks are not included in the calculation for average size, thus size for smaller
215+ // blocks is more accurate .
216216 if (size < threshold) {
217217 totalSize += size
218218 } else {
You can’t perform that action at this time.
0 commit comments