Skip to content

Commit 7523d17

Browse files
zsxwingJackey Lee
authored andcommitted
[SPARK-26042][SS][TESTS] Fix a potential hang in KafkaContinuousSourceTopicDeletionSuite
## What changes were proposed in this pull request? As initializing lazy vals shares the same lock, a thread is trying to initialize `executedPlan` when `isRDD` is running, this thread will hang forever. This PR just materializes `executedPlan` so that accessing it when `toRdd` is running doesn't need to wait for a lock ## How was this patch tested? Jenkins Closes apache#23023 from zsxwing/SPARK-26042. Authored-by: Shixiong Zhu <[email protected]> Signed-off-by: Shixiong Zhu <[email protected]>
1 parent 0d8b0ca commit 7523d17

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,12 @@ class ContinuousExecution(
262262

263263
reportTimeTaken("runContinuous") {
264264
SQLExecution.withNewExecutionId(
265-
sparkSessionForQuery, lastExecution)(lastExecution.toRdd)
265+
sparkSessionForQuery, lastExecution) {
266+
// Materialize `executedPlan` so that accessing it when `toRdd` is running doesn't need to
267+
// wait for a lock
268+
lastExecution.executedPlan
269+
lastExecution.toRdd
270+
}
266271
}
267272
} catch {
268273
case t: Throwable if StreamExecution.isInterruptionException(t, sparkSession.sparkContext) &&

0 commit comments

Comments
 (0)