Skip to content

Commit 1222f1e

Browse files
committed
Add more info.
1 parent d480632 commit 1222f1e

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

  • sql/catalyst/src/main/scala/org/apache/spark/sql/internal

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1386,7 +1386,11 @@ object SQLConf {
13861386
buildConf("spark.sql.streaming.statefulOperator.correctnessCheck")
13871387
.internal()
13881388
.doc("When true, the stateful operators for streaming query will be checked for possible " +
1389-
"correctness issue. Once the issue is detected, Spark will throw analysis exception. " +
1389+
"correctness issue due to global watermark. The correctness issue comes from queries " +
1390+
"containing stateful operation which can emit rows older than the current watermark " +
1391+
"plus allowed late record delay, which are \"late rows\" in downstream stateful " +
1392+
"operations and these rows can be discarded. Please refer the programming guide doc for " +
1393+
"more details. Once the issue is detected, Spark will throw analysis exception. " +
13901394
"When this config is disabled, Spark will just print warning message for users. " +
13911395
"Prior to Spark 3.1.0, the behavior is disabling this config.")
13921396
.version("3.1.0")

0 commit comments

Comments
 (0)