File tree Expand file tree Collapse file tree
sql/catalyst/src/main/scala/org/apache/spark/sql/internal Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -1386,7 +1386,11 @@ object SQLConf {
13861386 buildConf(" spark.sql.streaming.statefulOperator.correctnessCheck" )
13871387 .internal()
13881388 .doc(" When true, the stateful operators for streaming query will be checked for possible " +
1389- " correctness issue. Once the issue is detected, Spark will throw analysis exception. " +
1389+ " correctness issue due to global watermark. The correctness issue comes from queries " +
1390+ " containing stateful operation which can emit rows older than the current watermark " +
1391+ " plus allowed late record delay, which are \" late rows\" in downstream stateful " +
1392+ " operations and these rows can be discarded. Please refer the programming guide doc for " +
1393+ " more details. Once the issue is detected, Spark will throw analysis exception. " +
13901394 " When this config is disabled, Spark will just print warning message for users. " +
13911395 " Prior to Spark 3.1.0, the behavior is disabling this config." )
13921396 .version(" 3.1.0" )
You can’t perform that action at this time.
0 commit comments