Skip to content

Commit b949420

Browse files
belieferHyukjinKwon
authored andcommitted
[SPARK-31372][SQL][TEST][FOLLOW-UP] Improve ExpressionsSchemaSuite so that easy to track the diff
### What changes were proposed in this pull request? This PR follows up #28194. As discussed at https://github.com/apache/spark/pull/28194/files#r418418796. This PR will improve `ExpressionsSchemaSuite` so that easy to track the diff. Although `ExpressionsSchemaSuite` at line https://github.com/apache/spark/blob/b7cde42b04b21c9bfee6535199cf385855c15853/sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala#L165 just want to compare the total size between expected output size and the newest output size, the scalatest framework will output the extra information contains all the content of expected output and newest output. This PR will try to avoid this issue. After this PR, the exception looks like below: ``` [info] - Check schemas for expression examples *** FAILED *** (7 seconds, 336 milliseconds) [info] 340 did not equal 341 Expected 332 blocks in result file but got 333. Try regenerate the result files. (ExpressionsSchemaSuite.scala:167) [info] org.scalatest.exceptions.TestFailedException: [info] at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530) [info] at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529) [info] at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560) [info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:503) [info] at org.apache.spark.sql.ExpressionsSchemaSuite.$anonfun$new$1(ExpressionsSchemaSuite.scala:167) ``` ### Why are the changes needed? Make the exception more concise and clear. ### Does this PR introduce _any_ user-facing change? 'No'. ### How was this patch tested? Jenkins test. Closes #28430 from beliefer/improve-expressions-schema-suite. Authored-by: beliefer <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
1 parent 372ccba commit b949420

1 file changed

Lines changed: 24 additions & 22 deletions

File tree

sql/core/src/test/scala/org/apache/spark/sql/ExpressionsSchemaSuite.scala

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -136,38 +136,39 @@ class ExpressionsSchemaSuite extends QueryTest with SharedSparkSession {
136136
}
137137
}
138138

139+
val header = Seq(
140+
s"<!-- Automatically generated by${getClass.getSimpleName} -->",
141+
"## Summary",
142+
s" - Number of queries: ${outputs.size}",
143+
s" - Number of expressions that missing example: ${missingExamples.size}",
144+
s" - Expressions missing examples: ${missingExamples.mkString(",")}",
145+
"## Schema of Built-in Functions",
146+
"| Class name | Function name or alias | Query example | Output schema |",
147+
"| ---------- | ---------------------- | ------------- | ------------- |"
148+
)
149+
139150
if (regenerateGoldenFiles) {
140-
val missingExampleStr = missingExamples.mkString(",")
141-
val goldenOutput = {
142-
s"<!-- Automatically generated by${getClass.getSimpleName} -->\n" +
143-
"## Summary\n" +
144-
s" - Number of queries: ${outputs.size}\n" +
145-
s" - Number of expressions that missing example: ${missingExamples.size}\n" +
146-
s" - Expressions missing examples: $missingExampleStr\n" +
147-
"## Schema of Built-in Functions\n" +
148-
"| Class name | Function name or alias | Query example | Output schema |\n" +
149-
"| ---------- | ---------------------- | ------------- | ------------- |\n" +
150-
outputBuffer.mkString("\n")
151-
}
151+
val goldenOutput = (header ++ outputBuffer).mkString("\n")
152152
val parent = resultFile.getParentFile
153153
if (!parent.exists()) {
154154
assert(parent.mkdirs(), "Could not create directory: " + parent)
155155
}
156156
stringToFile(resultFile, goldenOutput)
157157
}
158158

159+
val outputSize = outputs.size
160+
val headerSize = header.size
159161
val expectedOutputs: Seq[QueryOutput] = {
160-
val goldenOutput = fileToString(resultFile)
161-
val lines = goldenOutput.split("\n")
162+
val expectedGoldenOutput = fileToString(resultFile)
163+
val lines = expectedGoldenOutput.split("\n")
164+
val expectedSize = lines.size
162165

163-
// The header of golden file has one line, plus four lines of the summary and three
164-
// lines of the header of schema table.
165-
assert(lines.size == outputs.size + 8,
166-
s"Expected ${outputs.size + 8} blocks in result file but got ${lines.size}. " +
167-
s"Try regenerate the result files.")
166+
assert(expectedSize == outputSize + headerSize,
167+
s"Expected $expectedSize blocks in result file but got " +
168+
s"${outputSize + headerSize}. Try regenerate the result files.")
168169

169-
Seq.tabulate(outputs.size) { i =>
170-
val segments = lines(i + 8).split('|')
170+
Seq.tabulate(outputSize) { i =>
171+
val segments = lines(i + headerSize).split('|')
171172
QueryOutput(
172173
className = segments(1).trim,
173174
funcName = segments(2).trim,
@@ -177,7 +178,8 @@ class ExpressionsSchemaSuite extends QueryTest with SharedSparkSession {
177178
}
178179

179180
// Compare results.
180-
assert(expectedOutputs.size == outputs.size, s"Number of queries not equals")
181+
assert(expectedOutputs.size == outputSize,
182+
"The number of queries not equals the number of expected queries.")
181183

182184
outputs.zip(expectedOutputs).foreach { case (output, expected) =>
183185
assert(expected.sql == output.sql, "SQL query did not match")

0 commit comments

Comments
 (0)