[SPARK-49246][SQL][TESTS][FOLLOWUP] Fix flaky CollationSuite #48031
Conversation
panbingkun
changed the title
MaxGekk
left a comment
MaxGekk
left a comment
There was a problem hiding this comment.
cc @amaliujia @cloud-fan Please, review this since your changes cause the test failure.
|
I'm confused, the test were all green when I merged it. |
| val result = sql(s"select distinct a from $table").collect() | ||
| assert(result.length === 1) | ||
| val data = result.head.get(0).asInstanceOf[scala.collection.mutable.ArraySeq[String]] | ||
| assert(data === Array("aaa") || data === Array("AAA")) |
There was a problem hiding this comment.
so the test becomes flaky? Reading the test query I think both aaa and AAA are correct result. cc @stefankandic
There was a problem hiding this comment.
Yes, it becomes flaky. The returned value is not non-deterministic.
There was a problem hiding this comment.
You mean just non-deterministic, I guess.
|
I guess we should extend |
I want to do this, but maybe we can let it in first and turn GA green. |
Agree, let's unblock other PRs first of all. |
|
@panbingkun |
Branch-3.5 does not have this class, as follows: |
LuciferYang
changed the title
| checkAnswer(sql(s"select distinct a from $table"), Seq(Row(Seq("aaa")))) | ||
| val result = sql(s"select distinct a from $table").collect() | ||
| assert(result.length === 1) | ||
| val data = result.head.get(0).asInstanceOf[scala.collection.mutable.ArraySeq[String]] |
There was a problem hiding this comment.
| val data = result.head.get(0).asInstanceOf[scala.collection.mutable.ArraySeq[String]] | |
| val data = result.head.getSeq[String](0) |
panbingkun
changed the title
|
thanks @panbingkun for working on it, I was also confused for a while |
|
We should discuss the general fix carefully. I don't think we should do case insensitive string comparison in |
|
thanks, merging to master! |
5 participants
What changes were proposed in this pull request?
The pr is following up #48019.
Why are the changes needed?
Fix flaky
CollationSuite.After this PR, GA's
CollationSuitefailed, let's fix it first.https://github.com/panbingkun/spark/actions/runs/10765984604/job/29851047311
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Update existed UT.
Was this patch authored or co-authored using generative AI tooling?
No.