[SPARK-32100][CORE][TESTS][FOLLOWUP] Reduce the required test resources #29001

dongjoon-hyun · 2020-07-05T20:23:54Z

What changes were proposed in this pull request?

This PR aims to reduce the required test resources in WorkerDecommissionExtendedSuite.

Why are the changes needed?

When Jenkins farms is crowded, the following failure happens currently here

java.util.concurrent.TimeoutException: Can't find 20 executors before 60000 milliseconds elapsed
	at org.apache.spark.TestUtils$.waitUntilExecutorsUp(TestUtils.scala:326)
	at org.apache.spark.scheduler.WorkerDecommissionExtendedSuite.$anonfun$new$2(WorkerDecommissionExtendedSuite.scala:45)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the Jenkins.

SparkQA · 2020-07-05T20:32:38Z

Test build #124962 has finished for PR 29001 at commit 8b29c74.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-05T21:30:45Z

Retest this please.

SparkQA · 2020-07-05T21:44:29Z

Test build #124965 has finished for PR 29001 at commit 8b29c74.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-05T23:11:26Z

Retest this please.

SparkQA · 2020-07-05T23:23:34Z

Test build #124976 has finished for PR 29001 at commit 8b29c74.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-05T23:26:58Z

Retest this please

dongjoon-hyun · 2020-07-05T23:27:10Z

Retest this please.

SparkQA · 2020-07-06T00:49:22Z

Test build #124980 has finished for PR 29001 at commit 8b29c74.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-07-06T01:45:10Z

retest this please

SparkQA · 2020-07-06T01:52:04Z

Test build #124996 has finished for PR 29001 at commit 8b29c74.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-07-06T01:59:17Z

Thank you, @HyukjinKwon !

dongjoon-hyun · 2020-07-06T01:59:24Z

Retest this please.

SparkQA · 2020-07-06T02:06:05Z

Test build #124998 has finished for PR 29001 at commit 8b29c74.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-07-06T02:36:00Z

core/src/test/scala/org/apache/spark/scheduler/WorkerDecommissionExtendedSuite.scala

  private val conf = new org.apache.spark.SparkConf()
    .setAppName(getClass.getName)
-    .set(SPARK_MASTER, "local-cluster[20,1,512]")
+    .set(SPARK_MASTER, "local-cluster[10,1,512]")


Any particular reason we need 10 or 20 executors? It's still too many compares to other tests, whose average number should be 2 or 3. cc @holdenk

So I think this test came from the situation where we were experiencing a deadlock and we wanted to make sure we re-created the potential deadlock which happened when we decommissioned most of the executors. Now this deadlock never made it into OSS Spark, but having the test here to catch it just incase is good. I think we could catch the same deadlock with 5 executors and decommissioning 4 of them, but @dongjoon-hyun is the one who found this potential issue so I'll let him clarify :)

Thank you for your explanation.

holdenk · 2020-07-06T02:45:59Z

LGTM, thanks for working around the crowded Jenkins env :) I think we can explore if we want to reduce it further in a follow on, but I think removing causes of test flakiness sooner rather than later is better for everyone working on Spark.

dongjoon-hyun · 2020-07-06T02:54:10Z

Thank you, @holdenk and @Ngone51 . I'll reduce it to 5 in the community.

### What changes were proposed in this pull request? This PR aims to disable dependency tests(test-dependencies.sh) from Jenkins. ### Why are the changes needed? - First of all, GitHub Action provides the same test capability already and stabler. - Second, currently, `test-dependencies.sh` fails very frequently in AmpLab Jenkins environment. For example, in the following irrelevant PR, it fails 5 times during 6 hours. - #29001 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the Jenkins without `test-dependencies.sh` invocation. Closes #29004 from dongjoon-hyun/SPARK-32178. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

Ngone51 · 2020-07-06T03:05:42Z

+1, LGTM

dongjoon-hyun · 2020-07-06T03:12:08Z

This is a test-only PR and I verified this manually.

$ build/sbt "core/testOnly *.WorkerDecommissionExtendedSuite"
...
[info] WorkerDecommissionExtendedSuite:
[info] - Worker decommission and executor idle timeout (14 seconds, 679 milliseconds)
[info] - Decommission 4 executors from 5 executors in total (21 seconds, 740 milliseconds)
[info] ScalaTest
[info] Run completed in 38 seconds, 271 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

dongjoon-hyun · 2020-07-06T03:12:15Z

Merged to master.

SparkQA · 2020-07-06T04:24:49Z

Test build #125010 has finished for PR 29001 at commit 3790334.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

This PR aims to reduce the required test resources in WorkerDecommissionExtendedSuite. When Jenkins farms is crowded, the following failure happens currently [here](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2-hive-2.3/890/testReport/junit/org.apache.spark.scheduler/WorkerDecommissionExtendedSuite/Worker_decommission_and_executor_idle_timeout/) ``` java.util.concurrent.TimeoutException: Can't find 20 executors before 60000 milliseconds elapsed at org.apache.spark.TestUtils$.waitUntilExecutorsUp(TestUtils.scala:326) at org.apache.spark.scheduler.WorkerDecommissionExtendedSuite.$anonfun$new$2(WorkerDecommissionExtendedSuite.scala:45) ``` No. Pass the Jenkins. Closes apache#29001 from dongjoon-hyun/SPARK-32100-2. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[SPARK-32100][TESTS][FOLLOWUP] Reduce the required test resources

8b29c74

probot-autolabeler bot added the CORE label Jul 5, 2020

dongjoon-hyun added the TESTS label Jul 5, 2020

dongjoon-hyun changed the title ~~[SPARK-32100][TESTS][FOLLOWUP] Reduce the required test resources~~ [SPARK-32100][CORE][TESTS][FOLLOWUP] Reduce the required test resources Jul 5, 2020

HyukjinKwon approved these changes Jul 6, 2020

View reviewed changes

dongjoon-hyun mentioned this pull request Jul 6, 2020

[SPARK-32178][TESTS] Disable test-dependencies.sh from Jenkins jobs #29004

Closed

Ngone51 reviewed Jul 6, 2020

View reviewed changes

Address comments

3790334

dongjoon-hyun closed this in dea7bc4 Jul 6, 2020

[SPARK-32100][CORE][TESTS][FOLLOWUP] Reduce the required test resources #29001

[SPARK-32100][CORE][TESTS][FOLLOWUP] Reduce the required test resources #29001

Uh oh!

Conversation

dongjoon-hyun commented Jul 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jul 5, 2020

Uh oh!

dongjoon-hyun commented Jul 5, 2020

Uh oh!

SparkQA commented Jul 5, 2020

Uh oh!

dongjoon-hyun commented Jul 5, 2020

Uh oh!

SparkQA commented Jul 5, 2020

Uh oh!

dongjoon-hyun commented Jul 5, 2020

Uh oh!

dongjoon-hyun commented Jul 5, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

HyukjinKwon commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

dongjoon-hyun commented Jul 6, 2020

Uh oh!

dongjoon-hyun commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

Ngone51 Jul 6, 2020

Choose a reason for hiding this comment

Uh oh!

holdenk Jul 6, 2020

Choose a reason for hiding this comment

Uh oh!

Ngone51 Jul 6, 2020

Choose a reason for hiding this comment

Uh oh!

holdenk commented Jul 6, 2020

Uh oh!

dongjoon-hyun commented Jul 6, 2020

Uh oh!

Ngone51 commented Jul 6, 2020

Uh oh!

dongjoon-hyun commented Jul 6, 2020

Uh oh!

dongjoon-hyun commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dongjoon-hyun commented Jul 5, 2020 •

edited

Loading