[SPARK-28582][PYSPARK] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7 #25315

WeichenXu123 · 2019-07-31T15:52:27Z

What changes were proposed in this pull request?

Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon.

How was this patch tested?

Run test

python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests"

Before
Fail on test "test_termination_sigterm". And we can see daemon process do not exit.
After
Test passed

SparkQA · 2019-07-31T16:46:00Z

Test build #108472 has finished for PR 25315 at commit 545736f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

python/pyspark/daemon.py

HyukjinKwon · 2019-08-01T00:36:42Z

cc @JoshRosen as well.

HyukjinKwon · 2019-08-01T00:50:22Z

cc @shivaram and @felixcheung too who I talked with a similar symptom but by a different reason at R worker before.

WeichenXu123 · 2019-08-02T11:46:52Z

@HyukjinKwon The issue is specific to py3.7 and can only triggered in the python test case. I add a sleep in test which can avoid this issue happen and avoid change daemon code.

HyukjinKwon

Let's fix the PR description and title. LGTM if the test passes.

HyukjinKwon · 2019-08-02T12:08:26Z

Tested in Python 3.7. LGTM

SparkQA · 2019-08-02T12:12:19Z

Test build #108566 has finished for PR 25315 at commit 688b660.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-08-02T12:22:43Z

Test build #108565 has finished for PR 25315 at commit a334251.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-08-02T13:06:40Z

Merged to master, branch-2.4 and branch-2.3.

Thanks for investigating this @WeichenXu123.

… which fail on Python 3.7 Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes #25315 from WeichenXu123/fix_py37_daemon. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit fbeee0c) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

dongjoon-hyun · 2019-08-02T17:05:17Z

python/pyspark/tests/test_daemon.py

        # request shutdown
        terminator(daemon)
-        time.sleep(1)
+        daemon.wait(5)


Hi, Guys.

The timeout argument is Python 3 only feature.

https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait

Since this will break Python 2.7 test in all branches. I'll revert this commit from all the branches.

dongjoon-hyun · 2019-08-02T17:21:16Z

PR builder also failed at the last commit in this PR.

HyukjinKwon · 2019-08-03T00:46:27Z

Oops, sorry it's my bad. Let me reopen the PR.

dongjoon-hyun · 2019-08-03T00:48:18Z

Thanks~ BTW, the Jenkins will be shutdown from tonight.

… which fail on Python 3.7 Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes apache#25315 from WeichenXu123/fix_py37_daemon. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…which fail on Python 3.7 This PR picks up #25315 back after removing `Popen.wait` usage which exists in Python 3 only. I saw the last test results wrongly and thought it was passed. Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes #25343 from HyukjinKwon/SPARK-28582. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit b3394db) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

… which fail on Python 3.7 Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes apache#25315 from WeichenXu123/fix_py37_daemon. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit fbeee0c) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…which fail on Python 3.7 This PR picks up apache#25315 back after removing `Popen.wait` usage which exists in Python 3 only. I saw the last test results wrongly and thought it was passed. Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes apache#25343 from HyukjinKwon/SPARK-28582. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit b3394db) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

… which fail on Python 3.7 Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes apache#25315 from WeichenXu123/fix_py37_daemon. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit fbeee0c) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…which fail on Python 3.7 This PR picks up apache#25315 back after removing `Popen.wait` usage which exists in Python 3 only. I saw the last test results wrongly and thought it was passed. Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7. I add a sleep after the test connection to daemon. Run test ``` python/run-tests --python-executables=python3.7 --testname "pyspark.tests.test_daemon DaemonTests" ``` **Before** Fail on test "test_termination_sigterm". And we can see daemon process do not exit. **After** Test passed Closes apache#25343 from HyukjinKwon/SPARK-28582. Authored-by: WeichenXu <weichen.xu@databricks.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit b3394db) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

init pr

545736f

dongjoon-hyun changed the title ~~[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on py3.7~~ [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7 Jul 31, 2019

mattf-apache reviewed Jul 31, 2019

View reviewed changes

python/pyspark/daemon.py Show resolved Hide resolved

HyukjinKwon reviewed Aug 1, 2019

View reviewed changes

python/pyspark/daemon.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Aug 1, 2019

View reviewed changes

python/pyspark/daemon.py Outdated Show resolved Hide resolved

dongjoon-hyun added the PYSPARK label Aug 1, 2019

update

a334251

HyukjinKwon approved these changes Aug 2, 2019

View reviewed changes

update

688b660

WeichenXu123 changed the title ~~[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7~~ [SPARK-28582][PYSPARK] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7 Aug 2, 2019

HyukjinKwon closed this in fbeee0c Aug 2, 2019

dongjoon-hyun reviewed Aug 2, 2019

View reviewed changes

HyukjinKwon mentioned this pull request Aug 3, 2019

[SPARK-28582][PYTHON] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7 #25343

Closed

WeichenXu123 deleted the fix_py37_daemon branch August 3, 2019 03:00

[SPARK-28582][PYSPARK] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7 #25315

[SPARK-28582][PYSPARK] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7 #25315

Uh oh!

Conversation

WeichenXu123 commented Jul 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jul 31, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Aug 1, 2019

Uh oh!

HyukjinKwon commented Aug 1, 2019

Uh oh!

WeichenXu123 commented Aug 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Aug 2, 2019

Uh oh!

SparkQA commented Aug 2, 2019

Uh oh!

SparkQA commented Aug 2, 2019

Uh oh!

HyukjinKwon commented Aug 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun Aug 2, 2019

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Aug 3, 2019

Uh oh!

dongjoon-hyun commented Aug 3, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

WeichenXu123 commented Jul 31, 2019 •

edited

Loading

WeichenXu123 commented Aug 2, 2019 •

edited

Loading

HyukjinKwon commented Aug 2, 2019 •

edited

Loading

dongjoon-hyun commented Aug 2, 2019 •

edited

Loading