[SPARK-28302] [CORE] Make sure to generate unique output file for SparkLauncher on Windows #25076

Ngone51 · 2019-07-08T12:24:32Z

What changes were proposed in this pull request?

When using SparkLauncher to submit applications concurrently with multiple threads under Windows, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. The issue can be reproduced by this demo.

After digging into the code, I find that, Windows cmd %RANDOM% would return the same number if we call it instantly(e.g. < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it tries to write the same file which has already been opened for writing by another app.

We should make sure to generate unique output file for SparkLauncher on Windows to avoid this issue.

How was this patch tested?

Tested manually on Windows.

Ngone51 · 2019-07-08T12:26:19Z

ping @jiangxb1987 @vanzin @srowen

srowen

Looks reasonable to me. That's the only usage of %RANDOM%. Is there any 'do-while' loop in Windows scripts? if not seems OK to me.

Ngone51 · 2019-07-08T14:38:31Z

Is there any 'do-while' loop in Windows scripts?

Still didn't find it after searching a lot.

SparkQA · 2019-07-08T14:46:59Z

Test build #107346 has finished for PR 25076 at commit b55dfc6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-07-08T20:08:19Z

cc @HyukjinKwon , too.

HyukjinKwon · 2019-07-09T00:37:17Z

let me retrigger the AppVeyor. Possibily related.

HyukjinKwon · 2019-07-09T01:21:59Z

Seems a different issue. I made a fix - #25081.

dongjoon-hyun

For CMD %RANDOM%, I also checked that the seed is based on the clock time when the CMD session started. The current workaround looks okay to me, too.

dongjoon-hyun · 2019-07-09T02:02:15Z

bin/spark-class2.cmd

+:gen
 set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
+rem SPARK-28302: %RANDOM% would return the same number if we call it instantly after last call,
+rem so we should make sure to generate unique file to avoid process collision of writing into a


nit.

make sure -> make it sure.

into a same file -> into the same file.

Ngone51 · 2019-07-09T02:41:34Z

Thank you @HyukjinKwon for your fix.

dongjoon-hyun · 2019-07-09T03:07:51Z

Retest this please.

dongjoon-hyun · 2019-07-09T03:08:14Z

Since R fix is merged, I retriggered this.

dongjoon-hyun · 2019-07-09T03:08:41Z

Oh, there is a better way (close and reopen). :)

SparkQA · 2019-07-09T04:19:01Z

Test build #107383 has finished for PR 25076 at commit 9273f33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-07-09T04:24:23Z

retest this please

SparkQA · 2019-07-09T04:57:16Z

Test build #107385 has finished for PR 25076 at commit 9273f33.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-07-09T06:46:46Z

Test build #107389 has finished for PR 25076 at commit 9273f33.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-07-09T06:49:10Z

Merged to master, branch-2.4 and branch-2.3.

…kLauncher on Windows ## What changes were proposed in this pull request? When using SparkLauncher to submit applications **concurrently** with multiple threads under **Windows**, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. The issue can be reproduced by this [demo](https://issues.apache.org/jira/secure/attachment/12973920/Main.scala). After digging into the code, I find that, Windows cmd `%RANDOM%` would return the same number if we call it instantly(e.g. < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it tries to write the same file which has already been opened for writing by another app. We should make sure to generate unique output file for SparkLauncher on Windows to avoid this issue. ## How was this patch tested? Tested manually on Windows. Closes #25076 from Ngone51/SPARK-28302. Authored-by: wuyi <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 925f620) Signed-off-by: HyukjinKwon <[email protected]>

Ngone51 · 2019-07-09T11:58:32Z

Thanks! @srowen @dongjoon-hyun @HyukjinKwon

…kLauncher on Windows ## What changes were proposed in this pull request? When using SparkLauncher to submit applications **concurrently** with multiple threads under **Windows**, some apps would show that "The process cannot access the file because it is being used by another process" and remains in LOST state at the end. The issue can be reproduced by this [demo](https://issues.apache.org/jira/secure/attachment/12973920/Main.scala). After digging into the code, I find that, Windows cmd `%RANDOM%` would return the same number if we call it instantly(e.g. < 500ms) after last call. As a result, SparkLauncher would get same output file(spark-class-launcher-output-%RANDOM%.txt) for apps. Then, the following app would hit the issue when it tries to write the same file which has already been opened for writing by another app. We should make sure to generate unique output file for SparkLauncher on Windows to avoid this issue. ## How was this patch tested? Tested manually on Windows. Closes apache#25076 from Ngone51/SPARK-28302. Authored-by: wuyi <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 925f620) Signed-off-by: HyukjinKwon <[email protected]>

casagrande-stefano · 2020-05-26T14:45:34Z

The correct way is modify spark-class2.cmd.
I changed the row:
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt

with:
rem set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
SET RND=
for /f "delims=" %%F in ('powershell Get-Random') do (SET "RND=%%F")
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RND%.txt

and the system works fine.

make sure to generate unique output file

b55dfc6

srowen approved these changes Jul 8, 2019

View reviewed changes

dongjoon-hyun added the SPARK CORE label Jul 8, 2019

HyukjinKwon approved these changes Jul 9, 2019

View reviewed changes

HyukjinKwon closed this Jul 9, 2019

HyukjinKwon reopened this Jul 9, 2019

dongjoon-hyun approved these changes Jul 9, 2019

View reviewed changes

dongjoon-hyun reviewed Jul 9, 2019

View reviewed changes

address comment

9273f33

HyukjinKwon closed this Jul 9, 2019

HyukjinKwon reopened this Jul 9, 2019

HyukjinKwon closed this in 925f620 Jul 9, 2019

Rafnel mentioned this pull request Nov 7, 2023

[SPARK-23015][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure #43706

Closed

[SPARK-28302] [CORE] Make sure to generate unique output file for SparkLauncher on Windows #25076

[SPARK-28302] [CORE] Make sure to generate unique output file for SparkLauncher on Windows #25076

Uh oh!

Conversation

Ngone51 commented Jul 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Ngone51 commented Jul 8, 2019

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Ngone51 commented Jul 8, 2019

Uh oh!

SparkQA commented Jul 8, 2019

Uh oh!

dongjoon-hyun commented Jul 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jul 9, 2019

Uh oh!

HyukjinKwon commented Jul 9, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jul 9, 2019

Choose a reason for hiding this comment

Uh oh!

Ngone51 commented Jul 9, 2019

Uh oh!

dongjoon-hyun commented Jul 9, 2019

Uh oh!

dongjoon-hyun commented Jul 9, 2019

Uh oh!

dongjoon-hyun commented Jul 9, 2019

Uh oh!

SparkQA commented Jul 9, 2019

Uh oh!

HyukjinKwon commented Jul 9, 2019

Uh oh!

SparkQA commented Jul 9, 2019

Uh oh!

SparkQA commented Jul 9, 2019

Uh oh!

HyukjinKwon commented Jul 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ngone51 commented Jul 9, 2019

Uh oh!

casagrande-stefano commented May 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Ngone51 commented Jul 8, 2019 •

edited

Loading

dongjoon-hyun commented Jul 8, 2019 •

edited

Loading

HyukjinKwon commented Jul 9, 2019 •

edited

Loading

casagrande-stefano commented May 26, 2020 •

edited

Loading