Skip to content

[SPARK-23015][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure #43706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Rafnel wants to merge 5 commits into from
Closed

[SPARK-23015][WINDOWS] Fix bug in Windows where starting multiple Spark instances within the same second causes a failure #43706

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
94070ab
Update spark-class2.cmd to mitigate bug in Windows where starting mul…
Rafnel Nov 7, 2023
eed35fe
Remove space between RANDOM and TIME in Spark launch file
Rafnel Nov 8, 2023
316fc48
Remove %RANDOM% from temporary Spark launcher file to fix race condit…
Rafnel Mar 14, 2024
a5b952f
Remove unnecessary goto statement in Spark Windows launcher
Rafnel Mar 21, 2024
ba8b056
Update Windows Spark launcher to use check if Powershell is installed…
Rafnel Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions bin/spark-class2.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,24 @@ if not "x%JAVA_HOME%"=="x" (
)
)

rem SPARK-23015: We create a temporary text file when launching Spark.
rem This file must be given a unique name or else we risk a race condition when launching multiple instances close together.
rem The best way to create a unique file name is to add a GUID to the file name. Use Powershell to generate the GUID.
where powershell.exe >nul 2>&1
if %errorlevel%==0 (
FOR /F %%a IN ('POWERSHELL -COMMAND "$([guid]::NewGuid().ToString())"') DO (set RANDOM_SUFFIX=%%a)
) else (
rem If Powershell is not installed, try to create a random file name suffix using the Windows %RANDOM%.
rem %RANDOM% is seeded with 1-second granularity so it is highly likely that two Spark instances
rem launched within the same second will fail to start.
rem Note that Powershell is automatically installed on all Windows OS from Windows 7/Windows Server 2008 R2 and onward.
set RANDOM_SUFFIX=%RANDOM%
)

rem The launcher library prints the command to be executed in a single line suitable for being
rem executed by the batch interpreter. So read all the output of the launcher into a variable.
:gen
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
rem SPARK-28302: %RANDOM% would return the same number if we call it instantly after last call,
rem so we should make it sure to generate unique file to avoid process collision of writing into
rem the same file concurrently.
if exist %LAUNCHER_OUTPUT% goto :gen
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM_SUFFIX%.txt

rem unset SHELL to indicate non-bash environment to launcher/Main
set SHELL=
"%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" org.apache.spark.launcher.Main %* > %LAUNCHER_OUTPUT%
Expand Down