Skip to content

[SPARK-3068]remove MaxPermSize option for jvm 1.8 #1967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adrian-wang wants to merge 2 commits into from
Closed

[SPARK-3068]remove MaxPermSize option for jvm 1.8 #1967

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9c32941
remove MaxPermSize option for jvm 1.8
adrian-wang Aug 15, 2014
142cbb4
code refine
adrian-wang Aug 18, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion bin/spark-class
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,14 @@ else
exit 1
fi
fi
JAVA_VERSION=$($RUNNER -version 2>&1 | sed 's/java version "\(.*\)\.\(.*\)\..*"/\1\2/; 1q')

# Set JAVA_OPTS to be able to load native libraries and to set heap size
JAVA_OPTS="-XX:MaxPermSize=128m $OUR_JAVA_OPTS"
if [ "$JAVA_VERSION" -ge 18 ]; then
JAVA_OPTS="$OUR_JAVA_OPTS"
else
JAVA_OPTS="-XX:MaxPermSize=128m $OUR_JAVA_OPTS"
fi
JAVA_OPTS="$JAVA_OPTS -Xms$OUR_JAVA_MEM -Xmx$OUR_JAVA_MEM"
# Load extra JAVA_OPTS from conf/java-opts, if it exists
if [ -e "$FWDIR/conf/java-opts" ] ; then
Expand Down
8 changes: 7 additions & 1 deletion bin/spark-class2.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,13 @@ rem All drivers use SPARK_JAVA_OPTS + SPARK_DRIVER_MEMORY. The repl also uses SP
)

rem Set JAVA_OPTS to be able to load native libraries and to set heap size
set JAVA_OPTS=-XX:MaxPermSize=128m %OUR_JAVA_OPTS% -Djava.library.path=%SPARK_LIBRARY_PATH% -Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
for /f "tokens=3" %%i in ('java -version 2^>^&1 ^| find "version"') do set jversion=%%i
for /f "tokens=1 delims=_" %%i in ("%jversion:~1,-1%") do set jversion=%%i
if "%jversion%" geq "1.8.0" (
set JAVA_OPTS=%OUR_JAVA_OPTS% -Djava.library.path=%SPARK_LIBRARY_PATH% -Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
) else (
set JAVA_OPTS=-XX:MaxPermSize=128m %OUR_JAVA_OPTS% -Djava.library.path=%SPARK_LIBRARY_PATH% -Xms%OUR_JAVA_MEM% -Xmx%OUR_JAVA_MEM%
)
rem Attention: when changing the way the JAVA_OPTS are assembled, the change must be reflected in CommandUtils.scala!

rem Test whether the user has built Spark
Expand Down
13 changes: 11 additions & 2 deletions core/src/main/scala/org/apache/spark/deploy/worker/CommandUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,17 @@ object CommandUtils extends Logging {
extraEnvironment = command.environment)
val userClassPath = command.classPathEntries ++ Seq(classPath)

Seq("-cp", userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
permGenOpt ++ libraryOpts ++ workerLocalOpts ++ command.javaOpts ++ memoryOpts
val runner = getEnv("JAVA_HOME", command).map(_ + "/bin/java").getOrElse("java")
Copy link
Contributor

@aarondav aarondav Aug 15, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is all this code and munging in case JAVA_HOME is not the same Java as the Worker was started with? Otherwise it seems we could just do sys.env("java.version").

Copy link
Contributor Author

@adrian-wang adrian-wang Aug 18, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are now using the java which the worker started with to run executor, so to judge perm size setting we should refer to the specific java version.

val jvmversion = Utils.executeAndGetOutput(Seq(runner + " -version "),
extraEnvironment = command.environment)
val version = jvmversion.substring(jvmversion.indexOf("\"") + 1, jvmversion.indexOf("_"))
if (version.compareTo("1.8.0") < 0) {
Seq("-cp", userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
permGenOpt ++ libraryOpts ++ workerLocalOpts ++ command.javaOpts ++ memoryOpts
Copy link
Contributor

@aarondav aarondav Aug 15, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not copy-paste this, just make permGenOpt = if (version == 1.8.0) Some("-XX:MaxPerMSize=128m") else None and unconditionally include it in this Seq.

} else {
Seq("-cp", userClassPath.filterNot(_.isEmpty).mkString(File.pathSeparator)) ++
libraryOpts ++ workerLocalOpts ++ command.javaOpts ++ memoryOpts
}
}

/** Spawn a thread that will redirect a given stream to a file */
Expand Down