Skip to content

[SPARK-11595] [SQL] [BRANCH-1.5] Fixes ADD JAR when the input path contains URL scheme #9570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 2 commits into from
Closed

[SPARK-11595] [SQL] [BRANCH-1.5] Fixes ADD JAR when the input path contains URL scheme #9570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8d06b98
Fixes SPARK-11595 for branch-1.5
liancheng Nov 9, 2015
3c0f368
Fixes test failure
liancheng Nov 11, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@

package org.apache.spark.sql.hive.execution

import java.io.File

import org.apache.hadoop.fs.Path
import org.apache.hadoop.hive.metastore.MetaStoreUtils
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.{TableIdentifier, SqlParser}
Expand Down Expand Up @@ -90,7 +93,17 @@ case class AddJar(path: String) extends RunnableCommand {
val currentClassLoader = Utils.getContextOrSparkClassLoader

// Add jar to current context
val jarURL = new java.io.File(path).toURI.toURL
val jarURL = {
val uri = new Path(path).toUri
if (uri.getScheme == null) {
// `path` is a local file path without a URL scheme
new File(path).toURI.toURL
} else {
// `path` is a URL with a scheme
uri.toURL
}
}

val newClassLoader = new java.net.URLClassLoader(Array(jarURL), currentClassLoader)
Thread.currentThread.setContextClassLoader(newClassLoader)
// We need to explicitly set the class loader associated with the conf in executionHive's
Expand Down
7 changes: 6 additions & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,12 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils {
private val sqlContext = _sqlContext

test("UDTF") {
sql(s"ADD JAR ${TestHive.getHiveFile("TestUDTF.jar").getCanonicalPath()}")
val jarPath = TestHive.getHiveFile("TestUDTF.jar").getCanonicalPath

// SPARK-11595 Fixes ADD JAR when input path contains URL scheme
val jarURL = s"file://$jarPath"
Copy link
Contributor

@yhuai yhuai Nov 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also check a URL without a scheme?

Copy link
Contributor Author

@liancheng liancheng Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to remove an added jar? I'm afraid if we add the other test case, these two may interfere with each other.

Copy link
Contributor Author

@liancheng liancheng Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually there are other test cases that already covered the case without a scheme. So I think it's fine.

Copy link
Contributor

@yhuai yhuai Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not possible.

Copy link
Contributor

@yhuai yhuai Nov 12, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok


sql(s"ADD JAR $jarURL")
// The function source code can be found at:
// https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide+UDTF
sql(
Expand Down