Skip to content

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liancheng wants to merge 4 commits into from
+6 −16
Closed

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e5aa31a
Fixed metastore JDBC URI quotation
liancheng Aug 19, 2014
a87f83d
Extended timeout
liancheng Aug 19, 2014
26b82a0
Run SQL tests when dff contains bin/spark-sql and/or sbin/start-thrif…
liancheng Aug 19, 2014
f38c4eb
Fixed the same quotation issue in CliSuite
liancheng Aug 20, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dev/run-tests
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ JAVA_VERSION=$($java_cmd -version 2>&1 | sed 's/java version "\(.*\)\.\(.*\)\..*
# Partial solution for SPARK-1455. Only run Hive tests if there are sql changes.
if [ -n "$AMPLAB_JENKINS" ]; then
git fetch origin master:master
diffs=`git diff --name-only master | grep "^sql/"`
diffs=`git diff --name-only master | grep "^\(sql/\)\|\(bin/spark-sql\)\|\(sbin/start-thriftserver.sh\)"`
if [ -n "$diffs" ]; then
echo "Detected changes in SQL. Will run Hive test suite."
_RUN_SQL_TESTS=true
Expand Down
2 changes: 1 addition & 1 deletion sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ class CliSuite extends FunSuite with BeforeAndAfterAll with TestUtils {
val commands =
s"""../../bin/spark-sql
| --master local
| --hiveconf ${ConfVars.METASTORECONNECTURLKEY}="$jdbcUrl"
| --hiveconf ${ConfVars.METASTORECONNECTURLKEY}=$jdbcUrl
| --hiveconf ${ConfVars.METASTOREWAREHOUSE}=$WAREHOUSE_PATH
""".stripMargin.split("\\s+")

Expand Down
18 changes: 4 additions & 14 deletions ...server/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,9 +51,6 @@ class HiveThriftServer2Suite extends FunSuite with BeforeAndAfterAll with TestUt
port
}

// If verbose is true, the test program will print all outputs coming from the Hive Thrift server.
val VERBOSE = Option(System.getenv("SPARK_SQL_TEST_VERBOSE")).getOrElse("false").toBoolean

Class.forName(DRIVER_NAME)

override def beforeAll() { launchServer() }
Expand All @@ -68,21 +65,18 @@ class HiveThriftServer2Suite extends FunSuite with BeforeAndAfterAll with TestUt
val command =
s"""../../sbin/start-thriftserver.sh
| --master local
| --hiveconf hive.root.logger=INFO,console
| --hiveconf ${ConfVars.METASTORECONNECTURLKEY}="$jdbcUrl"
| --hiveconf ${ConfVars.METASTORECONNECTURLKEY}=$jdbcUrl
| --hiveconf ${ConfVars.METASTOREWAREHOUSE}=$METASTORE_PATH
| --hiveconf ${ConfVars.HIVE_SERVER2_THRIFT_BIND_HOST}=$HOST
| --hiveconf ${ConfVars.HIVE_SERVER2_THRIFT_PORT}=$PORT
""".stripMargin.split("\\s+")

val pb = new ProcessBuilder(command ++ args: _*)
val environment = pb.environment()
environment.put("HIVE_SERVER2_THRIFT_PORT", PORT.toString)
environment.put("HIVE_SERVER2_THRIFT_BIND_HOST", HOST)
process = pb.start()
inputReader = new BufferedReader(new InputStreamReader(process.getInputStream))
errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream))
waitForOutput(inputReader, "ThriftBinaryCLIService listening on")
waitForOutput(inputReader, "ThriftBinaryCLIService listening on", 300000)

// Spawn a thread to read the output from the forked process.
// Note that this is necessary since in some configurations, log4j could be blocked
Expand All @@ -91,12 +85,8 @@ class HiveThriftServer2Suite extends FunSuite with BeforeAndAfterAll with TestUt
while (true) {
val stdout = readFrom(inputReader)
val stderr = readFrom(errorReader)
if (VERBOSE && stdout.length > 0) {
println(stdout)
}
if (VERBOSE && stderr.length > 0) {
println(stderr)
}
print(stdout)
Copy link
Contributor

@marmbrus marmbrus Aug 20, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really want to print this always? Perhaps we can just print when there is a failure like we do in HiveContext

Copy link
Contributor Author

@liancheng liancheng Aug 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this in #1856, which provides a faster and robuster version of these test suites, and is much more quiet when everything's OK. We can take care of that one later though.

print(stderr)
Thread.sleep(50)
}
}
Expand Down