Skip to content

[SPARK-2929][SQL] Refactored Thrift server and CLI suites #1856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
liancheng wants to merge 3 commits into from
Closed

[SPARK-2929][SQL] Refactored Thrift server and CLI suites #1856

liancheng wants to merge 3 commits into from

Conversation

@liancheng
Copy link
Contributor

@liancheng liancheng commented Aug 8, 2014

Removed most hard coded timeout, timing assumptions and all Thread.sleep. Simplified IPC and synchronization with scala.sys.process and future/promise so that the test suites can run more robustly and faster.

@SparkQA
Copy link

SparkQA commented Aug 8, 2014

QA tests have started for PR 1856. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18212/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 8, 2014

QA results for PR 1856:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class CliSuite extends FunSuite with BeforeAndAfterAll with ProcessIOTest with Logging {
class HiveThriftServer2Suite extends FunSuite with ProcessIOTest with Logging {
trait ProcessIOTest {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18212/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 10, 2014

QA tests have started for PR 1856. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18280/consoleFull

@liancheng
Copy link
Contributor Author

liancheng commented Aug 10, 2014

The reason of the timeout occurred in build failure is unknown due to lacking of necessary logs (maybe something's wrong in the test suites, or maybe it's just running too slow). Logged all external process output by default to help diagnosing.

@SparkQA
Copy link

SparkQA commented Aug 10, 2014

QA results for PR 1856:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class CliSuite extends FunSuite with BeforeAndAfterAll with Logging {
class HiveThriftServer2Suite extends FunSuite with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18280/consoleFull

@liancheng
Copy link
Contributor Author

liancheng commented Aug 10, 2014

Jenkins complains

14/08/10 06:10:28 ERROR ClientBase: Can't get Master Kerberos principal for use as renewer

Not sure why Kerberos is involved here...

@liancheng
Copy link
Contributor Author

liancheng commented Aug 11, 2014

Checked the Jenkins server and a fresh build passes these suites successfully.

(I wonder do I have the privilege to issue a retest? Let me try...)

@liancheng
Copy link
Contributor Author

liancheng commented Aug 11, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 11, 2014

QA tests have started for PR 1856. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18313/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 11, 2014

QA results for PR 1856:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class CliSuite extends FunSuite with BeforeAndAfterAll with Logging {
class HiveThriftServer2Suite extends FunSuite with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18313/consoleFull

@SparkQA
Copy link

SparkQA commented Aug 15, 2014

QA tests have started for PR 1856 at commit c783024.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 15, 2014

QA tests have finished for PR 1856 at commit c783024.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Aug 15, 2014

QA tests have started for PR 1856 at commit a175255.

  • This patch merges cleanly.

@liancheng
Copy link
Contributor Author

liancheng commented Aug 19, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 1856 at commit f68cee7.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 1856 at commit f68cee7.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@liancheng
Copy link
Contributor Author

liancheng commented Aug 19, 2014

With a significant longer timeout (1 minute), CliSuite passes while HiveThriftServer2Suite still timeout. Trying an even larger timeout to see whether the test suite is slowed down by parallel builds on Jenkins.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have started for PR 1856 at commit e47e510.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 19, 2014

QA tests have finished for PR 1856 at commit e47e510.

  • This patch fails unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class CliSuite extends FunSuite with BeforeAndAfterAll with Logging
    • class HiveThriftServer2Suite extends FunSuite with Logging

@liancheng
Refactored Thrift server and CLI suites
0ee921d 
Logged necessary process output for diagnosis

Rebased to master and more refactoring

Fixed Metastore JDBC url option in CliSuite

Set longer timeouts

Using an even larger timeout, and adding more logs
@liancheng
Cleaned up test output
0e12e71
@SparkQA
Copy link

SparkQA commented Aug 22, 2014

QA tests have started for PR 1856 at commit 0e12e71.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 22, 2014

QA tests have finished for PR 1856 at commit 0e12e71.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class CliSuite extends FunSuite with BeforeAndAfterAll with Logging
    • class HiveThriftServer2Suite extends FunSuite with Logging

@SparkQA
Copy link

SparkQA commented Aug 24, 2014

QA tests have started for PR 1856 at commit 2d914ca.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Aug 24, 2014

QA tests have finished for PR 1856 at commit 2d914ca.

  • This patch passes unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class CliSuite extends FunSuite with BeforeAndAfterAll with Logging
    • class HiveThriftServer2Suite extends FunSuite with Logging

@liancheng liancheng mentioned this pull request Aug 25, 2014
Closed
asfgit pushed a commit that referenced this pull request Aug 25, 2014
@liancheng @marmbrus
[SPARK-2929][SQL] Refactored Thrift server and CLI suites
292f28d 
Removed most hard coded timeout, timing assumptions and all `Thread.sleep`. Simplified IPC and synchronization with `scala.sys.process` and future/promise so that the test suites can run more robustly and faster.

Author: Cheng Lian <[email protected]>

Closes #1856 from liancheng/thriftserver-tests and squashes the following commits:

2d914ca [Cheng Lian] Minor refactoring
0e12e71 [Cheng Lian] Cleaned up test output
0ee921d [Cheng Lian] Refactored Thrift server and CLI suites

(cherry picked from commit cae9414)
Signed-off-by: Michael Armbrust <[email protected]>
@marmbrus
Copy link
Contributor

marmbrus commented Aug 25, 2014

Thanks! I've merged this into master and 1.1

@asfgit asfgit closed this in cae9414 Aug 25, 2014
xiliu82 pushed a commit to xiliu82/spark that referenced this pull request Sep 4, 2014
@liancheng @conviva-zz
[SPARK-2929][SQL] Refactored Thrift server and CLI suites
5e3271f 
Removed most hard coded timeout, timing assumptions and all `Thread.sleep`. Simplified IPC and synchronization with `scala.sys.process` and future/promise so that the test suites can run more robustly and faster.

Author: Cheng Lian <[email protected]>

Closes apache#1856 from liancheng/thriftserver-tests and squashes the following commits:

2d914ca [Cheng Lian] Minor refactoring
0e12e71 [Cheng Lian] Cleaned up test output
0ee921d [Cheng Lian] Refactored Thrift server and CLI suites
@liancheng liancheng deleted the thriftserver-tests branch September 24, 2014 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liancheng @SparkQA @marmbrus