[SPARK-1477]: Add the lifecycle interface #991

witgo · 2014-06-06T05:53:08Z

No description provided.

AmplabJenkins · 2014-06-06T05:57:51Z

Merged build triggered.

AmplabJenkins · 2014-06-06T05:58:53Z

Merged build started.

AmplabJenkins · 2014-06-06T06:55:32Z

Merged build finished.

AmplabJenkins · 2014-06-06T06:55:32Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15497/

AmplabJenkins · 2014-06-09T03:02:50Z

Merged build triggered.

AmplabJenkins · 2014-06-09T03:38:35Z

Merged build started.

AmplabJenkins · 2014-06-09T04:18:49Z

Merged build finished.

AmplabJenkins · 2014-06-09T04:18:49Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15552/

rxin · 2014-06-14T20:55:43Z

core/src/main/java/org/apache/spark/Service.java

shouldn't we declare this in scala?

The reason for this is java code inherited this interface easier.

Java code can actually extend Scala traits as well. Do you have other concerns?

AmplabJenkins · 2014-06-26T09:50:24Z

Merged build triggered.

AmplabJenkins · 2014-06-26T09:50:34Z

Merged build started.

AmplabJenkins · 2014-06-26T09:56:26Z

Merged build finished.

AmplabJenkins · 2014-06-26T09:56:26Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16156/

AmplabJenkins · 2014-06-26T10:25:22Z

Merged build triggered.

AmplabJenkins · 2014-06-26T10:25:29Z

Merged build started.

AmplabJenkins · 2014-06-26T11:04:14Z

Merged build finished.

AmplabJenkins · 2014-06-26T11:04:14Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16157/

witgo · 2014-06-26T12:11:13Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-26T12:15:22Z

Merged build triggered.

AmplabJenkins · 2014-06-26T12:15:30Z

Merged build started.

AmplabJenkins · 2014-06-26T12:58:29Z

Merged build finished.

AmplabJenkins · 2014-06-26T12:58:29Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16158/

SparkQA · 2014-07-15T15:57:56Z

QA tests have started for PR 991. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16684/consoleFull

SparkQA · 2014-07-15T17:02:57Z

QA tests have started for PR 991. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16686/consoleFull

SparkQA · 2014-07-15T17:15:21Z

QA results for PR 991:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
trait Lifecycle extends Service {
class SparkContext(config: SparkConf) extends Logging with Lifecycle {
class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle {
class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle {
class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverLauncher extends Lifecycle {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16684/consoleFull

SparkQA · 2014-07-15T18:38:05Z

QA results for PR 991:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
trait Lifecycle extends Service {
trait Service extends java.io.Closeable {
class SparkContext(config: SparkConf) extends Logging with Lifecycle {
class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle {
class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle {
class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverLauncher extends Lifecycle {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16686/consoleFull

SparkQA · 2014-07-16T18:17:48Z

QA tests have started for PR 991. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16741/consoleFull

SparkQA · 2014-07-16T19:58:18Z

QA results for PR 991:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
trait Lifecycle extends Service {
trait Service extends java.io.Closeable {
class SparkContext(config: SparkConf) extends Logging with Lifecycle {
class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle {
class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle {
class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverLauncher extends Lifecycle {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16741/consoleFull

SparkQA · 2014-07-29T16:19:01Z

QA tests have started for PR 991. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17361/consoleFull

SparkQA · 2014-08-03T05:59:24Z

QA tests have started for PR 991. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17805/consoleFull

SparkQA · 2014-08-03T06:29:09Z

QA results for PR 991:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
trait Lifecycle extends Service {
trait Service extends java.io.Closeable {
class SparkContext(config: SparkConf) extends Logging with Lifecycle {
class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle {
class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle {
class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverLauncher extends Lifecycle {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17805/consoleFull

witgo · 2014-08-03T06:37:13Z

Jenkins, retest this please.

SparkQA · 2014-08-03T06:39:12Z

QA tests have started for PR 991. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17806/consoleFull

SparkQA · 2014-08-03T07:34:01Z

QA results for PR 991:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
trait Lifecycle extends Service {
trait Service extends java.io.Closeable {
class SparkContext(config: SparkConf) extends Logging with Lifecycle {
class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle {
class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle {
class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle {
class ReceiverLauncher extends Lifecycle {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17806/consoleFull

rxin · 2014-08-28T07:45:40Z

@witgo I'm going to take a look at this later for 1.2. I think it's a good idea to have a Service abstraction for service that we can start/stop. The current API is slightly more complicated than necessary, but it is in good direction.

SparkQA · 2014-09-01T16:34:04Z

QA tests have started for PR 991 at commit 79f48dc.

This patch merges cleanly.

SparkQA · 2014-09-01T17:36:01Z

QA tests have finished for PR 991 at commit 79f48dc.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait Lifecycle extends Service
- trait Service extends java.io.Closeable
- class SparkContext(config: SparkConf) extends Logging with Lifecycle
- class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle
- class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle
- class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle
- class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle
- class ReceiverLauncher extends Lifecycle

rxin · 2014-09-13T05:31:28Z

Hey @witgo I thought about this more, and at this point I'm not sure if it is worth it to standardize this interface. The reason is we have a lot of "services" in Spark, but they have different ways to be initialized (some of them would require having different initialize methods that might not fit into the LifeCycle trait.

I'm also worried about the use of mixins with multiple traits having default implementations. In the past that has caused trouble in Java interoperability and binary compatibility.

rxin · 2014-09-13T05:31:41Z

Maybe we should just implement Closeable?

witgo · 2014-09-13T05:44:49Z

core/src/main/scala/org/apache/spark/Service.scala

@rxin Here has inherited Closeable class

pwendell · 2014-09-16T17:50:13Z

I also don't think we should add extra interfaces that we aren't going to use. In Spark we never interact with these components in a generic way, so I don't see any value to having this extra cruft all over the place.

pwendell · 2014-09-18T17:33:52Z

Let's close this issue for now given the comments from me and @rxin.

rxin reviewed Jun 14, 2014
View reviewed changes

witgo changed the title ~~[WIP][SPARK-1477]: Add the lifecycle interface~~ [SPARK-1477]: Add the lifecycle interface Jun 22, 2014

Add the lifecycle interface

79f48dc

witgo force-pushed the SPARK-1477 branch from 8215dc0 to 79f48dc Compare September 1, 2014 16:27

witgo mentioned this pull request Sep 10, 2014

SPARK-3470 [CORE] [STREAMING] Add Closeable / close() to Java context objects #2346

Closed

witgo reviewed Sep 13, 2014
View reviewed changes

core/src/main/scala/org/apache/spark/Service.scala

Copy link

Contributor Author

witgo Sep 13, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rxin Here has inherited Closeable class

witgo closed this Sep 18, 2014

[SPARK-1477]: Add the lifecycle interface #991

[SPARK-1477]: Add the lifecycle interface #991

Uh oh!

Conversation

witgo commented Jun 6, 2014

Uh oh!

AmplabJenkins commented Jun 6, 2014

Uh oh!

AmplabJenkins commented Jun 6, 2014

Uh oh!

AmplabJenkins commented Jun 6, 2014

Uh oh!

AmplabJenkins commented Jun 6, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

rxin Jun 14, 2014

Choose a reason for hiding this comment

Uh oh!

witgo Jun 16, 2014

Choose a reason for hiding this comment

Uh oh!

rxin Jun 26, 2014

Choose a reason for hiding this comment

Uh oh!

witgo Jun 26, 2014

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

witgo commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

AmplabJenkins commented Jun 26, 2014

Uh oh!

SparkQA commented Jul 15, 2014

Uh oh!

SparkQA commented Jul 15, 2014

Uh oh!

SparkQA commented Jul 15, 2014

Uh oh!

SparkQA commented Jul 15, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 16, 2014

Uh oh!

SparkQA commented Jul 29, 2014

Uh oh!

SparkQA commented Aug 3, 2014

Uh oh!

SparkQA commented Aug 3, 2014

Uh oh!

witgo commented Aug 3, 2014

Uh oh!