Skip to content

[SPARK-15480][UI][Streaming]show missed InputInfo in streaming UI #13259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mwws wants to merge 3 commits into from
Closed

[SPARK-15480][UI][Streaming]show missed InputInfo in streaming UI #13259

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
83351b8
show missed input info in streaming info page
wei-mao-intel May 23, 2016
a3d05e0
revert perivous change and fix bug in other way
wei-mao-intel Jun 1, 2016
ffd1787
add unit test
wei-mao-intel Jun 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import java.util.Properties
import java.util.concurrent.{ConcurrentHashMap, TimeUnit}

import scala.collection.JavaConverters._
import scala.collection.mutable.HashSet
import scala.util.Failure

import org.apache.commons.lang.SerializationUtils
Expand Down Expand Up @@ -64,6 +65,8 @@ class JobScheduler(val ssc: StreamingContext) extends Logging {

private var eventLoop: EventLoop[JobSchedulerEvent] = null

private val inputInfoMissedTimes = HashSet[Time]()

def start(): Unit = synchronized {
if (eventLoop != null) return // scheduler has already been started

Expand Down Expand Up @@ -139,6 +142,7 @@ class JobScheduler(val ssc: StreamingContext) extends Logging {
def submitJobSet(jobSet: JobSet) {
if (jobSet.jobs.isEmpty) {
logInfo("No jobs added for time " + jobSet.time)
inputInfoMissedTimes.add(jobSet.time)
} else {
listenerBus.post(StreamingListenerBatchSubmitted(jobSet.toBatchInfo))
jobSets.put(jobSet.time, jobSet)
Expand Down Expand Up @@ -193,6 +197,14 @@ class JobScheduler(val ssc: StreamingContext) extends Logging {
listenerBus.post(StreamingListenerOutputOperationCompleted(job.toOutputOperationInfo))
logInfo("Finished job " + job.id + " from job set of time " + jobSet.time)
if (jobSet.hasCompleted) {
// submit fake BatchCompleted event to show missing inputInfo on Streaming UI
inputInfoMissedTimes.foreach (time => {
val streamIdToInputInfos = inputInfoTracker.getInfo(time)
val fakeJobSet = JobSet(time, Seq(), streamIdToInputInfos)
listenerBus.post(StreamingListenerBatchCompleted(fakeJobSet.toBatchInfo))
Copy link
Contributor

@jerryshao jerryshao May 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this change the semantics if posing fake StreamingListenerBatchCompleted, if user's code rely on this, will this break their assumptions?

Copy link
Author

@mwws mwws May 24, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, it would be a breaking change in that case. Give the fact that some information is indeed missing now, we need either send more events or add additional fields in current event. The later might be better and more correct in semantics, although it can still break current implementation of user's custom listener. And I noticed that listener interface is annotated as DeveloperAPI, so it's expected that API might be changed. How do you think?

})
inputInfoMissedTimes.clear()

jobSets.remove(jobSet.time)
jobGenerator.onBatchCompletion(jobSet.time)
logInfo("Total delay: %.3f s for time %s (execution: %.3f s)".format(
Expand Down
2 changes: 1 addition & 1 deletion streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobSet.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ case class JobSet(

def hasStarted: Boolean = processingStartTime > 0

def hasCompleted: Boolean = incompleteJobs.isEmpty
def hasCompleted: Boolean = incompleteJobs.isEmpty && processingStartTime >= 0

// Time taken to process all the jobs from the time they started processing
// (i.e. not including the time they wait in the streaming scheduler queue)
Expand Down