-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-33223][SS][FOLLOWUP] Clarify the meaning of "number of rows dropped by watermark" in SS UI page #30439
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…opped by watermark" in SS UI page
ec1fba1 to
d700231
Compare
|
cc. @zsxwing @xuanyuanking @gaborgsomogyi as reviewers & author of #30151 |
|
Kubernetes integration test starting |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Kubernetes integration test status success |
|
Test build #131396 has finished for PR 30439 at commit
|
|
Test build #131397 has finished for PR 30439 at commit
|
|
retest this, please |
gaborgsomogyi
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Yeah, that's the accurate meaning, thanks!
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #131408 has finished for PR 30439 at commit
|
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, @HeartSaVioR and @gaborgsomogyi .
+1, LGTM. For me, this PR is ready for merging for Apache Spark 3.1.0.
Do you have something more, @HeartSaVioR ?
|
I'll leave this PR to you, @HeartSaVioR . |
|
Thanks all for reviewing. Given this change is just a textural change and the metric was added by me, I don't feel I can get further review comments. Merging to master. |
What changes were proposed in this pull request?
This PR fixes the representation to clarify the meaning of "number of rows dropped by watermark" in SS UI page.
Why are the changes needed?
Aggregated Number Of State Rows Dropped By Watermarksays that the dropped rows are from the state, whereas they're not. We say "evicted from the state" for the case, which is "normal" to emit outputs and reduce memory usage of the state.The metric actually represents the number of "input" rows dropped by watermark, and the meaning of "input" is relative to the "stateful operator". That's a bit confusing as we normally think "input" as "input from source" whereas it's not.
Does this PR introduce any user-facing change?
Yes, UI element & tooltip change.
How was this patch tested?
Only text change in UI, so we know how thing will be changed intuitively.