[SPARK-45209][CORE][UI] Flame Graph Support For Executor Thread Dump Page

[yaooqinn]

What changes were proposed in this pull request?

This PR draws a CPU Flame Graph by Java stack traces for executors and drivers. Currently, the Java stack traces is just a SNAPSHOT, not sampling at a certain frequency for a period. Sampling might be considered an upcoming feature out of the scope of this PR.

If you are new to flame graphs, there are also some references you can refer to learn about the basic concepts and details.

[1] Flame Graphs
[2] FLIP-165: Operator's Flame Graphs
[3] Java in Flames. mixed-mode flame graphs provide a… | by Netflix Technology Blog
[4] HProf

Pending features

This PR mainly focuses on the UI, independent of the profiling steps. What we might have in the future are:

• Flame Graph Support For Task Thread Page which SPARK-45151 added
• Add ProfilingExecutor(max, interval) message to profile whole executor
• Add ProfileTask(taskId, max, interval) message to profile an certain task
• Different views for on/off/full CPUs
• Mixed mode profiling, which might rely upon some ext libs at runtime
• And so on.

Why are the changes needed?

Performance is always an important design factor in Spark. It is desirable to provide better visibility into the distribution of CPU resources while executing user code alongside the Spark kernel. One of the most visually effective means to do that is Flame Graphs, which visually presents the data gathered by performance profiling tools used by developers for performance tuning their applications.

Does this PR introduce any user-facing change?

yes

How was this patch tested?

locally

Was this patch authored or co-authored using generative AI tooling?

no

[HyukjinKwon]

Looks cool. cc @mridulm FYI

[zhengruifeng]

awesome!

[mridulm]

The UI looks nice ! Thanks for working on this @yaooqinn :-)

My main concern is around effectively capturing stack frames without safepoint bias, correlating it to the specific task, and for executor flamegraph which threads to capture. @HyukjinKwon might remember what we had built for Safari, but we did that outside of the scope of Spark due to limitations (which might not be relevant now to be honest).

This has been called out of scope - but details around what we planning to support (in the first cut) would be great to understand.

[yaooqinn]

This PR mainly focuses on the UI, independent of the profiling steps. What we might have in the future are:

• Flame Graph Support For Task Thread Page, which SPARK-45151 added
• Add ProfilingExecutor(max, interval) message to profile the whole executor, which returns the same data structure with TriggerThreadDump message
• Add ProfileTask(taskId, max, interval) message to profile a specific task, which returns the same data structure with TaskThreadDump message
• Different views for on/off/full CPUs
• Mixed mode profiling, which might rely upon some ext libs at runtime
• And so on.

[rednaxelafx]

One important aspect of flame graphs is the semantics of the "width" of the bars. It can be defined to mean anything, e.g. aggregated profiling ticks (i.e. number of samples) or wall clock duration etc.

What's the intended semantics of "width" here for the thread dump snapshot?

[yaooqinn]

Currently, the width represents the number of samples. the CPU time is not gathered yet, see also SPARK-44896

[mridulm]

This PR mainly focuses on the UI, independent of the profiling steps. What we might have in the future are:

• Flame Graph Support For Task Thread Page, which SPARK-45151 added

SPARK-45151 added ability to fetch stack dump, not flame graph support - and more importantly, from point of view of flamegraphs and performance analysis, the stack dump returned suffers from safepoint bias.

If this is an initial implementation, which we plan to refine over 4.0 release, that sounds fine to me - at a minimum, users/deployments should have the ability to override how the stack dump - for flamegraphs not SPARK-45151 - is generated.

• Add ProfilingExecutor(max, interval) message to profile the whole executor, which returns the same data structure with TriggerThreadDump message

We can iterate more on this when we have a PR for it.
Thanks for the details.

• Add ProfileTask(taskId, max, interval) message to profile a specific task, which returns the same data structure with TaskThreadDump message
• Different views for on/off/full CPUs
• Mixed mode profiling, which might rely upon some ext libs at runtime
• And so on.

[mridulm]

+CC @thejdeep as well.

[rednaxelafx]

Currently, the width represents the number of samples.

In the current implementation in this PR, the width is essentially representing the "number of threads" that's sharing the same bottom portion of the stack, from a single snapshot of the threads dump, right?

[yaooqinn]

@rednaxelafx Yes

[beliefer]

@yaooqinn Looks beautiful!

[yaooqinn]

Thank you @beliefer

[LuciferYang]

+1, LGTM
Very cool features should be very helpful to frontline maintenance engineers.

[LuciferYang]

Merged into master for Spark 4.0.
Thanks @yaooqinn
Thanks for your review. @HyukjinKwon @zhengruifeng @rednaxelafx @mridulm @beliefer

[cloud-fan]

This is a great feature! can we add a config to turn it on/off? I just worried about any possible bugs and stop the UI from working.