[CELEBORN-1299] Introduce JVM profiling in Celeborn Worker using async-profiler

[SteNicholas]

What changes were proposed in this pull request?

Introduce JVM profiling JVMProfier in Celeborn Worker using async-profiler to capture CPU and memory profiles.

Why are the changes needed?

async-profiler is a sampling profiler for any JDK based on the HotSpot JVM that does not suffer from Safepoint bias problem. It has low overhead and doesn’t rely on JVMTI. It avoids the safepoint bias problem by using the AsyncGetCallTrace API provided by HotSpot JVM to profile the Java code paths, and Linux’s perf_events to profile the native code paths. It features HotSpot-specific APIs to collect stack traces and to track memory allocations.
The feature introduces a profier plugin that does not add any overhead unless enabled and can be configured to accept profiler arguments as a configuration parameter. It should support to turn profiling on/off, includes the jar/binaries needed for profiling.

Backport [SPARK-46094] Support Executor JVM Profiling.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Worker cluster test.

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 49.03%. Comparing base (a371f93) to head (dc629d2).
Report is 2 commits behind head on main.

Files	Patch %	Lines
...cala/org/apache/celeborn/common/CelebornConf.scala	85.72%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2409      +/-   ##
==========================================
+ Coverage   48.85%   49.03%   +0.18%     
==========================================
  Files         209      209              
  Lines       13102    13123      +21     
  Branches     1134     1134              
==========================================
+ Hits         6400     6433      +33     
+ Misses       6282     6272      -10     
+ Partials      420      418       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[SteNicholas]

cc @mridulm, @RexXiong. PTAL.

[mridulm]

The change itself looks good to me, thanks for working on this !
Why are we targeting only worker ?

[SteNicholas]

@mridulm, thanks for review. IMO, most scenarios only need coding profiling of Celeborn Worker in production practice, therefore worker coding profiling is given priority to support. If there is strong need to target master, I would like to support in future.
cc @RexXiong.

[mridulm]

As the fleet size increases (both compute cluster (20k+ nodes) and Celeborn cluster (2k+ workers) ), master scaling is also a challenge.
We dont necessarily need to add support for master in this PR, as long as adding that support is not a challenge

[SteNicholas]

@mridulm, I agree with the above point of view. Thanks for your suggestion.
@RexXiong, do you have other comments?

[RexXiong]

LGTM, thanks! merge to main(v0.5.0)

[pan3793]

"macOS" - the case is matter, let's respect the official product name https://support.apple.com/macos

[pan3793]

LICENSE/NOTICE are missed

[pan3793]

arm64, don't split it out.

[pan3793]

if possible, please choose a default value independent of the working dir, the user likely uses the absolute path of the script to start the process under an arbitrary dir, which would cause indeterminate results. one solution is use ${CELEBORN_HOME}/profiling and resolve the ${CELEBORN_HOME} in def workerJvmProfilerLocalDir: String

[pan3793]

where can I find all the option candidates and valid combinations? A detailed description or a hyperlink is required.