[Feature][Core] Support monitoring the busyness level of Zeta engine tasks

[zhangshenghang]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

Background

Currently, when a job is running, it is difficult to quickly determine "where it is slowing down/where there is congestion/whether it is being held back by downstream processes". Troubleshooting often relies only on logs or experience.

What needs to be achieved

Provide real-time operation indicators and trends on the job running page to visually display three types of information:

1. Whether the link is being slowed down by downstream processes (whether upstream sending often needs to wait)
2. Whether the intermediate buffer is congested (whether it is nearly full, whether it is repeatedly full)
3. Whether operators in each stage are slowing down (busyness/idleness of reading/processing/writing and time consumption per unit of data)

Which indicators to look at

Link side: Proportion of downstream waiting time; Buffer occupancy (occupancy ratio + occupancy/capacity); Auxiliary information such as waiting duration
Node side:
Source: Reading ratio, idle ratio
Transform: Busy ratio, average processing time (per record), input volume/output volume
Sink: Busy ratio, average writing time (per record), writing volume, and time consumption of submission-related stages

Screenshot of the effect

• The function has been implemented so far. I will contribute it in version 3.0. Below are several example diagrams that can accurately locate the specific where it gets stuck.
• It will also support viewing via API

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zhangshenghang]

This feature will complement #10434 to complete the overall engine performance monitoring

[DanielCarter-stack]

Thanks for the proposal! This feature doesn't exist in the dev branch.

Current state:

• Metrics in MetricNames.java focus on throughput (QPS, bytes) and cumulative counts
• BaseService.getJobMetrics() returns basic aggregated metrics without task busyness ratios or wait time breakdowns
• IntermediateDisruptor (buffer size: 1024) doesn't expose occupancy rate or wait time metrics
• Web UI docs lack the busyness visualization panels

Implementation suggestions:

1. Add new metric constants in MetricNames.java (e.g., SOURCE_BUSY_RATIO, SINK_WAIT_TIME_NS, BUFFER_OCCUPANCY_RATIO)
2. Track execution time breakdowns in ConnectorMetricsCalcContext using System.nanoTime()
3. Expose buffer occupancy statistics in IntermediateQueue implementations
4. Extend BaseService.getJobMetrics() to include new metric types while maintaining backward compatibility

Questions:

1. How should the time window for busyness calculation be configured (rolling 1s/5s vs. since job start)?
2. Should these time-based metrics be configurable due to potential performance overhead?
3. Will the API response format be backward-compatible with existing clients?

[chl-wxp]

good job