[tests]: Add tests for high frequency telemetry by Pterosaur · Pull Request #20379 · sonic-net/sonic-mgmt

Pterosaur · 2025-08-22T16:18:29Z

Description of PR

Summary:
Fixes # (issue)

Type of change

Back port request

Approach

What is the motivation for this PR?

Add tests for high frequency telemetry

How did you do it?

Add new test cases

How did you verify/test it?

Check in the sn5600 platform locally

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-08-22T16:18:37Z

/azp run

azure-pipelines · 2025-08-22T16:18:51Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-08-22T22:48:56Z

/azp run

azure-pipelines · 2025-08-22T22:49:09Z

Azure Pipelines successfully started running 1 pipeline(s).

DavidZagury · 2025-09-08T13:57:47Z

tests/high_frequency_telemetry/test_high_frequency_telemetry.py

+        pytest_assert(
+            validation_results['total_counters'] >= min_expected_counters,
+            f"Expected at least {min_expected_counters} counters, "
+            f"got {actual_counters}"


@Pterosaur actual_counters was not assigned on this point

Fixed by 603e284

DavidZagury · 2025-09-08T14:10:28Z

tests/high_frequency_telemetry/utilities.py

+        counter_values.append(counter_value)
+        pytest_assert(
+            counter_value > min_counter_value,
+            f"Counter value {counter_value} should be greater "


The tests assumes that there should always be values greater then zero on all counters.
Can we assume that? If (for example) no drop happened on the system, and the tests does not force in any way a drop to happen, we can't guarantee that IF_IN_DISCARDS will not have zero as the correct value.

Thanks for your point, I will fix it.

DavidZagury · 2025-09-09T08:01:27Z

tests/high_frequency_telemetry/utilities.py

+def run_countersyncd_and_capture_output(duthost, timeout=120):
+    """
+    Run countersyncd command and capture output.
+
+    Args:
+        duthost: DUT host object
+        timeout: Timeout in seconds (default: 120)
+
+    Returns:
+        dict: Command result with stdout, stderr, rc
+    """
+    countersyncd_cmd = (
+        f'timeout {timeout} docker exec swss countersyncd -e '
+        f'--max-stats-per-report 0 '
+    )


Suggested change

def run_countersyncd_and_capture_output(duthost, timeout=120):

"""

Run countersyncd command and capture output.

Args:

duthost: DUT host object

timeout: Timeout in seconds (default: 120)

Returns:

dict: Command result with stdout, stderr, rc

"""

countersyncd_cmd = (

f'timeout {timeout} docker exec swss countersyncd -e '

f'--max-stats-per-report 0 '

)

def run_countersyncd_and_capture_output(duthost, timeout=120, stats_interval=60):

...

f'--max-stats-per-report 0 '

f'--stats-interval {stats_interval}'

)

I suggest adding the option the override the default stats-interval, from my check this is needed for the longer poll interval tests

DavidZagury · 2025-09-09T08:10:09Z

tests/high_frequency_telemetry/test_high_frequency_telemetry.py

+        timeout = max(120, int(10 / expected_msg_per_sec) + 60) if expected_msg_per_sec > 0 else 180
+        logger.info(f"Running countersyncd for {timeout} seconds to capture stable measurements")
+
+        result = run_countersyncd_and_capture_output(duthost, timeout=timeout)


As we discussed on Teams, on slower poll intervals you also need to increase the reporting interval to get the correct Msg/s

mssonicbld · 2025-09-18T09:45:10Z

/azp run

azure-pipelines · 2025-09-18T09:45:25Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-09-18T11:32:47Z

/azp run

azure-pipelines · 2025-09-18T11:33:01Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-09-18T11:49:52Z

/azp run

azure-pipelines · 2025-09-18T11:50:07Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-09-19T02:13:58Z

/azp run

azure-pipelines · 2025-09-19T02:14:11Z

Azure Pipelines successfully started running 1 pipeline(s).

tests/high_frequency_telemetry/utilities.py

DavidZagury · 2025-10-21T12:15:49Z

Hi @Pterosaur,
I think we need to rework on how the tests verify the number of MSG per second capture by the tool.

I see an issue with the tests Msg/s verification
For example, this is the failure from one run of test_hft_port_counters

Failed: Average Msg/s 77.97 is outside expected range: 85.00 - 115.00.
Individual values: [0.0, 0.0, 233.9, 233.9, 0.0, 0.0]

The problem is that the test configures SONiC to poll counters every 10ms (expecting 100 Msg/s), but HFT batches approximately 2339 messages before sending each report.

If HFT polls correctly at 10ms intervals but batches 2339 messages per report:

Time to accumulate: 2339 messages × 10ms = ~23.39 seconds of polling
Transmission burst: Sending 2339 messages over ~10 seconds = 233.9 Msg/s

The observed pattern [0.0, 0.0, 233.9, 233.9, 0.0, 0.0] matches this behavior:

Reports during accumulation: 0.0 Msg/s (buffering, not sending)
Reports during transmission: 233.9 Msg/s (burst of 2339 messages)

The test averages these values (77.97 Msg/s) which doesn't account for the batching cycle. HFT may be polling at the correct 10ms frequency, but the validation logic expects continuous streaming rather than batch-and-send behavior.

@weiguo-nvidia

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-11-11T11:54:00Z

/azp run

azure-pipelines · 2025-11-11T11:54:15Z

Azure Pipelines successfully started running 1 pipeline(s).

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Priyansh Tratiya <[email protected]>

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Andoni Sanguesa <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: Lakshmi Yarramaneni <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: Yael Tzur <[email protected]>

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Yael Tzur <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]>

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: nnelluri-cisco <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: Raghavendran Ramanathan <[email protected]>

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Raghavendran Ramanathan <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: Raghavendran Ramanathan <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: Zhuohui Tan <[email protected]>

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Zhuohui Tan <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: mssonicbld <[email protected]>

mssonicbld · 2026-03-06T16:17:07Z

Cherry-pick PR to 202511: #22779

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Priyansh Tratiya <[email protected]>

…#23008) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: #20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Priyansh Tratiya <[email protected]> Co-authored-by: Ze Gan <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: Abhishek <[email protected]>

…et#21482) What is the motivation for this PR? Wrong condition for skipping HFT test How did you do it? Correct it as described in this comment: sonic-net#20379 (comment) Signed-off-by: Ze Gan <[email protected]> Signed-off-by: Abhishek <[email protected]>

What is the motivation for this PR? Add tests for high frequency telemetry How did you do it? Add new test cases How did you verify/test it? Check in the sn5600 platform locally Signed-off-by: mssonicbld <[email protected]> Co-authored-by: Ze Gan <[email protected]>

Add tests for high frequency telemetry

c149a2d

Signed-off-by: Ze Gan <[email protected]>

Skip test

4b9b22c

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hft_test branch from 3c7bc98 to 4b9b22c Compare August 22, 2025 22:48

DavidZagury requested changes Sep 8, 2025

View reviewed changes

DavidZagury reviewed Sep 9, 2025

View reviewed changes

Pterosaur force-pushed the hft_test branch from 6ac07dd to cebf942 Compare September 18, 2025 11:32

Fix comments

603e284

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hft_test branch from cebf942 to 603e284 Compare September 18, 2025 11:49

Add dummy table in golden image

ed84ddf

Signed-off-by: Ze Gan <[email protected]>

Pterosaur requested review from wangxin and yxieca as code owners September 19, 2025 02:13

DavidZagury reviewed Oct 6, 2025

View reviewed changes

tests/high_frequency_telemetry/utilities.py Show resolved Hide resolved

Fix comment: Increase timeout and fix time format

355edf3

Signed-off-by: Ze Gan <[email protected]>

Merge branch 'master' into hft_test

4af3d7a

Pterosaur added Request for 202511 branch Request to backport a change to 202511 branch and removed Cherry Pick Conflict_msft-202412 labels Mar 6, 2026

vmittal-msft added the Approved for 202511 branch label Mar 6, 2026

mssonicbld added the Created PR to 202511 branch label Mar 6, 2026

mssonicbld mentioned this pull request Mar 6, 2026

[action] [PR:20379] [tests]: Add tests for high frequency telemetry #22779

Merged

11 tasks

PriyanshTratiya mentioned this pull request Mar 16, 2026

[202511][cherry][hft]: Fix conditional mark in high frequency telemetry test (#21482) #23008

Merged

12 tasks

mssonicbld added Included in 202511 branch and removed Created PR to 202511 branch labels Mar 23, 2026

Conversation

Pterosaur commented Aug 22, 2025

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Aug 22, 2025

Uh oh!

azure-pipelines bot commented Aug 22, 2025

Uh oh!

mssonicbld commented Aug 22, 2025

Uh oh!

azure-pipelines bot commented Aug 22, 2025

Uh oh!

DavidZagury Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

DavidZagury Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

DavidZagury Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Pterosaur Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

DavidZagury Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

DavidZagury Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Sep 18, 2025

Uh oh!

azure-pipelines bot commented Sep 18, 2025

Uh oh!

mssonicbld commented Sep 18, 2025

Uh oh!

azure-pipelines bot commented Sep 18, 2025

Uh oh!

mssonicbld commented Sep 18, 2025

Uh oh!

azure-pipelines bot commented Sep 18, 2025

Uh oh!

mssonicbld commented Sep 19, 2025

Uh oh!

azure-pipelines bot commented Sep 19, 2025

Uh oh!

Uh oh!

DavidZagury commented Oct 21, 2025

Uh oh!

mssonicbld commented Nov 11, 2025

Uh oh!

azure-pipelines bot commented Nov 11, 2025

Uh oh!

mssonicbld commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants