[doc]: High frequency telemetry HLD by Pterosaur · Pull Request #1795 · sonic-net/SONiC

Pterosaur · 2024-09-01T10:15:34Z

High frequency telemetry HLD initial version

PRs:
sonic-net/sonic-swss-common#1006
sonic-net/sonic-sairedis#1573

doc/stream-telemetry/stream-telemetry-hld.md

r12f · 2024-09-03T07:09:10Z

doc/stream-telemetry/stream-telemetry-hld.md

same here. these "guidance" words are better to be removed. very confusing to read.

Sorry, Yes, there are many template "guidance" words I haven't removed them.

r12f · 2024-09-03T07:12:02Z

doc/stream-telemetry/stream-telemetry-hld.md

i know the numbered prefix are not there in the template, but the doc will be much more readable with it. we should consider add them here.

Hi @zhangyanzhao , Do you have any suggestions or concerns about adding a number prefix to each title in the template?

Hi @zhangyanzhao , Do you have any suggestions or concerns about adding a number prefix to each title in the template?

Good suggestion. Updated the template. Github markdown does not support auto number yet, need community member to update it manually when creating HLD doc with the template.

doc/stream-telemetry/stream-telemetry-hld.md

r12f · 2024-09-03T07:20:53Z

doc/stream-telemetry/stream-telemetry-hld.md

are we still uses the chunk size concept in our API?

Yes, we have.

Or leverage the existing solution, time-based reporting?

Signed-off-by: Ze Gan <[email protected]>

marian-pritsak · 2024-09-16T17:00:51Z

doc/stream-telemetry/stream-telemetry-hld.md

+
+- For high-frequency counters, the native IPFIX timestamp unit of seconds is insufficient. Therefore, we introduce an additional element, `observationTimeNanoseconds`, for each record to meet our requirements.
+- The enterprise bit is always set to 1 for stats records.
+- The element ID of IPFIX is derived from the object index. For example, for `Ethernet5`, the element ID will be `0x5 | 0x8000 = 0x8005`.


Having a variable element ID per SAI object will blow up the number of templates in the report (separate template ID per port, queue etc.).

In my mind, I assume all stats(per port, queue etc) in one profile will be encoded into ONE template. I will generate a unique template ID for a profile.

marian-pritsak · 2024-09-16T19:29:52Z

doc/stream-telemetry/stream-telemetry-hld.md

+
+### Data format
+
+We will use IPFIX as the report format, with all numbers in the IPFIX message in network-order (Big-endian).


You cannot make the assumption that data will be Big- or Little-endian. If it is collected by the ASIC and DMAd to a ring buffer, then kernel driver would need to correct the endianness of each data set. We need to have a bit in the element ID or Enterprise number that would indicate endianness.

Are you worried that endian conversion will consume too many CPU clocks?

marian-pritsak · 2024-09-16T20:38:02Z

doc/stream-telemetry/stream-telemetry-hld.md

+```
+STREAM_TELEMETRY_GROUP|{{profile_name}}|{{group_name}}
+    "object_names": {{list of object name}}
+    "object_counters": {{list of stats of object}}


How do you plan to handle the update of the counter list or object list from the report POV? is it going to be a new template ID?

Yes, I assume we will use a new template ID.
We treat any configuration update as a new profile. The previous session will be interrupted. The SAI report object will get a new SAI_TAM_REPORT_ATTR_REPORT_IPFIX_TEMPLATE_ID

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-03-22T11:45:36Z

/azp run

azure-pipelines · 2025-03-22T11:45:42Z

No pipelines are associated with this pull request.

r12f · 2025-03-27T01:24:08Z

doc/high-frequency-telemetry/high-frequency-telemetry-hld.md

+key                = HIGH_FREQUENCY_TELEMETRY_PROFILE|profile_name a string as the identifier of high frequency telemetry
+; field            = value
+stream_state       = enabled/disabled ; Enabled/Disabled stream.
+poll_interval      = uint32 ; The interval to poll counter, unit microseconds.


hi Ze, should we add a remote write endpoint in this profile?

What kind of endpoint? for OpenTelemetry collector or outside service like InflexDB?

Done， please check it

Signed-off-by: Ze Gan <[email protected]>

mssonicbld · 2025-03-27T03:01:06Z

/azp run

azure-pipelines · 2025-03-27T03:01:13Z

No pipelines are associated with this pull request.

bendrapubalareddy · 2025-04-08T19:30:58Z

doc/high-frequency-telemetry/high-frequency-telemetry-hld.md

+
+    syncd ->> dma_engine: Config stats
+    syncd ->> hft_orch: Config was applied in the ASIC
+    syncd ->> hft_orch: Query IPFIX template


By this time DMA/Netlink module already know the IPFIX template to use for streaming of configured stats ? After syncd query the IPFIX template, what does it do with it? It's not clear from this flow or document on this. Please elaborate the flow sequence on how IPFIX template is programmed to DMA Engine/Netlink kernel modules?

Hi @bendrapubalareddy , the dma engine and netlink module will both be provided by the vendor, we don't know when the IPFix template is programmed to the DMA Engine/Netlink kernel modules. So, we have a callback from SAI to notify the template is ready in SAI.
Here is its state machine, hope it can answer your question:
https://github.com/opencomputeproject/SAI/blob/master/doc/TAM/SAI-Proposal-TAM-stream-telemetry.md#tam-telemetry-type-state-machine

bendrapubalareddy · 2025-04-08T20:46:25Z

doc/high-frequency-telemetry/high-frequency-telemetry-hld.md

+``` shell
+
+# Add a new profile
+sudo config high_frequency_telemetry profile add $profile_name --stream_state=$stream_state --poll_interval=$poll_interval --chunk_size=$chunk_size --chunk_count=$chunk_count


"otel_endpoint": {{string of endpoint}} (Optional)
"otel_certs": {{string of path}} (Optional)
should these optional attributes be added to high_frequency_telemetry profile config or they should be part of another config?

Thanks for your point, we shuold support this new options in CLI.

bendrapubalareddy · 2025-04-08T22:35:48Z

doc/high-frequency-telemetry/high-frequency-telemetry-hld.md

+            loop collect a chunk of stats
+                dma_engine ->> asic: query stats from asic
+                asic --) dma_engine: stats
+                dma_engine ->> netlink_module: Push stats


As per the "title: High frequency telemetry architecture" flow, dma_engine shall push the stats in IPFIX format to netlink module. If so, should we change "dma_engine ->> netlink_module: Push stats" to "dma_engine ->> netlink_module: Push stats in IPFIX format" ?

Good suggestion. will fix it.

…rmat in stats push and adding new parameters for profile configuration

mssonicbld · 2025-04-13T15:28:52Z

/azp run

azure-pipelines · 2025-04-13T15:28:59Z

No pipelines are associated with this pull request.

sharad-brcm · 2025-04-14T16:42:13Z

doc/high-frequency-telemetry/high-frequency-telemetry-hld.md

+128-159: "Enterprise Number = 0x00010004"
+160-160: "1"
+161-175: "Element ID = 5 (port index)"
+176-191: "Field Length = 8 bytes"


Are the length is always expected to be 8 bytes only or different lengths other than 8 bytes are also accepted?

It's always expected to be 8 bytes. Because we want to follow the design of SAI. The type of stats in SAI is uint64.

sharad-brcm · 2025-04-14T16:53:05Z

doc/high-frequency-telemetry/high-frequency-telemetry-hld.md

+
+### 7.4. Config DB
+
+Any configuration changes in the config DB will interrupt the existing session and initiate a new one.


What is meant by 'interrupt' here? We are assuming the existing session is deleted and a new session is created.

From the perspective of a collector, a session is a set of groups with the same profile.
In the current SONiC implementation, we don't need to support changing the collector's session on the fly. But in the future, we would like to be able to add/remove or change the interval without interrupting any stats collection.

bandaru-viswanath

This document is generally light on SONIC-SAI workflow. Can you add some text describing the same ? Specifically -

Are the counter capabilities retrieved and used ? What are the expectations ?
If a counter reports as not supporting clear-on-read how does SONIC clear the counters ?
How does SONIC handle the wrap-around conditions ?
how does SONIC deal with the cases where HostCPU is too slow to empty the circular buffer ?
If IPFIX template supplies a length (other than 8) for the counter, how is it handled ?
How does SONIC use the SAI_TAM_TEL_TYPE_ATTR_MODE ?
General overview of the design for SAI_SWITCH_ATTR_TAM_TEL_TYPE_CONFIG_CHANGE_NOTIFY ?
How are the config and data threads kept synchronised ?

bendrapubalareddy · 2025-04-16T21:03:59Z

@Pterosaur If the counters are to be streamed from PIE port or front panel port of the ASIC directly, without sending to control plane sonic applications, that would be different config and this HLD does NOT want to have that option in it's config model. Is that understanding correct?

Pterosaur · 2025-04-22T14:14:50Z

@Pterosaur If the counters are to be streamed from PIE port or front panel port of the ASIC directly, without sending to control plane sonic applications, that would be different config and this HLD does NOT want to have that option in it's config model. Is that understanding correct?

@bendrapubalareddy We don't cover this scenario in this HLD, at least not in this version of HLD.

Pterosaur · 2025-04-22T14:41:00Z

This document is generally light on SONIC-SAI workflow. Can you add some text describing the same ? Specifically -

Are the counter capabilities retrieved and used ? What are the expectations ?

Yes, we have query_stats_st_capability API in SAI, if the capability isn't supported, we will raise an error from syslog.

If a counter reports as not supporting clear-on-read how does SONIC clear the counters ?

It's a typical problem of SONiC. According to the current SONiC solution, we will store the current snapshot of stats when we call clear action from SONiC side. And the UI will show the difference value of the stats.

How does SONIC handle the wrap-around conditions ?

Since we are using the 8bit(Uint64) for the counter, I believe that most of the time, it won't happen.

how does SONIC deal with the cases where HostCPU is too slow to empty the circular buffer ?

Drop the new stats. That's why we propose the ring buffer as the data structure.
We also provide an API, query_stats_st_capability, to query the minimal polling interval. https://github.com/opencomputeproject/SAI/blob/a56c38ed33aab0f81ec1c1e9f716ff722dc94f96/inc/saiobject.h#L323

If IPFIX template supplies a length (other than 8) for the counter, how is it handled ?

It's always expected to be 8 bytes. Because we want to follow the design of SAI. The type of stats in SAI is uint64.

How does SONIC use the SAI_TAM_TEL_TYPE_ATTR_MODE ?

SONiC prefers SAI_TAM_TEL_TYPE_MODE_SINGLE_TYPE. We may use SAI_TAM_TEL_TYPE_MODE_MIXED_TYPE in the future for better efficiency. But due to the limitation of some vendors, we prefer SINGLE TYPE.

General overview of the design for SAI_SWITCH_ATTR_TAM_TEL_TYPE_CONFIG_CHANGE_NOTIFY ?

We have a state machine: https://github.com/opencomputeproject/SAI/blob/master/doc/TAM/SAI-Proposal-TAM-stream-telemetry.md#tam-telemetry-type-state-machine

How are the config and data threads kept synchronised ?

In the phase 1, if we want to update the config, we have to delete the existing stream and create a new stream, this should be guarantee by SONiC side(orchagent) . And we leverage the tam telemetry type state machine to submit the config into the SAI level.

@bandaru-viswanath

r12f · 2025-06-04T00:19:02Z

Sync'ed with Bandaru offline and got his approval via email before the merge as well.

Pterosaur force-pushed the hfc branch 3 times, most recently from bc0e247 to c14e643 Compare September 3, 2024 06:38

r12f reviewed Sep 3, 2024

View reviewed changes

init doc

7c8beea

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hfc branch from c14e643 to 7c8beea Compare September 3, 2024 11:00

Fix length

3303c56

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hfc branch 8 times, most recently from 290f7f9 to 0604b27 Compare September 9, 2024 12:33

Pterosaur requested a review from r12f September 9, 2024 12:33

Pterosaur marked this pull request as ready for review September 9, 2024 12:33

Pterosaur requested a review from lguohan September 9, 2024 12:34

Fix comments

5ac1481

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hfc branch from 0604b27 to 5ac1481 Compare September 9, 2024 12:41

Remove subscription index

b60811d

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hfc branch from 23f03d2 to 754fca7 Compare September 13, 2024 12:30

Refactor STREAM_TELEMETRY_GROUP

f092315

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hfc branch from 754fca7 to f092315 Compare September 13, 2024 12:31

marian-pritsak requested changes Sep 16, 2024

View reviewed changes

Refactor SAI for hostif trap

6c2888e

Signed-off-by: Ze Gan <[email protected]>

Pterosaur force-pushed the hfc branch from dd4f0f0 to 6c2888e Compare September 19, 2024 12:54

remove useless content

748d68b

Signed-off-by: Ze Gan <[email protected]>

rck-innovium approved these changes Mar 20, 2025

View reviewed changes

Fix typo in SVG

9eb7693

r12f reviewed Mar 27, 2025

View reviewed changes

Add otel configuration in HFT profile

3c104cc

Signed-off-by: Ze Gan <[email protected]>

zhangyanzhao moved this from 📋 In Plan Features to 🏗 In Progress in SONiC 202505 Release Mar 27, 2025

zhangyanzhao requested a review from liat-grozovik March 31, 2025 23:18

bendrapubalareddy reviewed Apr 8, 2025

View reviewed changes

Enhance high frequency telemetry documentation by specifying IPFIX fo…

861d0b4

…rmat in stats push and adding new parameters for profile configuration

sharad-brcm reviewed Apr 14, 2025

View reviewed changes

bandaru-viswanath suggested changes Apr 14, 2025

View reviewed changes

marian-pritsak approved these changes Apr 16, 2025

View reviewed changes

anilpannala moved this from 🏗 In Progress to MovedToBacklog in SONiC 202505 Release May 30, 2025

r12f approved these changes Jun 3, 2025

View reviewed changes

JaiOCP approved these changes Jun 3, 2025

View reviewed changes

yxieca merged commit 0965546 into sonic-net:master Jun 3, 2025
1 check passed

zhangyanzhao moved this from MovedToBacklog to ✅ Done in SONiC 202505 Release Jun 7, 2025

zhangyanzhao moved this from ✅ Done to MovedToBacklog in SONiC 202505 Release Jun 7, 2025

zhangyanzhao moved this from MovedToBacklog to ✅ Done in SONiC 202505 Release Jun 7, 2025


		### Data format

		We will use IPFIX as the report format, with all numbers in the IPFIX message in network-order (Big-endian).


		### 7.4. Config DB

		Any configuration changes in the config DB will interrupt the existing session and initiate a new one.

Conversation

Pterosaur commented Sep 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marian-pritsak Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Mar 22, 2025

Uh oh!

azure-pipelines bot commented Mar 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Mar 27, 2025

Uh oh!

azure-pipelines bot commented Mar 27, 2025

Uh oh!

bendrapubalareddy Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Apr 13, 2025

Uh oh!

azure-pipelines bot commented Apr 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bandaru-viswanath left a comment

Choose a reason for hiding this comment

Uh oh!

bendrapubalareddy commented Apr 16, 2025

Pterosaur commented Sep 1, 2024 •

edited

Loading

marian-pritsak Sep 16, 2024 •

edited

Loading

bendrapubalareddy Apr 8, 2025 •

edited

Loading

Pterosaur commented Apr 22, 2025 •

edited

Loading

Pterosaur commented Apr 22, 2025 •

edited

Loading

r12f commented Jun 4, 2025 •

edited

Loading