Skip to content

Update Interface-Link-bring-up-sequence.md #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shyam77git merged 5 commits into from
Jan 22, 2022
Merged

Update Interface-Link-bring-up-sequence.md #2

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4b56c6d
Update Interface-Link-bring-up-sequence.md
jaganbal-a Jan 21, 2022
aea144b
Update Interface-Link-bring-up-sequence.md
jaganbal-a Jan 21, 2022
d5666ca
Update Interface-Link-bring-up-sequence.md
jaganbal-a Jan 21, 2022
57cee58
Update Interface-Link-bring-up-sequence.md
jaganbal-a Jan 21, 2022
01950be
Update Interface-Link-bring-up-sequence.md
jaganbal-a Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 18 additions & 17 deletions doc/sfp-cmis/Interface-Link-bring-up-sequence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Deterministic Approach for Interface Link bring-up sequence

# High Level Design Document
#### Rev 0.2
#### Rev 0.3

# Table of Contents
* [List of Tables](#list-of-tables)
Expand All @@ -25,7 +25,7 @@ Deterministic Approach for Interface Link bring-up sequence
|:---:|:-----------:|:----------------------------------:|------------------------------|
| 0.1 | 08/16/2021 | Shyam Kumar | Initial version
| 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows
| 0.3 | 01/19/2022 | Shyam Kumar | Addressed review-comments |
| 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments |


# About this Manual
Expand Down Expand Up @@ -56,22 +56,22 @@ Interface link bring-up sequence and workflows for use-cases around it

# Problem Definition

1. Presently in SONiC, there is no synchronization between 400G Datapath Init operation and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios.
1. Presently in SONiC, there is no synchronization between Datapath Init operation of CMIS complaint optical module and enabling ASIC (NPU/PHY) Tx which may cause link instability during administrative interface enable “config interface startup Ethernet” configuration and bootup scenarios.

For 400G optics module, the Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state.
For CMIS-compliant active (optical) modules, the Host (NPU/PHY) needs to provide a valid high-speed Tx input signal at the required signaling rate and encoding type prior to causing a DPSM to exit from DPDeactivated state and to move to DP Init transient state.

Fundamentally it means - have a deterministic approach to bring-up the interface.

Also, this problem is mentioned ‘as outside-the-scope’ of ‘CMIS Application Initialization’ high-level design document
**(https://github.com/ds952811/SONiC/blob/0e4516d7bf707a36127438c7f2fa9cc2b504298e/doc/sfp-cmis/cmis-init.md#outside-the-scope)**

2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the optics laser for 100G/400G.
2. During administrative interface disable “config interface shutdown Ethernet”, only the ASIC(NPU) Tx is disabled and not the opticcal module Tx/laser.
This will lead to power wastage and un-necessary fan power consumption to keep the module temperature in operating range

# Background

Per the ‘QSFPDD spec’, ‘validation, diagnostics’ done by HW team' and 'agreement with vendors',
need to follow following bring-up seq to enable port/interface with 400G optics in LC/chassis:
Per the ‘CMIS spec’, ‘validation, diagnostics’ done by HW team' and 'agreement with vendors',
need to follow following bring-up seq to enable port/interface with CMIS compliant optical modules in LC/chassis:

a) Enable port on NPU (bring-up port, serdes on the NPU ; enable signals) : syncd
b) Enable port on PHY (bring-up port, serdes on the PHY ; enable signals) : gbsyncd
Expand All @@ -80,14 +80,14 @@ Interface link bring-up sequence and workflows for use-cases around it

In boards not having PHY, #b) not needed but #a) and #c) sequence to be followed.

## Clause from QSFP-DD (CMIS4.0 spec)
## Clause from CMIS4.0 spec

Excerpt from CMIS4.0 spec providing detailed reasoning for the above-mentioned bring-up sequence

![61f5b485-cf3b-4ca8-beac-9102b6feabfe](https://user-images.githubusercontent.com/69485234/147173702-f124fc9d-ef27-4816-b1a1-b4a44a5833a7.PNG)


## Clause from QSFP-DD (CMIS5.0 spec)
## Clause from CMIS5.0 spec

Excerpt from CMIS5.0 spec providing detailed reasoning for the above-mentioned bring-up sequence

Expand All @@ -96,26 +96,27 @@ Interface link bring-up sequence and workflows for use-cases around it

# Objective

Have a determistic approach for Interface link bring-up sequence i.e. below sequence to be followed:
Have a determistic approach for Interface link bring-up sequence for all interfaces types i.e. below sequence to be followed:
1. Initialize and enable NPU Tx and Rx path
2. For system with 'External' PHY: Initialize and enable PHY Tx and Rx on both line and host sides; ensure host side link is up
3. Then only perform optics data path initialization/activation/Tx enable (for CMIS complaint optical modules) and Tx enable (for SFF complaint optical modules)
3. Then only perform optics data path initialization/activation/Tx enable (for CMIS complaint optical modules) and Tx enable (for SFF complaint optical modules)

# Proposal

Recommend following this high-level work-flow sequence to accomplish the Objective:
- xcvrd to subscribe to a new field “host_tx_ready” in port table state-DB
- Orchagent will set the “host_tx_ready” to true/false based on the SET_ADMIN_STATE attribute return status to syncd/gbsyncd. (As part of SET_ADMIN_STATE attribute enable, the NPU Tx is enabled)
- xcvrd process the “host_tx_ready” value change event and do optics datapath init / de-init using CMIS API
- Recommendation is to follow this proposal for all the known interfaces - 400G/100G/40G/25G/10G. Reason being:
- 400G - as mentioned above the CMIS spec to be followed
- 100G/40G/25G/10G -
- Recommendation is to follow this proposal for all the known interfaces types- 400G/100G/40G/25G/10G. Reason being:
- CMIS complaint optical modules:-
All CMIS complaint optical modules will follow this approach as recommended in the CMIS spec.
- SFF complaint optical modules:-
- deterministic approach to bring the interface will eliminate any link stability issue which will be difficult to chase in the production network
e.g. If there is a PHY device in between, and this 'deterministic approach' is not followed, PHY may adapt to a bad signal or interface flaps may occur when the optics tx/rx enabled during PHY initialization.
- there is a possibility of interface link flaps with non-quiescent optical modules <QSFP+/SFP28/SFP+> if this 'deterministic approach' is not followed
- It helps bring down the optical module laser when interface is adminstiratively shutdown. Per the workflow here, this is acheived by xcvrd listening to host_tx_ready field from PORT_TABLE of STATE_DB. Turning the laser off would reduce the power consumption and avoid any lab hazard
- Additionally provides uniform workflow (from SONiC NOS) across all interface types instead of just 400G
- This synchronization will also benefit native 10G SFPs interfaces as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx)
- Additionally provides uniform workflow (from SONiC NOS) across all interface types with or without module presence.
- This synchronization will also benefit SFP+ optical modules as they are "plug N play" and may not have quiescent functionality. (xcvrd can use the optional 'soft tx disable' ctrl reg to disable the tx)

# Proposed Work-Flows
Please refer to the flow/sequence diagrams which covers the following required use-cases
Expand All @@ -142,7 +143,7 @@ Please refer to the flow/sequence diagrams which covers the following required
Following items are not in the scope of this document. They would be taken up separately
1. xcvrd restart
- If the xcvrd goes for restart, then all the DB events will be replayed.
Here the Datapath init/activate (for 400G), tx-disable register set (for 100G), will be a no-op if the optics is already in that state
Here the Datapath init/activate for CMIS compliant optical modules, tx-disable register set (for SFF complaint optical modules), will be a no-op if the optics is already in that state
2. syncd/gbsyncd/swss docker container restart
- Cleanup scenario - the host_tx_ready field in STATE-DB should be updated to “False” to respective ports that a PHY/NPU interface with
3. CMIS API feature is not part of this design and the APIs will be used in this design. For CMIS HLD, Please refer to:
Expand Down