Fix DPU restart message drop by Zmq lazy bind. by liuh-80 · Pull Request #3837 · sonic-net/sonic-swss

liuh-80 · 2025-08-19T01:57:27Z

Fix DPU restart message drop by Zmq lazy bind.

Why I did it

Fix issue:
sonic-net/sonic-buildimage#23110

When creating a ZmqServer followed by a ZmqProducerStateTable, there may be a time gap between the server starting to receive messages and the producer state table registering its handler.

This gap can lead to dropped messages. To avoid this, use lazy binding and invoke bind() only after the handler is registered.

How I did it

Update Orchagent to use lazy binding for ZMQ.
ZMQ is now created with lazy bind in Orchagent, and the bind() operation is deferred until all ZmqProducerStateTable instances have been initialized. This ensures handlers are registered before any messages are received, preventing potential data loss during startup.

This PR depends on sonic-net/sonic-swss-common#1068

Work item tracking

Microsoft ADO: 33995986

How to verify it

Pass all test cases.

Which release branch to backport (provide reason below if selected)

Description for the changelog

Fix DPU restart message drop by Zmq lazy bind.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

mssonicbld · 2025-08-19T01:57:35Z

/azp run

azure-pipelines · 2025-08-19T01:57:45Z

Azure Pipelines successfully started running 1 pipeline(s).

liuh-80 · 2025-08-21T01:35:59Z

swss-common PR merged
Will build again when new artifact ready.

liuh-80 · 2025-08-21T01:36:14Z

/azpw run Azure.sonic-swss

mssonicbld · 2025-08-21T01:36:16Z

/AzurePipelines run Azure.sonic-swss

azure-pipelines · 2025-08-21T01:36:26Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-21T01:51:55Z

/azp run

azure-pipelines · 2025-08-21T01:52:05Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-22T02:46:29Z

/azp run

azure-pipelines · 2025-08-22T02:46:39Z

Azure Pipelines successfully started running 1 pipeline(s).

liuh-80 · 2025-08-25T02:19:05Z

/azpw run Azure.sonic-swss

mssonicbld · 2025-08-25T02:19:08Z

/AzurePipelines run Azure.sonic-swss

azure-pipelines · 2025-08-25T02:19:18Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-25T05:38:28Z

/azp run

azure-pipelines · 2025-08-25T05:38:38Z

Azure Pipelines successfully started running 1 pipeline(s).

prabhataravind

lgtm

Copilot

Pull Request Overview

This PR fixes message drops during DPU restart by implementing lazy binding for ZMQ servers. Instead of binding immediately when the ZmqServer is created, the binding is deferred until after message handlers are registered, preventing messages from being lost during the initialization gap.

Introduces lazy binding for ZmqServer instances by passing a true parameter to the constructor
Updates the main orchestration agent to call bind() after all handlers are registered
Modifies unit tests to explicitly call bind() after registering handlers

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
lib/orch_zmq_config.cpp	Enables lazy binding by passing true to ZmqServer constructor
orchagent/main.cpp	Adds explicit bind() call after handler registration with logging
tests/mock_tests/zmq_orch_ut.cpp	Updates unit test to call bind() after registering message handler

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Fix DPU restart message drop by Zmq lazy bind. Why I did it Fix issue: sonic-net/sonic-buildimage#23110 When creating a ZmqServer followed by a ZmqProducerStateTable, there may be a time gap between the server starting to receive messages and the producer state table registering its handler. This gap can lead to dropped messages. To avoid this, use lazy binding and invoke bind() only after the handler is registered. How I did it Update Orchagent to use lazy binding for ZMQ. ZMQ is now created with lazy bind in Orchagent, and the bind() operation is deferred until all ZmqProducerStateTable instances have been initialized. This ensures handlers are registered before any messages are received, preventing potential data loss during startup.

* upstream/master: [ssw][ha] set `SAI_HA_SCOPE_ATTR_ADMIN_STATE` (sonic-net#3841) Fix DPU restart message drop by Zmq lazy bind. (sonic-net#3837) [ssw][ha] consume new ha_scope fields (sonic-net#3825) Add PFC historical statistics estimation to the PFCWD Orch (sonic-net#3533)

Fix DPU restart message drop by Zmq lazy bind. Why I did it Fix issue: sonic-net/sonic-buildimage#23110 When creating a ZmqServer followed by a ZmqProducerStateTable, there may be a time gap between the server starting to receive messages and the producer state table registering its handler. This gap can lead to dropped messages. To avoid this, use lazy binding and invoke bind() only after the handler is registered. How I did it Update Orchagent to use lazy binding for ZMQ. ZMQ is now created with lazy bind in Orchagent, and the bind() operation is deferred until all ZmqProducerStateTable instances have been initialized. This ensures handlers are registered before any messages are received, preventing potential data loss during startup.

mssonicbld · 2025-08-30T17:40:22Z

Cherry-pick PR to msft-202506: Azure/sonic-swss.msft#144

Fix DPU restart message drop by Zmq lazy bind. Why I did it Fix issue: sonic-net/sonic-buildimage#23110 When creating a ZmqServer followed by a ZmqProducerStateTable, there may be a time gap between the server starting to receive messages and the producer state table registering its handler. This gap can lead to dropped messages. To avoid this, use lazy binding and invoke bind() only after the handler is registered. How I did it Update Orchagent to use lazy binding for ZMQ. ZMQ is now created with lazy bind in Orchagent, and the bind() operation is deferred until all ZmqProducerStateTable instances have been initialized. This ensures handlers are registered before any messages are received, preventing potential data loss during startup.

Fix DPU restart message drop by Zmq lazy bind. Why I did it Fix issue: sonic-net/sonic-buildimage#23110 When creating a ZmqServer followed by a ZmqProducerStateTable, there may be a time gap between the server starting to receive messages and the producer state table registering its handler. This gap can lead to dropped messages. To avoid this, use lazy binding and invoke bind() only after the handler is registered. How I did it Update Orchagent to use lazy binding for ZMQ. ZMQ is now created with lazy bind in Orchagent, and the bind() operation is deferred until all ZmqProducerStateTable instances have been initialized. This ensures handlers are registered before any messages are received, preventing potential data loss during startup. Signed-off-by: Lawrence Lee <lawlee@microsoft.com>

Fix DPU restart message drop by Zmq lazy bind. Why I did it Fix issue: sonic-net/sonic-buildimage#23110 When creating a ZmqServer followed by a ZmqProducerStateTable, there may be a time gap between the server starting to receive messages and the producer state table registering its handler. This gap can lead to dropped messages. To avoid this, use lazy binding and invoke bind() only after the handler is registered. How I did it Update Orchagent to use lazy binding for ZMQ. ZMQ is now created with lazy bind in Orchagent, and the bind() operation is deferred until all ZmqProducerStateTable instances have been initialized. This ensures handlers are registered before any messages are received, preventing potential data loss during startup. Signed-off-by: Baorong Liu <96146196+baorliu@users.noreply.github.com>

Fix DPU restart message drop by Zmq lazy bind

f0fb105

liuh-80 mentioned this pull request Aug 19, 2025

Bug: [Smartswitch] gnmi configuration fails for the first dash object when DPU is powered off and powered back on due to zmq connection sonic-net/sonic-buildimage#23110

Open

prsunny requested a review from prabhataravind August 20, 2025 22:34

Merge branch 'master' into dev/liuh/zmq_lazy_bind

519e9f5

Merge branch 'master' into dev/liuh/zmq_lazy_bind

e4aceb8

Update zmq_orch_ut.cpp

3ecb097

liuh-80 marked this pull request as ready for review August 25, 2025 08:19

liuh-80 requested a review from prsunny as a code owner August 25, 2025 08:19

prabhataravind approved these changes Aug 26, 2025

View reviewed changes

prabhataravind requested a review from Copilot August 26, 2025 03:40

Copilot AI reviewed Aug 26, 2025

View reviewed changes

prsunny approved these changes Aug 26, 2025

View reviewed changes

prsunny added the Request for msft-202506 Branch label Aug 26, 2025

prsunny merged commit f9bf770 into sonic-net:master Aug 26, 2025
15 checks passed

prabhataravind added the Approved for msft-202506 Branch label Aug 30, 2025

mssonicbld mentioned this pull request Aug 30, 2025

[action] [PR:3837] Fix DPU restart message drop by Zmq lazy bind. Azure/sonic-swss.msft#144

Merged

6 tasks

mssonicbld added the Created PR to msft-202506 Branch label Aug 30, 2025

mssonicbld added Included in msft-202506 Branch and removed Created PR to msft-202506 Branch labels Aug 30, 2025

Conversation

liuh-80 commented Aug 19, 2025

Why I did it

How I did it

Work item tracking

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

mssonicbld commented Aug 19, 2025

Uh oh!

azure-pipelines bot commented Aug 19, 2025

Uh oh!

liuh-80 commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liuh-80 commented Aug 21, 2025

Uh oh!

mssonicbld commented Aug 21, 2025

Uh oh!

azure-pipelines bot commented Aug 21, 2025

Uh oh!

mssonicbld commented Aug 21, 2025

Uh oh!

azure-pipelines bot commented Aug 21, 2025

Uh oh!

mssonicbld commented Aug 22, 2025

Uh oh!

azure-pipelines bot commented Aug 22, 2025

Uh oh!

liuh-80 commented Aug 25, 2025

Uh oh!

mssonicbld commented Aug 25, 2025

Uh oh!

azure-pipelines bot commented Aug 25, 2025

Uh oh!

mssonicbld commented Aug 25, 2025

Uh oh!

azure-pipelines bot commented Aug 25, 2025

Uh oh!

prabhataravind left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

mssonicbld commented Aug 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

liuh-80 commented Aug 21, 2025 •

edited

Loading