Skip to content

[vs]: Remove non-default objects on dynamic buffer model teardown#3931

Merged
dgsudharsan merged 1 commit intosonic-net:masterfrom
nazariig:master-dyn-buf-vs-fix
Oct 13, 2025
Merged

[vs]: Remove non-default objects on dynamic buffer model teardown#3931
dgsudharsan merged 1 commit intosonic-net:masterfrom
nazariig:master-dyn-buf-vs-fix

Conversation

@nazariig
Copy link
Copy Markdown
Collaborator

@nazariig nazariig commented Oct 10, 2025

Signed-off-by: Nazarii Hnydyn nazariig@nvidia.com

Packet Trimming VS tests sporadic failures are caused by a timing issue during dynamic buffer model teardown flow.
The temporary zero buffer profiles generated by a dynamic buffer manager are not passed the proper clean up flow when switching from dynamic to traditional buffer model.

PYTEST:

2025-10-08T17:11:37.8511767Z + echo 'Running the py test for tests: test_trimming.py, 1/3...'
2025-10-08T17:11:37.8512142Z + py.test -v --force-flaky --junitxml=test_trimming_tr.xml --enable-coverage --force-recreate-dvs --imgname=docker-sonic-vs:Azure.sonic-swss.20251008.32.asan-False test_trimming.py
2025-10-08T17:11:37.8512435Z Running the py test for tests: test_trimming.py, 1/3...
2025-10-08T17:11:38.2420875Z ============================= test session starts ==============================
2025-10-08T17:12:24.4469349Z test_trimming.py::TestTrimmingBasicFlows::test_TrimSwitchGlobalConfiguration[symmetric-dscp-dynamic-queue-index] PASSED [  4%]
2025-10-08T17:12:24.5903476Z test_trimming.py::TestTrimmingBasicFlows::test_TrimSwitchGlobalConfiguration[asymmetric-dscp-static-queue-index] PASSED [  7%]
2025-10-08T17:12:24.7354256Z test_trimming.py::TestTrimmingBasicFlows::test_TrimSwitchGlobalConfiguration[asymmetric-dscp-dynamic-queue-index] PASSED [  9%]
2025-10-08T17:12:24.9139801Z test_trimming.py::TestTrimmingAdvancedFlows::test_TrimAsymToSymMigration PASSED [ 11%]
2025-10-08T17:12:25.0759139Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[size-empty] PASSED [ 14%]
2025-10-08T17:12:25.1934883Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[size-min-1] PASSED [ 16%]
2025-10-08T17:12:25.3315506Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[size-max+1] PASSED [ 19%]
2025-10-08T17:12:25.4509343Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[dscp-empty] PASSED [ 21%]
2025-10-08T17:12:25.5724444Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[dscp-min-1] PASSED [ 23%]
2025-10-08T17:12:25.7315821Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[dscp-max+1] PASSED [ 26%]
2025-10-08T17:12:25.8769825Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[tc-empty] PASSED [ 28%]
2025-10-08T17:12:26.0347039Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[tc-min-1] PASSED [ 30%]
2025-10-08T17:12:26.1768350Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[tc-max+1] PASSED [ 33%]
2025-10-08T17:12:26.3097304Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[queue-empty] PASSED [ 35%]
2025-10-08T17:12:26.4504641Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[queue-min-1] PASSED [ 38%]
2025-10-08T17:12:26.5741821Z test_trimming.py::TestTrimmingNegativeFlows::test_TrimNegValueOutOfBound[queue-max+1] PASSED [ 40%]
2025-10-08T17:12:27.3660137Z test_trimming.py::TestTrimmingTraditionalBufferModel::test_TrimStaticBufferProfileConfiguration[drop-packet] PASSED [ 42%]
2025-10-08T17:12:27.5555043Z test_trimming.py::TestTrimmingTraditionalBufferModel::test_TrimStaticBufferProfileConfiguration[trim-packet] PASSED [ 45%]
2025-10-08T17:12:28.0137679Z test_trimming.py::TestTrimmingDynamicBufferModel::test_TrimDynamicBufferProfileConfiguration[drop-packet] PASSED [ 20%]
2025-10-08T17:13:48.0952848Z test_trimming.py::TestTrimmingDynamicBufferModel::test_TrimDynamicBufferProfileConfiguration[trim-packet] PASSED       [ 37%]
2025-10-08T17:16:07.9650315Z test_trimming.py::TestTrimmingNegativeTraditionalBufferModel::test_TrimNegStaticBufferProfileAttach[priority-group] PASSED        [ 33%]
2025-10-08T17:16:27.8808786Z test_trimming.py::TestTrimmingNegativeTraditionalBufferModel::test_TrimNegStaticBufferProfileAttach[ingress-buffer-profile-list] ERROR [ 54%]
2025-10-08T17:16:27.9370310Z test_trimming.py::TestTrimmingNegativeTraditionalBufferModel::test_TrimNegStaticBufferProfileAttach[egress-buffer-profile-list] ERROR [ 57%]
2025-10-08T17:16:28.0119881Z test_trimming.py::TestTrimmingNegativeTraditionalBufferModel::test_TrimNegStaticBufferProfileEdit[priority-group] ERROR [ 59%]
2025-10-08T17:16:28.0771963Z test_trimming.py::TestTrimmingNegativeTraditionalBufferModel::test_TrimNegStaticBufferProfileEdit[ingress-buffer-profile-list] ERROR [ 61%]
2025-10-08T17:16:45.7027093Z test_trimming.py::TestTrimmingNegativeTraditionalBufferModel::test_TrimNegStaticBufferProfileEdit[egress-buffer-profile-list] 
2025-10-08T17:17:08.1605774Z test_trimming.py::TestTrimmingNegativeDynamicBufferModel::test_TrimNegDynamicBufferProfileAttach[priority-group] PASSED        [ 50%]
2025-10-08T17:17:27.1276992Z test_trimming.py::TestTrimmingNegativeDynamicBufferModel::test_TrimNegDynamicBufferProfileAttach[ingress-buffer-profile-list] PASSED [ 69%]
2025-10-08T17:17:28.2084008Z test_trimming.py::TestTrimmingNegativeDynamicBufferModel::test_TrimNegDynamicBufferProfileAttach[egress-buffer-profile-list] PASSED [ 71%]
2025-10-08T17:17:29.3034144Z test_trimming.py::TestTrimmingNegativeDynamicBufferModel::test_TrimNegDynamicBufferProfileEdit[priority-group] PASSED [ 73%]
2025-10-08T17:17:30.3987494Z test_trimming.py::TestTrimmingNegativeDynamicBufferModel::test_TrimNegDynamicBufferProfileEdit[ingress-buffer-profile-list] PASSED [ 76%]
2025-10-08T17:17:44.0484787Z test_trimming.py::TestTrimmingNegativeDynamicBufferModel::test_TrimNegDynamicBufferProfileEdit[egress-buffer-profile-list] PASSED [ 78%]PASSED    [  6%]
2025-10-08T17:18:15.5843812Z test_trimming.py::TestTrimmingStats::test_TrimSwitchStats[tx-packet] PASSED [ 80%]
2025-10-08T17:18:16.6048058Z test_trimming.py::TestTrimmingStats::test_TrimSwitchStats[drop-packet] PASSED [ 83%]
2025-10-08T17:18:17.6729918Z test_trimming.py::TestTrimmingStats::test_TrimPortStats[trim-packet] PASSED [ 85%]
2025-10-08T17:18:18.6844302Z test_trimming.py::TestTrimmingStats::test_TrimPortStats[tx-packet] PASSED [ 88%]
2025-10-08T17:18:19.6940689Z test_trimming.py::TestTrimmingStats::test_TrimPortStats[drop-packet] PASSED [ 90%]
2025-10-08T17:18:20.7385643Z test_trimming.py::TestTrimmingStats::test_TrimQueueStats[trim-packet] PASSED [ 92%]
2025-10-08T17:18:21.7422199Z test_trimming.py::TestTrimmingStats::test_TrimQueueStats[tx-packet] PASSED [ 95%]
2025-10-08T17:18:22.5869810Z test_trimming.py::TestTrimmingStats::test_TrimQueueStats[drop-packet] PASSED                    [ 50%]
2025-10-08T17:18:44.6434501Z test_trimming.py::test_nonflaky_dummy PASSED
2025-10-08T17:19:30.3390787Z ==================================== ERRORS ====================================
2025-10-08T17:19:30.3391847Z _ ERROR at setup of TestTrimmingNegativeTraditionalBufferModel.test_TrimNegStaticBufferProfileAttach[priority-group] _
2025-10-08T17:19:30.3392342Z 
2025-10-08T17:19:30.3392700Z self = <test_trimming.TestTrimmingNegativeTraditionalBufferModel object at 0x718dce0a6080>
2025-10-08T17:19:30.3392898Z 
2025-10-08T17:19:30.3393164Z     @pytest.fixture(scope="class")
2025-10-08T17:19:30.3393467Z     def bufferData(self):
2025-10-08T17:19:30.3393782Z         trimlogger.info("Initialize buffer data")
2025-10-08T17:19:30.3394068Z     
2025-10-08T17:19:30.3394354Z         trimlogger.info("Verify buffer profiles are loaded")
2025-10-08T17:19:30.3394714Z >       self.dvs_buffer.wait_for_buffer_profiles()
2025-10-08T17:19:30.3394854Z 
2025-10-08T17:19:30.3395119Z test_trimming.py:885: 
2025-10-08T17:19:30.3395454Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2025-10-08T17:19:30.3395814Z dvslib/dvs_buffer.py:112: in wait_for_buffer_profiles
2025-10-08T17:19:30.3396361Z     self.asic_db.wait_for_n_keys(self.ASIC_BUFFER_PROFILE, len(bufferProfileList))
2025-10-08T17:19:30.3396766Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2025-10-08T17:19:30.3396915Z 
2025-10-08T17:19:30.3397270Z self = <dvslib.dvs_database.DVSDatabase object at 0x718dce0688b0>
2025-10-08T17:19:30.3398356Z table_name = 'ASIC_STATE:SAI_OBJECT_TYPE_BUFFER_PROFILE', num_keys = 3
2025-10-08T17:19:30.3398720Z wait_at_least_n_keys = False
2025-10-08T17:19:30.3398996Z polling_config = PollingConfig(polling_interval=0.01, timeout=20.0, strict=True)
2025-10-08T17:19:30.3399270Z failure_message = None
2025-10-08T17:19:30.3399382Z 
2025-10-08T17:19:30.3399586Z     def wait_for_n_keys(
2025-10-08T17:19:30.3399827Z         self,
2025-10-08T17:19:30.3400030Z         table_name: str,
2025-10-08T17:19:30.3400241Z         num_keys: int,
2025-10-08T17:19:30.3400501Z         wait_at_least_n_keys: bool = False,
2025-10-08T17:19:30.3400799Z         polling_config: PollingConfig = PollingConfig(),
2025-10-08T17:19:30.3401457Z         failure_message: str = None,
2025-10-08T17:19:30.3401699Z     ) -> List[str]:
2025-10-08T17:19:30.3401973Z         """Wait for the specified number of keys to exist in the table.
2025-10-08T17:19:30.3402259Z     
2025-10-08T17:19:30.3402479Z         Args:
2025-10-08T17:19:30.3402781Z             table_name: The name of the table from which to fetch the keys.
2025-10-08T17:19:30.3403162Z             num_keys: The expected number of keys to retrieve from the table.
2025-10-08T17:19:30.3403529Z             polling_config: The parameters to use to poll the db.
2025-10-08T17:19:30.3403956Z             failure_message: The message to print if the call times out. This will only take effect
2025-10-08T17:19:30.3404454Z                 if the PollingConfig is set to strict.
2025-10-08T17:19:30.3404717Z     
2025-10-08T17:19:30.3404944Z         Returns:
2025-10-08T17:19:30.3405278Z             The keys stored in the table. If no keys are found, then an empty List is returned.
2025-10-08T17:19:30.3405608Z         """
2025-10-08T17:19:30.3405821Z     
2025-10-08T17:19:30.3406179Z         def access_function():
2025-10-08T17:19:30.3406491Z             keys = self.get_keys(table_name)
2025-10-08T17:19:30.3406784Z             if wait_at_least_n_keys:
2025-10-08T17:19:30.3407101Z                 return (len(keys) >= num_keys, keys)
2025-10-08T17:19:30.3407383Z             else:
2025-10-08T17:19:30.3407669Z                 return (len(keys) == num_keys, keys)
2025-10-08T17:19:30.3407927Z     
2025-10-08T17:19:30.3408153Z         status, result = wait_for_result(
2025-10-08T17:19:30.3408432Z             access_function, self._disable_strict_polling(polling_config)
2025-10-08T17:19:30.3408686Z         )
2025-10-08T17:19:30.3408875Z     
2025-10-08T17:19:30.3409079Z         if not status:
2025-10-08T17:19:30.3409329Z             message = failure_message or (
2025-10-08T17:19:30.3409632Z                 f"Unexpected number of keys: expected={num_keys}, received={len(result)} "
2025-10-08T17:19:30.3409973Z                 f'({result}), table="{table_name}"'
2025-10-08T17:19:30.3410182Z             )
2025-10-08T17:19:30.3410400Z >           assert not polling_config.strict, message
2025-10-08T17:19:30.3410908Z E           AssertionError: Unexpected number of keys: expected=3, received=4 (('oid:0x1900000000061f', 'oid:0x190000000005f7', 'oid:0x190000000005f8', 'oid:0x190000000005f9')), table="ASIC_STATE:SAI_OBJECT_TYPE_BUFFER_PROFILE"
2025-10-08T17:19:30.3411304Z 
2025-10-08T17:19:30.3411578Z dvslib/dvs_database.py:424: AssertionError

SYSLOG:

Oct 10 08:19:51.183575 ea1963d840ac NOTICE #pytest: === start test test_trimming.py::TestTrimmingDynamicBufferModel::test_TrimDynamicBufferProfileConfiguration[drop-packet] ===
Oct 10 08:21:16.195568 ea1963d840ac NOTICE #pytest: === finish test test_trimming.py::TestTrimmingDynamicBufferModel::test_TrimDynamicBufferProfileConfiguration[drop-packet] ===
Oct 10 08:21:16.295516 ea1963d840ac NOTICE #pytest: === start test test_trimming.py::TestTrimmingDynamicBufferModel::test_TrimDynamicBufferProfileConfiguration[trim-packet] ===
Oct 10 08:21:16.307092 ea1963d840ac NOTICE #buffermgrd: :- handleBufferProfileTable: BUFFER_PROFILE egress_lossy_profile has been inserted into APPL_DB directly
Oct 10 08:22:07.178187 ea1963d840ac NOTICE #pytest: === finish test test_trimming.py::TestTrimmingDynamicBufferModel::test_TrimDynamicBufferProfileConfiguration[trim-packet] ===
Oct 10 08:22:07.190653 ea1963d840ac NOTICE #buffermgrd: :- handleBufferProfileTable: BUFFER_PROFILE egress_lossy_profile has been inserted into APPL_DB directly

Oct 10 08:20:00.872175 ea1963d840ac NOTICE #buffermgrd: :- handlePendingBufferObjects: Additional zero profiles will be appied after 20 seconds
Oct 10 08:20:10.872400 ea1963d840ac NOTICE #buffermgrd: :- handlePendingBufferObjects: Additional zero profiles will be appied after 10 seconds

Oct 10 08:22:07.263265 ea1963d840ac NOTICE #buffermgrd: :- reclaimReservedBufferForPort: Reclaiming buffer reserved for all priority group(s) from port Ethernet0
Oct 10 08:22:07.266157 ea1963d840ac NOTICE #orchagent: :- processPriorityGroup: Set buffer PG Ethernet0:0 to BUFFER_PROFILE_TABLE:ingress_lossy_pg_zero_profile
Oct 10 08:22:07.267450 ea1963d840ac NOTICE #buffermgrd: :- reclaimReservedBufferForPort: Reclaiming buffer reserved for ingress profile list from port Ethernet0
Oct 10 08:22:07.267462 ea1963d840ac NOTICE #buffermgrd: :- reclaimReservedBufferForPort: Reclaiming buffer reserved for all queue(s) from port Ethernet0
Oct 10 08:22:07.276653 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:0-2 to BUFFER_PROFILE_TABLE:egress_lossy_zero_profile
Oct 10 08:22:07.276813 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:3-4 to BUFFER_PROFILE_TABLE:egress_lossless_zero_profile
Oct 10 08:22:07.276882 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:5-6 to BUFFER_PROFILE_TABLE:egress_lossy_zero_profile
Oct 10 08:22:07.276947 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:7-19 to BUFFER_PROFILE_TABLE:egress_lossy_zero_profile
Oct 10 08:22:07.300750 ea1963d840ac NOTICE #portmgrd: :- doTask: Configure Ethernet0 admin status to down

Oct 10 08:25:39.732379 ea1963d840ac INFO #supervisord 2025-10-10 08:25:39,730 WARN stopped: buffermgrd (terminated by SIGTERM)
Oct 10 08:25:39.742444 ea1963d840ac INFO #supervisord 2025-10-10 08:25:39,741 INFO spawned: 'buffermgrd' with pid 957
Oct 10 08:25:39.809740 ea1963d840ac NOTICE #buffermgrd: :- main: --- Starting buffermgrd ---

Oct 10 08:25:39.865306 ea1963d840ac NOTICE #orchagent: :- processPriorityGroup: Set buffer PG Ethernet0:0 to BUFFER_PROFILE_TABLE:ingress_lossy_profile
Oct 10 08:25:39.882023 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:0-2 to BUFFER_PROFILE_TABLE:egress_lossy_profile
Oct 10 08:25:39.882259 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:3-4 to BUFFER_PROFILE_TABLE:egress_lossless_profile
Oct 10 08:25:39.882418 ea1963d840ac NOTICE #orchagent: :- processQueue: Set buffer queue Ethernet0:5-6 to BUFFER_PROFILE_TABLE:egress_lossy_profile

Oct 10 08:25:40.753301 ea1963d840ac INFO #supervisord 2025-10-10 08:25:40,748 INFO success: buffermgrd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Oct 10 08:25:40.833787 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Remove buffer profile ingress_lossless_zero_profile with type BUFFER_PROFILE_TABLE
Oct 10 08:25:40.837072 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Remove buffer profile egress_lossless_zero_profile with type BUFFER_PROFILE_TABLE
Oct 10 08:25:40.838165 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Can't remove object egress_lossy_zero_profile due to being referenced (BUFFER_PROFILE_TABLE egress_lossy_zero_profile one object: Ethernet0:7-19 reference count: 32)
Oct 10 08:25:40.840774 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Remove buffer profile ingress_lossy_zero_profile with type BUFFER_PROFILE_TABLE
Oct 10 08:25:40.840968 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Can't remove object egress_lossy_zero_profile due to being referenced (BUFFER_PROFILE_TABLE egress_lossy_zero_profile one object: Ethernet0:7-19 reference count: 32)
Oct 10 08:25:40.847279 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Can't remove object egress_lossy_zero_profile due to being referenced (BUFFER_PROFILE_TABLE egress_lossy_zero_profile one object: Ethernet0:7-19 reference count: 32)
Oct 10 08:25:40.849561 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Remove buffer profile ingress_lossy_pg_zero_profile with type BUFFER_PROFILE_TABLE
Oct 10 08:25:40.849633 ea1963d840ac NOTICE #orchagent: :- processBufferProfile: Can't remove object egress_lossy_zero_profile due to being referenced (BUFFER_PROFILE_TABLE egress_lossy_zero_profile one object: Ethernet0:7-19 reference count: 32)
Oct 10 08:25:40.851747 ea1963d840ac NOTICE #orchagent: :- processBufferPool: Removed buffer pool ingress_zero_pool with type BUFFER_POOL_TABLE

Eventually, the cleanup of BUFFER_PROFILE_TABLE:egress_lossy_zero_profile was missed for Ethernet0:7-19

What I did

  • Improved the dynamic buffer model teardown flow

Why I did it

  • To fix the issue related to non-default objects clean up flow

How I verified it

  1. Run VS

Details if related

  • N/A

@mssonicbld
Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@nazariig nazariig marked this pull request as ready for review October 10, 2025 20:20
@nazariig nazariig requested a review from prsunny as a code owner October 10, 2025 20:20
Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
@nazariig nazariig force-pushed the master-dyn-buf-vs-fix branch from d6a545a to c5f8982 Compare October 12, 2025 23:39
@mssonicbld
Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@nazariig
Copy link
Copy Markdown
Collaborator Author

@prsunny can you please help to merge?

@kperumalbfn
Copy link
Copy Markdown
Contributor

@dgsudharsan could you please merge this

@dgsudharsan dgsudharsan merged commit 287461b into sonic-net:master Oct 13, 2025
15 checks passed
@kperumalbfn
Copy link
Copy Markdown
Contributor

@jianyuewu please include this in 202505

@mssonicbld
Copy link
Copy Markdown
Collaborator

Cherry-pick PR to 202505: #3965

stepanblyschak pushed a commit to stepanblyschak/sonic-swss that referenced this pull request Oct 31, 2025
balanokia pushed a commit to balanokia/sonic-swss that referenced this pull request Nov 17, 2025
theasianpianist pushed a commit to theasianpianist/sonic-swss that referenced this pull request Feb 4, 2026
…nic-net#3931)

Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
Signed-off-by: Lawrence Lee <lawlee@microsoft.com>
baorliu pushed a commit to baorliu/sonic-swss that referenced this pull request Feb 23, 2026
…nic-net#3931)

Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
Signed-off-by: Baorong Liu <96146196+baorliu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants