Improve MIBUpdater to re-connect DBConnector when re-init data. by liuh-80 · Pull Request #290 · sonic-net/sonic-snmpagent

liuh-80 · 2023-09-14T08:24:41Z

Improve MIBUpdater to re-connect DBConnector when re-init data.

Work item tracking

Microsoft ADO (number only): 24705208

- What I did
Fix when redis restart, some MIBUpdater's db connection will broken and keeps report error to syslog issue.

There will be following message repeat in syslog:

#012  File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 674, in _update_per_namespace_data#
#012    msg = pubsub.get_message()#
#012  File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1626, in get_message#
#012    return _swsscommon.PubSub_get_message(self, timeout)#
#012RuntimeError: RedisError: Failed to select, err=3: errstr=Server closed the connection

- How I did it
Re-connect DBConnector in every MIBUpdater's reinit_data method.

- How to verify it
Pass all UT

Manually test with following steps:

in database container kill redis server.
start redis in database container later: service redis-server start
check syslog, confirm following log exist but now new log few minutes later after mibs re-init:

sudo cat /var/log/syslog | grep rfc2737.py | grep PubSub_get_message

Sep 22 03:30:17.223944 vlab-01 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#12 File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 43, in start#012 self.update_data()#12 File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 326, in update_data#012 updater.update_data(i, self.statedb[i])#12 File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 666, in update_data#012 self._update_per_namespace_data(self.pub_sub_dict[db_index])#12 File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 675, in _update_per_namespace_data#012 msg = pubsub.get_message()#12 File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1996, in get_message#012 return _swsscommon.PubSub_get_message(self, timeout, interrupt_on_signal)#012RuntimeError: RedisError: Failed to select, err=3: errstr=Server closed the connection

- Description for the changelog
Improve MIBUpdater to re-connect DBConnector when re-init data.

liuh-80 · 2023-09-15T05:26:44Z

Pipeline build break, already create a fix PR. pending for #293 merge first

liuh-80 · 2023-09-18T07:03:36Z

/azp run

azure-pipelines · 2023-09-18T07:03:44Z

Azure Pipelines successfully started running 1 pipeline(s).

src/sonic_ax_impl/mibs/ietf/rfc2737.py

qiluo-msft · 2023-10-09T22:14:55Z

src/ax_interface/mib.py


                # run the background update task
                self.update_data()
+                self.redis_exception_happen = False


redis_exception_happen

I am worried about the contract of member redis_exception_happen between multiple classes. How about define a new argument reconnect=False in reinit_data(), and keep redis_exception_happen as a private member only in this class. #Closed

Fixed, move redis_exception_happen to local variable. also because the python method override issue, add new method reinit_connection() to MIBUpdater interface.

qiluo-msft · 2023-10-22T04:01:59Z

src/ax_interface/mib.py

                if self.update_counter > self.reinit_rate:
+                    # reconnect when redis exception happen
+                    if redis_exception_happen:
+                            self.reinit_connection()


[](http://example.com/codeflow?start=24&length=4)

You can remove one extra level of indentation. #Closed

…c-net#290) Improve MIBUpdater to re-connect DBConnector when re-init data. #### Work item tracking Microsoft ADO (number only): 24705208 **- What I did** Fix when redis restart, some MIBUpdater's db connection will broken and keeps report error to syslog issue. There will be following message repeat in syslog: ``` sonic-net#12 File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 674, in _update_per_namespace_data# sonic-net#12 msg = pubsub.get_message()# sonic-net#12 File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1626, in get_message# sonic-net#12 return _swsscommon.PubSub_get_message(self, timeout)# #012RuntimeError: RedisError: Failed to select, err=3: errstr=Server closed the connection ``` **- How I did it** Re-connect DBConnector in every MIBUpdater's reinit_data method. **- How to verify it** Pass all UT Manually test with following steps: 1. in database container kill redis server. 2. start redis in database container later: service redis-server start 2. check syslog, confirm following log exist but now new log few minutes later after mibs re-init: sudo cat /var/log/syslog | grep rfc2737.py | grep PubSub_get_message Sep 22 03:30:17.223944 vlab-01 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):sonic-net#12 File "/usr/local/lib/python3.9/dist-packages/ax_interface/mib.py", line 43, in start#012 self.update_data()sonic-net#12 File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 326, in update_data#012 updater.update_data(i, self.statedb[i])sonic-net#12 File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 666, in update_data#012 self._update_per_namespace_data(self.pub_sub_dict[db_index])sonic-net#12 File "/usr/local/lib/python3.9/dist-packages/sonic_ax_impl/mibs/ietf/rfc2737.py", line 675, in _update_per_namespace_data#012 msg = pubsub.get_message()sonic-net#12 File "/usr/lib/python3/dist-packages/swsscommon/swsscommon.py", line 1996, in get_message#012 return _swsscommon.PubSub_get_message(self, timeout, interrupt_on_signal)#012RuntimeError: RedisError: Failed to select, err=3: errstr=Server closed the connection **- Description for the changelog** Improve MIBUpdater to re-connect DBConnector when re-init data.

Improve MIBUpdater to re-connect DBConnector when re-init data.

bbf45ab

liuh-80 marked this pull request as ready for review September 19, 2023 01:03

liuh-80 requested a review from qiluo-msft September 19, 2023 01:04

This comment was marked as resolved.

Sign in to view

qiluo-msft reviewed Sep 28, 2023

View reviewed changes

src/sonic_ax_impl/mibs/ietf/rfc2737.py Outdated Show resolved Hide resolved

liuh-80 added 3 commits October 7, 2023 06:09

Improve code

461334b

Improve UT coverage

e6042c2

Improve UT coverage

3f20ca8

qiluo-msft reviewed Oct 9, 2023

View reviewed changes

qiluo-msft mentioned this pull request Oct 9, 2023

Fixed, re-connect when exception happen. sonic-net/sonic-swss-common#822

Open

liuh-80 added 4 commits October 10, 2023 07:02

Improve code by PR comments

71a71cf

Fix UT

5612852

Fix UT issue

3f1f38a

Revert unecessary code

d2521e5

qiluo-msft reviewed Oct 22, 2023

View reviewed changes

Update mib.py

523ebb5

qiluo-msft approved these changes Oct 28, 2023

View reviewed changes

Merge branch 'master' into dev/liuh/improve-reinit

c9fc25c

liuh-80 merged commit 347a6e4 into sonic-net:master Oct 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve MIBUpdater to re-connect DBConnector when re-init data.#290

Improve MIBUpdater to re-connect DBConnector when re-init data.#290
liuh-80 merged 10 commits intosonic-net:masterfrom
liuh-80:dev/liuh/improve-reinit

liuh-80 commented Sep 14, 2023 •

edited by qiluo-msft

Loading

Uh oh!

liuh-80 commented Sep 15, 2023

Uh oh!

liuh-80 commented Sep 18, 2023

Uh oh!

azure-pipelines bot commented Sep 18, 2023

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

qiluo-msft Oct 9, 2023 •

edited

Loading

Uh oh!

liuh-80 Oct 10, 2023

Uh oh!

qiluo-msft Oct 22, 2023 •

edited

Loading

Uh oh!

liuh-80 Oct 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

liuh-80 commented Sep 14, 2023 • edited by qiluo-msft Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Work item tracking

Uh oh!

liuh-80 commented Sep 15, 2023

Uh oh!

liuh-80 commented Sep 18, 2023

Uh oh!

azure-pipelines bot commented Sep 18, 2023

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

qiluo-msft Oct 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liuh-80 Oct 10, 2023

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Oct 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liuh-80 Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

liuh-80 commented Sep 14, 2023 •

edited by qiluo-msft

Loading

qiluo-msft Oct 9, 2023 •

edited

Loading

qiluo-msft Oct 22, 2023 •

edited

Loading