Skip to content

[orchagent] Set ABRT signal in STATE_DB during a SAI failure#2556

Closed
vivekrnv wants to merge 18 commits intosonic-net:masterfrom
vivekrnv:orch_abrt
Closed

[orchagent] Set ABRT signal in STATE_DB during a SAI failure#2556
vivekrnv wants to merge 18 commits intosonic-net:masterfrom
vivekrnv:orch_abrt

Conversation

@vivekrnv
Copy link
Copy Markdown
Contributor

@vivekrnv vivekrnv commented Dec 5, 2022

Signed-off-by: Vivek Reddy Karri [email protected]

What I did

  • During a SAI programming failure, made orchagent set the ORCH_ABRT_STATUS flag in STATE_DB. Also, delete the flag once the orchagent restarts

Why I did it

How I verified it

Simulate a SAI failure:

81396-Nov 21 18:14:59.131240 mtbc-sonic-01-2410 ERR swss#orchagent: :- create: create status: SAI_STATUS_INVALID_ATTRIBUTE_MAX
81397-Nov 21 18:14:59.131240 mtbc-sonic-01-2410 ERR swss#orchagent: :- sflowCreateSession: Failed to create sample packet session with rate 512
81398:Nov 21 18:14:59.131240 mtbc-sonic-01-2410 ERR swss#orchagent: :- handleCreate: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_SAMPLEPACKET, status: SAI_STATUS_INVALID_ATTRIBUTE_MAX

Check if STATE_DB is updated and cleared once the orchagent is restarted

root@mtbc-sonic-01-2410:/home/admin# sonic-db-cli STATE_DB GET ORCH_ABRT_STATUS
1
root@mtbc-sonic-01-2410:/home/admin#

root@mtbc-sonic-01-2410:/home/admin# sonic-db-cli STATE_DB GET ORCH_ABRT_STATUS

root@mtbc-sonic-01-2410:/home/admin#

Details if related

vivekrnv added 14 commits October 21, 2022 01:31
@vivekrnv
Added ABRT orchagent signal
bda046e 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Include sai_status_handler headers
3b58799 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
remove sai_status header file
b8fd6e6 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
compilation successfully
2caf395 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Fix UT's
2f67282 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Updated Makefiles of cfgmgr
aa4d986 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Removed extra header file
b66ccc3 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
minor updates
0e2a8d2 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Remove refactoring
8e18792 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Merge branch 'master' of https://github.com/Azure/sonic-swss into orc…
0edd3e6 
…h_abrt
@vivekrnv
minor update
565969c 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
minor update
756bdc2 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
minor fix
c60629d 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv
Merge branch 'master' of https://github.com/Azure/sonic-swss into orc…
6a4140e 
…h_abrt
@vivekrnv vivekrnv requested a review from prsunny as a code owner December 5, 2022 06:09
@vivekrnv vivekrnv mentioned this pull request Dec 5, 2022
Closed
@vivekrnv
Copy link
Copy Markdown
Contributor Author

vivekrnv commented Dec 6, 2022

/azpw run Azure.sonic-swss

@mssonicbld
Copy link
Copy Markdown
Collaborator

mssonicbld commented Dec 6, 2022

/AzurePipelines run Azure.sonic-swss

@azure-pipelines
Copy link
Copy Markdown

azure-pipelines bot commented Dec 6, 2022

Azure Pipelines successfully started running 1 pipeline(s).

prsunny
prsunny reviewed Dec 6, 2022
View reviewed changes
orchagent/main.cpp
DBConnector state_db("STATE_DB", 0);

/* Clears the ORCH_ABORT_STATUS flag in STATE_DB */
state_db.del(ORCH_ABRT);
Copy link
Copy Markdown
Collaborator

@prsunny prsunny Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest have flush table at common place - https://github.com/sonic-net/sonic-buildimage/blob/master/files/scripts/swss.sh#L177

Copy link
Copy Markdown
Contributor Author

@vivekrnv vivekrnv Dec 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That can be done, but clearing it here gives more time buffer to the processes dependent on this flag.

orchagent/orch.cpp
extern bool gLogRotate;
extern string gRecordFile;

void notifyAbort(){
Copy link
Copy Markdown
Collaborator

@prsunny prsunny Dec 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API name is not aligned to the functionality. This is doing both notify and abort. API gives the indication that it is only notification. Please move abort to original code and use it only to notify or change the name to say notifyAndAbort

Copy link
Copy Markdown
Contributor Author

@vivekrnv vivekrnv Dec 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will update

Copy link
Copy Markdown
Contributor Author

@vivekrnv vivekrnv Dec 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Handled

@vivekrnv
Handled review comments
a3153ff 
Signed-off-by: Vivek Reddy Karri <[email protected]>
@vivekrnv vivekrnv requested a review from prsunny December 9, 2022 00:24
@vivekrnv
Copy link
Copy Markdown
Contributor Author

vivekrnv commented Dec 12, 2022

/azpw run Azure.sonic-swss

@mssonicbld
Copy link
Copy Markdown
Collaborator

mssonicbld commented Dec 12, 2022

/AzurePipelines run Azure.sonic-swss

@azure-pipelines
Copy link
Copy Markdown

azure-pipelines bot commented Dec 12, 2022

Azure Pipelines successfully started running 1 pipeline(s).

@vivekrnv
Copy link
Copy Markdown
Contributor Author

vivekrnv commented Dec 13, 2022

/azpw run Azure.sonic-swss

@mssonicbld
Copy link
Copy Markdown
Collaborator

mssonicbld commented Dec 13, 2022

/AzurePipelines run Azure.sonic-swss

@azure-pipelines
Copy link
Copy Markdown

azure-pipelines bot commented Dec 13, 2022

Azure Pipelines successfully started running 1 pipeline(s).

@vivekrnv
Copy link
Copy Markdown
Contributor Author

vivekrnv commented Dec 15, 2022

/azpw run Azure.sonic-swss

@mssonicbld
Copy link
Copy Markdown
Collaborator

mssonicbld commented Dec 15, 2022

/AzurePipelines run Azure.sonic-swss

@azure-pipelines
Copy link
Copy Markdown

azure-pipelines bot commented Dec 15, 2022

Azure Pipelines successfully started running 1 pipeline(s).

@vivekrnv vivekrnv closed this Jan 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dgsudharsan dgsudharsan dgsudharsan approved these changes

@prsunny prsunny Awaiting requested review from prsunny prsunny is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vivekrnv @mssonicbld @dgsudharsan @prsunny