Skip to content

Update syncd stop script to collect saisdkdump during orch abort #12936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vivekrnv wants to merge 12 commits into from
Closed

Update syncd stop script to collect saisdkdump during orch abort #12936

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c443ecf
update syncd stop script to collect saisdkdump during orch abort
vivekrnv Oct 28, 2022
3075768
scripts updated to new design
vivekrnv Nov 14, 2022
60b04aa
Minor updates
vivekrnv Nov 14, 2022
67cea83
Minor update
vivekrnv Nov 14, 2022
b0258b4
create the tmp file for all plaforms
vivekrnv Nov 15, 2022
046f171
Log updated
vivekrnv Nov 17, 2022
dcf3298
removed sleep
vivekrnv Nov 17, 2022
bbf0f56
Move cleaning after the copy
vivekrnv Nov 21, 2022
e0eaf6c
Add 1m timeout on invoking saisdkdump
vivekrnv Nov 23, 2022
4c36ba5
Merge branch 'master' of https://github.com/Azure/sonic-buildimage in…
vivekrnv Nov 29, 2022
05431b7
reduced timeout to 30s
vivekrnv Dec 1, 2022
0a5733d
Limit Number of dumps to 3
vivekrnv Dec 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions files/scripts/swss.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,8 @@ stop() {
if [[ x"$WARM_BOOT" != x"true" ]] && [[ x"$FAST_BOOT" != x"true" ]]; then
/usr/bin/${SERVICE}.sh stop $DEV
debug "Stopped ${SERVICE}$DEV service..."
$SONIC_DB_CLI APPL_DB DEL PORT_TABLE:PortInitDone
debug "Cleared PortInitDone from APPL_DB..."
else
debug "Killing Docker swss..."
/usr/bin/docker kill swss &> /dev/null || debug "Docker swss is not running ($?) ..."
Expand Down
34 changes: 34 additions & 0 deletions files/scripts/syncd.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,29 @@

. /usr/local/bin/syncd_common.sh

function collect_saisdkdump() {
TMP_DMP_DIR="/tmp/orch_abrt_sdkdump/"
HOST_SDKDUMP_LOC="/var/log/orch_abrt_sdkdump/"
local sai_dump_filename="sai_sdk_dump_$(date +"%m_%d_%Y_%I_%M_%p")"
local sai_dump_filename_epoch="sai_sdk_dump_$(date +%s)"

/usr/bin/docker exec syncd$DEV rm -rf ${TMP_DMP_DIR}
/usr/bin/docker exec syncd$DEV rm -f /tmp/${sai_dump_filename_epoch}.tar
/usr/bin/docker exec syncd$DEV mkdir -p ${TMP_DMP_DIR}
timeout 30s bash -c "/usr/bin/docker exec syncd$DEV saisdkdump -f ${TMP_DMP_DIR}/${sai_dump_filename} > /dev/null"
/usr/bin/docker exec syncd$DEV tar -czf /tmp/${sai_dump_filename_epoch}.tar.gz -C ${TMP_DMP_DIR} .

if [[ $? == 0 ]]; then
mkdir -p ${HOST_SDKDUMP_LOC}
/usr/bin/docker cp syncd$DEV:/tmp/${sai_dump_filename_epoch}.tar.gz ${HOST_SDKDUMP_LOC}
# Only retain the latest 3 files
ls -1td ${HOST_SDKDUMP_LOC}/sai_sdk_dump_* | tail -n +3 | xargs rm -rf
debug "${sai_dump_filename_epoch}.tar.gz collected before taking stopping syncd"
else
debug "Failed to collect saisdkdump before stopping syncd"
fi
}

function startplatform() {

# platform specific tasks
Expand Down Expand Up @@ -75,6 +98,17 @@ function waitplatform() {

function stopplatform1() {

if [[ x"$(${SONIC_DB_CLI} STATE_DB GET ORCH_ABRT_STATUS)" == x"1" ]]; then
# Collecting saisdkdump before restarting syncd
# Runs when orchagent is aborted because of SAI failure.
# Only enabled for mellanox platform
if [[ x$sonic_asic_platform == x"mellanox" ]]; then
collect_saisdkdump
fi
# This is used to notify auto-techsupport process
touch /tmp/saidump_collection_notify_flag
fi

if [[ x$sonic_asic_platform == x"mellanox" ]] && [[ x$TYPE == x"cold" ]]; then
debug "Stopping pmon service ahead of syncd..."
/bin/systemctl stop pmon
Expand Down