Skip to content

Survive pfc watchdog storm action across warm-reboot #794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wendani merged 24 commits into from
Apr 8, 2019
Merged

Survive pfc watchdog storm action across warm-reboot #794

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
5b25540
Survive PFC watchdog and storm action in warm-reboot
wendani Feb 16, 2019
1840d85
Remove logs used for debugging
wendani Feb 16, 2019
83ae4bf
Add queue index check before taking storm action during warm-reboot
wendani Feb 16, 2019
a925131
Correct log message
wendani Feb 16, 2019
6550073
Log storm event for all storm actions not only drop action
wendani Feb 16, 2019
383209e
Address review comments
wendani Feb 17, 2019
90f6db3
Address the situation that stoi() may throw an exception
wendani Feb 19, 2019
fb88541
Merge remote-tracking branch 'public/master' into pfcwd_wb_master
wendani Feb 28, 2019
722bc5f
Fine-gran handling of stoi exceptions
wendani Feb 26, 2019
8264eda
Shift temporarily to STATE_DB
wendani Feb 27, 2019
a5bc06a
Add debugging symbols
wendani Feb 27, 2019
d8e92a8
Revert "Shift temporarily to STATE_DB"
wendani Feb 27, 2019
ea7ba0d
Orthogonalize pfc wd table names
wendani Feb 28, 2019
8fce358
Implement doTask for the new Consumer, which subscribes to APPL_DB
wendani Feb 28, 2019
e3e789b
Clean up and touch-ups
wendani Feb 28, 2019
4500dae
Delete multiple fields in one hdel call
wendani Mar 1, 2019
bd53d49
Refactor codes with multi-fields hdel
wendani Mar 1, 2019
52df92d
Address comments: remove unnecessary catch blocks for stoi() call
wendani Mar 2, 2019
057fe50
Use RedisClient to do hset (previous through Table hset)
wendani Mar 2, 2019
5e0ff53
Remove debugging symbols
wendani Mar 1, 2019
36609e9
Address review comments: Replace PfcWdSwOrch<DropHandler, ForwardHand…
wendani Mar 23, 2019
cf597a6
Merge remote-tracking branch 'public/master' into pfc_wb_master
wendani Apr 1, 2019
2de7de5
Address review comments: Refactor existing codes to replace PfcWdSwOr…
wendani Apr 1, 2019
8b8e06c
Remove unused variable to correct compile error
wendani Apr 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions orchagent/pfc_detect_barefoot.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,15 @@ for i = n, 1, -1 do
-- DEBUG CODE END.
(occupancy_bytes == 0 and pfc_rx_packets - pfc_rx_packets_last > 0 and pfc_on2off - pfc_on2off_last == 0 and queue_pause_status_last == 'true' and queue_pause_status == 'true') then
if time_left <= poll_time then
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","storm"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","storm"]')
is_deadlock = true
time_left = detection_time
else
time_left = time_left - poll_time
end
else
if pfc_wd_action == 'alert' and pfc_wd_status ~= 'operational' then
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","restore"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","restore"]')
end
time_left = detection_time
end
Expand Down
4 changes: 2 additions & 2 deletions orchagent/pfc_detect_broadcom.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,15 @@ for i = n, 1, -1 do
-- DEBUG CODE END.
(occupancy_bytes == 0 and pfc_rx_packets - pfc_rx_packets_last > 0 and pfc_on2off - pfc_on2off_last == 0 and queue_pause_status_last == 'true' and queue_pause_status == 'true') then
if time_left <= poll_time then
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","storm"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","storm"]')
is_deadlock = true
time_left = detection_time
else
time_left = time_left - poll_time
end
else
if pfc_wd_action == 'alert' and pfc_wd_status ~= 'operational' then
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","restore"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","restore"]')
end
time_left = detection_time
end
Expand Down
4 changes: 2 additions & 2 deletions orchagent/pfc_detect_mellanox.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,15 +73,15 @@ for i = n, 1, -1 do
if time_left <= poll_time then
redis.call('HDEL', counters_table_name .. ':' .. port_id, pfc_rx_pkt_key .. '_last')
redis.call('HDEL', counters_table_name .. ':' .. port_id, pfc_duration_key .. '_last')
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","storm"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","storm"]')
is_deadlock = true
time_left = detection_time
else
time_left = time_left - poll_time
end
else
if pfc_wd_action == 'alert' and pfc_wd_status ~= 'operational' then
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","restore"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","restore"]')
end
time_left = detection_time
end
Expand Down
2 changes: 1 addition & 1 deletion orchagent/pfc_restore.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ for i = n, 1, -1 do
-- DEBUG CODE END.
then
if time_left <= poll_time then
redis.call('PUBLISH', 'PFC_WD', '["' .. KEYS[i] .. '","restore"]')
redis.call('PUBLISH', 'PFC_WD_ACTION', '["' .. KEYS[i] .. '","restore"]')
time_left = restoration_time
else
time_left = time_left - poll_time
Expand Down
Loading