Orchagent SAI error handling improvements (sonic-net#3587)

What I did
This change aims to reduce self-induced orchagent exit when any SAI API call fails (i.e returns anything other than SAI_STATUS_SUCCESS). This change is the first set of changes that does the following:

handleSaiCreateStatus() / handleSaiSetStatus() changes
Return 'task_success' for SAI_STATUS_ITEM_ALREADY_EXISTS, SAI_STATUS_ITEM_NOT_FOUND, SAI_STATUS_ADDR_NOT_FOUND and SAI_STATUS_OBJECT_IN_USE irrespective of the object type.
Return 'task_need_retry' for SAI_STATUS_INSUFFICIENT_RESOURCES, SAI_STATUS_TABLE_FULL, SAI_STATUS_NO_MEMORY and
SAI_STATUS_NV_STORAGE_FULL.
Call handleSaiFailure() and return 'task_failed' for other SAI errors. This will log a structured syslog via eventd and also take a SAI dump.
handleSaiRemoveStatus() changes
Return 'task_success' for SAI_STATUS_ITEM_ALREADY_EXISTS, SAI_STATUS_ITEM_NOT_FOUND, SAI_STATUS_ADDR_NOT_FOUND, SAI_STATUS_INSUFFICIENT_RESOURCES, SAI_STATUS_TABLE_FULL, SAI_STATUS_NO_MEMORY, SAI_STATUS_NV_STORAGE_FULL
Return 'task_need_retry' for SAI_STATUS_OBJECT_IN_USE
Call handleSaiFailure() and return 'task_failed' for other SAI errors. This will log a structured syslog via eventd and also take a SAI dump.
handleSaiGetStatus() changes
Log a NOTICE message and return task_failed. This is similar to what is being done today for GET calls.
handleSaiFailure() changes
Update handleSaiFailure() to take 3 arguments - namely the SAI API, operation type string and SAI API return status. This will be used in crafting a structured syslog error message when the failure happens. All callers of this function are updated accordingly.
Mock test changes
Added new tests for coverage and updated existing tests that do "ASSERT_DEATH" assertions when SAI API calls fail.
Fixed errors in existing portsorch_ut test cases
What is not done
There are changes needed to all orchs to handle scenarios where orchagent doesn't crash anymore when SAI API calls fail. There are also places in different orchs where an explicit exception is thrown in case of SAI errors. These and the remaining items in sonic-net/SONiC#1698 will be handled in phase-2.
 
Why I did it
Crashing orchagent on every SAI error is an overkill. Instead, we follow the approach that is called out in the HLD above to handle these errors in a more graceful manner.

Author: prabhataravind

orchagent/main.cpp

@@ -127,7 +127,8 @@ void syncd_apply_view()
     if (status != SAI_STATUS_SUCCESS)
     {
         SWSS_LOG_ERROR("Failed to notify syncd APPLY_VIEW %d", status);
-        handleSaiFailure(true);
+        handleSaiFailure(SAI_API_SWITCH, "set", status);
+        return;
     }
 }
 
@@ -701,7 +702,8 @@ int main(int argc, char **argv)
     if (status != SAI_STATUS_SUCCESS)
     {
         SWSS_LOG_ERROR("Failed to create a switch, rv:%d", status);
-        handleSaiFailure(true);
+        handleSaiFailure(SAI_API_SWITCH, "create", status);
+        return EXIT_FAILURE;
     }
     SWSS_LOG_NOTICE("Create a switch, id:%" PRIu64, gSwitchId);
 
@@ -732,7 +734,8 @@ int main(int argc, char **argv)
             if (status != SAI_STATUS_SUCCESS)
             {
                 SWSS_LOG_ERROR("Failed to get MAC address from switch, rv:%d", status);
-                handleSaiFailure(true);
+                handleSaiFailure(SAI_API_SWITCH, "get", status);
+                return EXIT_FAILURE;
             }
             else
             {
@@ -747,7 +750,8 @@ int main(int argc, char **argv)
         if (status != SAI_STATUS_SUCCESS)
         {
             SWSS_LOG_ERROR("Fail to get switch virtual router ID %d", status);
-            handleSaiFailure(true);
+            handleSaiFailure(SAI_API_SWITCH, "get", status);
+            return EXIT_FAILURE;
         }
 
         gVirtualRouterId = attr.value.oid;
@@ -789,7 +793,8 @@ int main(int argc, char **argv)
         if (status != SAI_STATUS_SUCCESS)
         {
             SWSS_LOG_ERROR("Failed to create underlay router interface %d", status);
-            handleSaiFailure(true);
+            handleSaiFailure(SAI_API_ROUTER_INTERFACE, "create", status);
+            return EXIT_FAILURE;
         }
 
         SWSS_LOG_NOTICE("Created underlay router interface ID %" PRIx64, gUnderlayIfId);

orchagent/orchdaemon.cpp

@@ -836,7 +836,7 @@ void OrchDaemon::flush()
     if (status != SAI_STATUS_SUCCESS)
     {
         SWSS_LOG_ERROR("Failed to flush redis pipeline %d", status);
-        handleSaiFailure(true);
+        handleSaiFailure(SAI_API_SWITCH, "set", status);
     }
 
     for (auto* orch: m_orchList)