Skip to content

Commit 1580ccc

Browse files
GCU generates suboptimal plan for CreateOnly paths (#4335)
* GCU generates suboptimal plan for CreateOnly paths When GCU hits a CreateOnly entry that has changed, it generates a suboptimal plan. One example is a simple change of: ``` [{"op": "replace", "path": "/MIRROR_SESSION/EVERFLOW_TUNNEL/dst_ip", "value": "200.1.1.203"}] ``` Should generate an optimal plan of: ``` [ [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION"}], [{"op": "remove", "path": "/MIRROR_SESSION"}], [{"op": "add", "path": "/MIRROR_SESSION", "value": {"EVERFLOW_TUNNEL": {"dscp": "8", "dst_ip": "200.1.1.203", "src_ip": "100.1.1.1", "ttl": "255", "type": "ERSPAN"}}}] [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION", "value": "EVERFLOW_TUNNEL"}] ] ``` But instead generates this plan (which removes all ACLs): ``` [ [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION"}], [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1", "value": {"PRIORITY": "1000"}}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION", "value": "EVERFLOW_TUNNEL"}], [{"op": "remove", "path": "/ACL_RULE"}], [{"op": "add", "path": "/ACL_RULE", "value": {"EVERFLOW|RULE_1": {"PRIORITY": "1000", "IP_TYPE": "IP", "MIRROR_INGRESS_ACTION": "EVERFLOW_TUNNEL"}}}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1", "value": {"PRIORITY": "10"}}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/DST_IP", "value": "192.168.1.1/32"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/IP_TYPE", "value": "IP"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/L4_DST_PORT", "value": "22"}], [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION"}], [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1", "value": {"PRIORITY": "1000"}}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION", "value": "EVERFLOW_TUNNEL"}], [{"op": "remove", "path": "/ACL_RULE/DATAACL|RULE_1"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1", "value": {"PRIORITY": "10"}}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/DST_IP", "value": "192.168.1.1/32"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/IP_TYPE", "value": "IP"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/PACKET_ACTION", "value": "DROP"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/IP_TYPE", "value": "IP"}], [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION"}], [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1", "value": {"PRIORITY": "1000"}}], [{"op": "remove", "path": "/ACL_RULE/DATAACL|RULE_1"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/IP_TYPE", "value": "IP"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1", "value": {"PRIORITY": "10"}}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/DST_IP", "value": "192.168.1.1/32"}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/IP_TYPE", "value": "IP"}], [{"op": "remove", "path": "/ACL_RULE/EVERFLOW|RULE_1"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1", "value": {"PRIORITY": "1000"}}], [{"op": "remove", "path": "/MIRROR_SESSION"}], [{"op": "add", "path": "/MIRROR_SESSION", "value": {"EVERFLOW_TUNNEL": {"dscp": "8", "dst_ip": "200.1.1.203", "src_ip": "100.1.1.1", "ttl": "255", "type": "ERSPAN"}}}], [{"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/L4_DST_PORT", "value": "22"}, {"op": "add", "path": "/ACL_RULE/DATAACL|RULE_1/PACKET_ACTION", "value": "DROP"}], [{"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/IP_TYPE", "value": "IP"}, {"op": "add", "path": "/ACL_RULE/EVERFLOW|RULE_1/MIRROR_INGRESS_ACTION", "value": "EVERFLOW_TUNNEL"}] ] ``` Modified`RemoveCreateOnlyDependencyMoveGenerator`: * it would previously short-circuit early due to only processing one child leaf in the same table. * it would previously attempt to iterate across all members of the table even though there was a complete path list. * it was missing logic to remove the create only path itself (and was relying on extenders to do that which was inefficient and wouldn't generate the right plan) * when removing dependents it wasn't recursing to ensure it would remove dependents of dependents Since this generator is now full and doesn't rely on any extenders, it has been moved to a non-extendable generator. These changes caused some existing (suboptimal) plans that got generated to change so those test cases have also been updated. Added test case to validate this behavior and ensure it does not regress. Signed-off-by: Brad House <bhouse@nexthop.ai> * Update generic_config_updater/patch_sorter.py Prevent possible infinite loop scenario Copilot identified, however it shouldn't be possible given self.__get_path_count() can't return 1 in that scenario to allow the loop to continue. But hardening isn't a bad practice. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Brad House <bhouse@nexthop.ai> * Update generic_config_updater/patch_sorter.py Fix spelling / gramatical error caught by Copilot. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Brad House <bhouse@nexthop.ai> * Code Review Comments 1. Add recursion depth protection in removing dependents because of a concern of recursive dependencies, which isn't actually possible with yang. None-the-less, implemented. 2. Remove a duplicate move that gets generated as when using it with a Depth-first sorter its not necessary though could be on other sorters. 3. The old code depended on 3 levels of depth for the create only leafs. Reworked the logic to not be dependent on depth. 4. __get_path_count() can no longer return a KeyError even though the caller paths would make that impossible, but future use cases may need it to not throw an exception when the path is not found. Signed-off-by: Brad House <bhouse@nexthop.ai> --------- Signed-off-by: Brad House <bhouse@nexthop.ai> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 369e703 commit 1580ccc

File tree

3 files changed

+239
-108
lines changed

3 files changed

+239
-108
lines changed

generic_config_updater/patch_sorter.py

Lines changed: 74 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
import sonic_yang
55
from collections import deque, OrderedDict
66
from enum import Enum
7-
from typing import IO, Optional, Tuple
7+
from typing import Any, IO, List, Optional, Tuple
88
from .gu_common import OperationWrapper, OperationType, GenericConfigUpdaterError, \
99
JsonChange, PathAddressing, genericUpdaterLogging
1010

@@ -1580,55 +1580,89 @@ def generate(self, diff):
15801580
target_config = diff.target_config # Final config after applying whole patch
15811581
reload_config = True
15821582

1583-
processed_tables = set()
15841583
for path in self.create_only_filter.get_paths(current_config):
15851584
tokens = self.path_addressing.get_path_tokens(path)
1586-
table_to_check, create_only_field = tokens[0], tokens[-1]
1587-
1588-
if table_to_check in processed_tables:
1589-
continue
1590-
else:
1591-
processed_tables.add(table_to_check)
1592-
1593-
if table_to_check not in current_config:
1594-
continue
1585+
current_field = self.__fetch_path(current_config, tokens)
1586+
target_field = self.__fetch_path(target_config, tokens)
15951587

1596-
current_members = current_config[table_to_check]
1597-
if not current_members:
1588+
# If field is deleted or created we don't care, we only care when it was
1589+
# already set and is changing value.
1590+
if current_field is None or target_field is None or current_field == target_field:
15981591
continue
15991592

1600-
if table_to_check not in target_config:
1601-
continue
1593+
# Create only filters may reference an exact leaf, but it's really the parent that is the
1594+
# object we're after.
1595+
tokens.pop()
16021596

1603-
target_members = target_config[table_to_check]
1604-
if not target_members:
1605-
continue
1597+
# First see if there are any dependents for the exact path
1598+
for move in self.__remove_dependents(diff, tokens, reload_config=reload_config,
1599+
remove_parent=False):
1600+
yield move
16061601

1607-
for member_name in current_members:
1608-
if member_name not in target_members:
1609-
continue
1602+
# No need to reload config after first call
1603+
reload_config = False
16101604

1611-
current_field = self._get_create_only_field(
1612-
current_config, table_to_check, member_name, create_only_field)
1613-
target_field = self._get_create_only_field(
1614-
target_config, table_to_check, member_name, create_only_field)
1605+
yield self.__remove_nonempty(diff, tokens)
16151606

1616-
if current_field == target_field:
1617-
continue
1607+
# If that didn't work, likely the parents of the dependent path needs to be removed.
1608+
for move in self.__remove_dependents(diff, tokens, reload_config=reload_config,
1609+
remove_parent=True):
1610+
yield move
16181611

1619-
member_path = f"/{table_to_check}/{member_name}"
1612+
# Remove self again after removing the parents of dependents
1613+
# NOTE: When we use the DFS sorter this is irrelevant. Right now we only use DFS so we don't need it
1614+
# as it will be called again after it removed any parent dependents. Commenting out for now.
1615+
#
1616+
# yield self.__remove_nonempty(diff, tokens)
16201617

1621-
for ref_path in self.path_addressing.find_ref_paths(member_path, current_config,
1622-
reload_config=reload_config):
1623-
yield JsonMoveGroup(self.__class__.__name__, JsonMove(diff, OperationType.REMOVE,
1624-
self.path_addressing.get_path_tokens(ref_path)))
1618+
def __fetch_path(self, config, tokens: List[str]) -> Any:
1619+
for token in tokens:
1620+
config = config.get(token)
1621+
if config is None:
1622+
return None
1623+
return config
1624+
1625+
def __get_path_count(self, config, tokens: List[str]) -> int:
1626+
config = self.__fetch_path(config, tokens)
1627+
if config is None:
1628+
return 0
1629+
return len(config)
1630+
1631+
def __remove_nonempty(self, diff: Diff, tokens: List[str]):
1632+
remove_tokens = list(tokens)
1633+
# Only trim while there is at least one table token and one deeper level.
1634+
# This prevents generating an empty path ("") and avoids an infinite loop
1635+
# when the top-level config has a single table.
1636+
while len(remove_tokens) > 1 and \
1637+
self.__get_path_count(diff.current_config, remove_tokens[:-1]) == 1:
1638+
remove_tokens = remove_tokens[:-1]
1639+
return JsonMoveGroup(self.__class__.__name__, JsonMove(diff, OperationType.REMOVE, remove_tokens))
1640+
1641+
def __remove_dependents(
1642+
self,
1643+
diff: Diff,
1644+
tokens: List[str],
1645+
reload_config: bool,
1646+
remove_parent: bool,
1647+
recursion_depth: int = 0,
1648+
):
1649+
if recursion_depth >= 10:
1650+
return
16251651

1626-
# No need to reload config after first call
1627-
reload_config = False
1652+
config = diff.current_config
1653+
path = self.path_addressing.create_path(tokens)
1654+
ref_paths = self.path_addressing.find_ref_paths(path, config, reload_config)
1655+
for ref in ref_paths:
1656+
ref_tokens = self.path_addressing.get_path_tokens(ref)
1657+
if remove_parent:
1658+
ref_tokens.pop()
1659+
1660+
# Recurse since there could be a dependency chain
1661+
for move in self.__remove_dependents(diff, ref_tokens, reload_config=False,
1662+
remove_parent=remove_parent, recursion_depth=recursion_depth+1):
1663+
yield move
16281664

1629-
def _get_create_only_field(self, config, table_to_check,
1630-
member_name, create_only_field):
1631-
return config[table_to_check][member_name].get(create_only_field, None)
1665+
yield self.__remove_nonempty(diff, ref_tokens)
16321666

16331667

16341668
class LowLevelMoveGenerator:
@@ -2222,10 +2256,10 @@ def __init__(self, operation_wrapper, config_wrapper, path_addressing):
22222256
self.path_addressing = path_addressing
22232257

22242258
def create(self, algorithm=Algorithm.DFS, path_trace: bool = False):
2225-
move_generators = [RemoveCreateOnlyDependencyMoveGenerator(self.path_addressing),
2226-
LowLevelMoveGenerator(self.path_addressing)]
2259+
move_generators = [LowLevelMoveGenerator(self.path_addressing)]
22272260
# TODO: Enable TableLevelMoveGenerator once it is confirmed whole table can be updated at the same time
2228-
move_non_extendable_generators = [BulkKeyLevelMoveGenerator(self.path_addressing),
2261+
move_non_extendable_generators = [RemoveCreateOnlyDependencyMoveGenerator(self.path_addressing),
2262+
BulkKeyLevelMoveGenerator(self.path_addressing),
22292263
KeyLevelMoveGenerator(self.path_addressing),
22302264
BulkKeyGroupLowLevelMoveGenerator(self.path_addressing),
22312265
BulkLowLevelMoveGenerator(self.path_addressing)]

0 commit comments

Comments
 (0)