IPv6 ACL support for DASH

[mhanif]

Added IPv6 ACL support for DASH

[mhanif]

@marian-pritsak @mukeshmv @chrispsommers Please help review this PR. Thanks!

[mukeshmv]

I don't think we need the is_overlay_pkt_ip_v6 field here in each ACL rule since unlike routing we have separate ACL groups for v4 and v6. So depending on whether the pkt is v4 or v6, the correct ACL group is picked up by the pipeline. And all entries in that group will have only v4 or only v6 but not both.

[mhanif]

Hi @mukeshmv, thanks for reviewing it. Are you mixing inbound/outbound ACL groups with v4/v6? Where (or how) in the pipeline it picks up the correct ACL group based on the v4/v6 packet? May be I am missing something here? Thanks!

[mukeshmv]

@mhanif pipeline picks up v4 or v6 ACL group here
https://github.com/Azure/DASH/blob/main/dash-pipeline/bmv2/dash_pipeline.p4#L120

[mhanif]

Hi @mukeshmv, Yes, you are right. I have made the requested changes and resubmitted to this PR. Thanks.

[lguohan]

what about bmv2 model, do we have test for the pipeline?

[chrispsommers]

The P4 code changes LGTM although am not a dash-pipeline code expert.

I think it'd be a good idea to create a test-case(s) to validate the design (and catch future regressions) since this is a significant enhancement. Perhaps the easiest way is to copy test_saithrift_vnet.py to a new file test_saithrift_vnet_v6.py (and perhaps rename the original with a _v4 in the filename). Then you can edit to create an IPv6 version of the test.

If you're having problems running the tests in your dev environment (i.e. #218) we can have a 1:1 working session, I'd like to help clear it. Also, perhaps this workflow will work, it's also faster for incremental test-case development.

[mukeshmv]

lgtm

[mhanif]

Hi @chrispsommers, ran the v6 equivalent of test_saithrift_vnet.py using the suggested workflow. Found few issues in the IPv6 packet processing and fixed them. v6 tests are now passing. Please review the recent changes. Thanks!

[mukeshmv]

@mhanif did you miss to add your v6 version of the vnet test to the commit ?

[chrispsommers]

@mhanif I have same question as @mukeshmv - did you intend to submit the test-case? It'd be a great reference test.

[mhanif]

@mukeshmv @chrispsommers thanks for looking into this. As soon as I submitted the changes last night, I knew that this question will come up and you folks didn't disappoint me :-) I should have mentioned why I didn't include the test. I ran the test by "enabling IPv6" which has disabled to fix the "IPv6 noise" and, hence, I didn't submit the test case file. I do intend to submit the test and should do it ASAP so as to protect any regression. I just have to see how both v4 and v6 can coexist. Do you guys want it to be done as part of this PR or I can work on it and do a separate PR as this might require a little bit of more work? Thanks.

[chrispsommers]

Hi @mhanif , I am OK with merging this now and then adding the IPv6 test later, especially since you've already invested the effort to create it. This general issue of filtering out unwanted IPv6 noise from Linux has to be solved eventually, and a number of folks are aware of it and thinking about it. For the drop case, you can add the expected packet to "veryify_no_packets()" and it will test that there are no instances of the dropped packet, ignoring anything else. The inverse case, verifying you in fact got the wanted packet in the midst of noise, is trickier and merits discussion.

Also, thanks for fixing the README gaffes. While you're at it, did you ever try running a single PTF test class from inside the container? If so, and you have an example of such, could you add it to the README? Two people asked about it on the same day and it needs to be documented.

[chrispsommers]

LGTM, see general comment also.

[mukeshmv]

I ran the test by "enabling IPv6" which has disabled to fix the "IPv6 noise" and

@mhanif I don't think you need to enable IPv6 on Linux since we are not using any Linux IPv6 capabilities per-se. My understanding is we are internally crafting a complete VXLAN encapsulated ethernet frame using PTF and sending it as a raw packet on the interface to the bmv2 and then expecting an ethernet frame containing the modified VXLAN encapsulated to egress out on the other interface. Can you try your test with IPv6 disabled ?

[mhanif]

Hi @mukeshmv, Yes, you are right. Since our underlay is still v4, I didn't need to enable v6. I tried with the same script which disables v6 and my test worked. However, I am not able to run both v4 and v6 test cases in the same "run". I can run them separately and they pass. I have created an identical v6 test with the class name defined as TestSaiThrift_outbound_udpv6_pkt. It doesn't matter whether I keep them (v4 and v6 classes) in the same file or in different files, after running the v4 test case, the v6 test case just hangs. However if I run them separately by going in to the saithrift client container, I can run them separately and they pass. What am I doing wrong? Below is the test class code.

class TestSaiThrift_outbound_udpv6_pkt(ThriftInterfaceDataPlane):
    """ Test saithrift vnet outbound"""
    def setUp(self):
        super(TestSaiThrift_outbound_udpv6_pkt, self).setUp()
        self.switch_id = 5
        self.outbound_vni = 60
        self.vnet_vni = 50
        self.eni_mac = "00:aa:aa:aa:aa:aa"
        self.our_mac = "00:00:06:07:08:09"
        self.dst_ca_mac = "00:bb:bb:bb:bb:bb"
        self.vip = "172.16.1.200"
        self.outbound_vni = 50
        self.dst_ca_ip = "2000:aaaa::232"
        self.dst_pa_ip = "172.16.1.30"
        self.src_vm_pa_ip = "172.16.1.2"
        self.cleaned_up = False

        try:
            vip = sai_thrift_ip_address_t(addr_family=SAI_IP_ADDR_FAMILY_IPV4,
                                          addr=sai_thrift_ip_addr_t(ip4=self.vip))
            self.vpe = sai_thrift_vip_entry_t(switch_id=self.switch_id, vip=vip)

            status = sai_thrift_create_vip_entry(self.client, self.vpe,
                                action=SAI_VIP_ENTRY_ACTION_ACCEPT)
            assert(status == SAI_STATUS_SUCCESS)

            
            self.dle = sai_thrift_direction_lookup_entry_t(switch_id=self.switch_id, vni=self.outbound_vni)
            status = sai_thrift_create_direction_lookup_entry(self.client, self.dle,
                                action=SAI_DIRECTION_LOOKUP_ENTRY_ACTION_SET_OUTBOUND_DIRECTION)
            assert(status == SAI_STATUS_SUCCESS)

            self.in_acl_group_id = sai_thrift_create_dash_acl_group(self.client,
                                                                    ip_addr_family=SAI_IP_ADDR_FAMILY_IPV6)
            assert (self.in_acl_group_id != SAI_NULL_OBJECT_ID)
            self.out_acl_group_id = sai_thrift_create_dash_acl_group(self.client,
                                                                     ip_addr_family=SAI_IP_ADDR_FAMILY_IPV6)
            assert (self.out_acl_group_id != SAI_NULL_OBJECT_ID)

            self.vnet = sai_thrift_create_vnet(self.client, vni=self.vnet_vni)
            assert (self.vnet != SAI_NULL_OBJECT_ID)

            vm_underlay_dip = sai_thrift_ip_address_t(addr_family=SAI_IP_ADDR_FAMILY_IPV4,
                                                      addr=sai_thrift_ip_addr_t(ip4=self.src_vm_pa_ip))
            self.eni = sai_thrift_create_eni(self.client, cps=10000,
                                             pps=100000, flows=100000,
                                             admin_state=True,
                                             vm_underlay_dip=vm_underlay_dip,
                                             vm_vni=9,
                                             vnet_id=self.vnet,
                                             # TODO: Enable ACL rule
                                             #inbound_v4_stage1_dash_acl_group_id = self.in_acl_group_id,
                                             #inbound_v4_stage2_dash_acl_group_id = self.in_acl_group_id,
                                             #inbound_v4_stage3_dash_acl_group_id = self.in_acl_group_id,
                                             #inbound_v4_stage4_dash_acl_group_id = self.in_acl_group_id,
                                             #inbound_v4_stage5_dash_acl_group_id = self.in_acl_group_id,
                                             #outbound_v4_stage1_dash_acl_group_id = self.out_acl_group_id,
                                             #outbound_v4_stage2_dash_acl_group_id = self.out_acl_group_id,
                                             #outbound_v4_stage3_dash_acl_group_id = self.out_acl_group_id,
                                             #outbound_v4_stage4_dash_acl_group_id = self.out_acl_group_id,
                                             #outbound_v4_stage5_dash_acl_group_id = self.out_acl_group_id,
                                             inbound_v4_stage1_dash_acl_group_id = 0,
                                             inbound_v4_stage2_dash_acl_group_id = 0,
                                             inbound_v4_stage3_dash_acl_group_id = 0,
                                             inbound_v4_stage4_dash_acl_group_id = 0,
                                             inbound_v4_stage5_dash_acl_group_id = 0,
                                             outbound_v4_stage1_dash_acl_group_id = 0,
                                             outbound_v4_stage2_dash_acl_group_id = 0,
                                             outbound_v4_stage3_dash_acl_group_id = 0,
                                             outbound_v4_stage4_dash_acl_group_id = 0,
                                             outbound_v4_stage5_dash_acl_group_id = 0,
                                             inbound_v6_stage1_dash_acl_group_id = 0,
                                             inbound_v6_stage2_dash_acl_group_id = 0,
                                             inbound_v6_stage3_dash_acl_group_id = 0,
                                             inbound_v6_stage4_dash_acl_group_id = 0,
                                             inbound_v6_stage5_dash_acl_group_id = 0,
                                             outbound_v6_stage1_dash_acl_group_id = 0,
                                             outbound_v6_stage2_dash_acl_group_id = 0,
                                             outbound_v6_stage3_dash_acl_group_id = 0,
                                             outbound_v6_stage4_dash_acl_group_id = 0,
                                             outbound_v6_stage5_dash_acl_group_id = 0)

            self.eam = sai_thrift_eni_ether_address_map_entry_t(switch_id=self.switch_id, address = self.eni_mac)
            status = sai_thrift_create_eni_ether_address_map_entry(self.client,
                                                        eni_ether_address_map_entry=self.eam,
                                                        eni_id=self.eni)
            assert(status == SAI_STATUS_SUCCESS)

            dip = sai_thrift_ip_address_t(addr_family=SAI_IP_ADDR_FAMILY_IPV6,
                                          addr=sai_thrift_ip_addr_t(ip6=self.dst_ca_ip))
            # TODO: Enable ACL rule
            #self.out_acl_rule_id = sai_thrift_create_dash_acl_rule(self.client, dash_acl_group_id=self.out_acl_group_id,
            #                                           dip=dip, priority=10, action=SAI_DASH_ACL_RULE_ACTION_PERMIT)
            #assert(status == SAI_STATUS_SUCCESS)

            ca_prefix = sai_thrift_ip_prefix_t(addr_family=SAI_IP_ADDR_FAMILY_IPV6,
                                               addr=sai_thrift_ip_addr_t(ip6="2000:aaaa::"),
                                               mask=sai_thrift_ip_addr_t(ip6="ffff:ffff:ffff:ffff:ffff:ffff:ffff:0"))
                                               
            self.ore = sai_thrift_outbound_routing_entry_t(switch_id=self.switch_id, eni_id=self.eni, destination=ca_prefix)
            status = sai_thrift_create_outbound_routing_entry(self.client, self.ore, action=SAI_OUTBOUND_ROUTING_ENTRY_ACTION_ROUTE_VNET, dst_vnet_id = self.vnet)
            print ("status returned is ", status)
            assert(status == SAI_STATUS_SUCCESS)

            underlay_dip = sai_thrift_ip_address_t(addr_family=SAI_IP_ADDR_FAMILY_IPV4,
                                          addr=sai_thrift_ip_addr_t(ip4=self.dst_pa_ip))
            self.ocpe = sai_thrift_outbound_ca_to_pa_entry_t(switch_id=self.switch_id, dst_vnet_id=self.vnet, dip=dip)
            status = sai_thrift_create_outbound_ca_to_pa_entry(self.client, self.ocpe, underlay_dip = underlay_dip, overlay_dmac=self.dst_ca_mac, use_dst_vnet_vni = True)
            assert(status == SAI_STATUS_SUCCESS)

        except AssertionError as ae:
            self.failure_teardown()
            raise ae

    def runTest(self):
        try:
            src_vm_ip = "2000:aaaa::10a"
            outer_smac = "00:00:03:06:06:06"
            inner_smac = "00:00:02:06:06:06"

            # check VIP drop
            wrong_vip = "172.16.100.100"
            inner_pkt = simple_udpv6_packet(eth_dst="02:02:02:02:02:02",
                                          eth_src=self.eni_mac,
                                          ipv6_dst=self.dst_ca_ip,
                                          ipv6_src=src_vm_ip)
            vxlan_pkt = simple_vxlan_packet(eth_dst=self.our_mac,
                                            eth_src=outer_smac,
                                            ip_dst=wrong_vip,
                                            ip_src=self.src_vm_pa_ip,
                                            udp_sport=11638,
                                            with_udp_chksum=False,
                                            vxlan_vni=self.outbound_vni,
                                            inner_frame=inner_pkt)
            print("\n\nSending packet with wrong vip...\n\n", vxlan_pkt.__repr__())
            send_packet(self, 0, vxlan_pkt)
            print("\nVerifying drop...")
            verify_no_other_packets(self)

            # check routing drop
            wrong_dst_ca = "2000:bbbb::232"
            inner_pkt = simple_udpv6_packet(eth_dst="02:02:02:02:02:02",
                                          eth_src=self.eni_mac,
                                          ipv6_dst=wrong_dst_ca,
                                          ipv6_src=src_vm_ip)
            vxlan_pkt = simple_vxlan_packet(eth_dst=self.our_mac,
                                            eth_src=outer_smac,
                                            ip_dst=self.vip,
                                            ip_src=self.src_vm_pa_ip,
                                            udp_sport=11638,
                                            with_udp_chksum=False,
                                            vxlan_vni=self.outbound_vni,
                                            inner_frame=inner_pkt)
            print("\nSending packet with wrong dst CA IP to verify routing drop...\n\n", vxlan_pkt.__repr__())
            send_packet(self, 0, vxlan_pkt)
            print("\nVerifying drop...")
            verify_no_other_packets(self)

            # check mapping drop
            wrong_dst_ca = "2000:aaaa::d3d3"
            inner_pkt = simple_udpv6_packet(eth_dst="02:02:02:02:02:02",
                                          eth_src=self.eni_mac,
                                          ipv6_dst=wrong_dst_ca,
                                          ipv6_src=src_vm_ip)
            vxlan_pkt = simple_vxlan_packet(eth_dst=self.our_mac,
                                            eth_src=outer_smac,
                                            ip_dst=self.vip,
                                            ip_src=self.src_vm_pa_ip,
                                            udp_sport=11638,
                                            with_udp_chksum=False,
                                            vxlan_vni=self.outbound_vni,
                                            inner_frame=inner_pkt)
            print("\nSending packet with wrong dst CA IP to verify mapping drop...\n\n", vxlan_pkt.__repr__())
            send_packet(self, 0, vxlan_pkt)
            print("\nVerifying drop...")
            verify_no_other_packets(self)

            # check forwarding
            inner_pkt = simple_udpv6_packet(eth_dst="02:02:02:02:02:02",
                                          eth_src=self.eni_mac,
                                          ipv6_dst=self.dst_ca_ip,
                                          ipv6_src=src_vm_ip)
            vxlan_pkt = simple_vxlan_packet(eth_dst=self.our_mac,
                                            eth_src=outer_smac,
                                            ip_dst=self.vip,
                                            ip_src=self.src_vm_pa_ip,
                                            udp_sport=11638,
                                            with_udp_chksum=False,
                                            vxlan_vni=self.outbound_vni,
                                            inner_frame=inner_pkt)

            inner_exp_pkt = simple_udpv6_packet(eth_dst=self.dst_ca_mac,
                                          eth_src=self.eni_mac,
                                          ipv6_dst=self.dst_ca_ip,
                                          ipv6_src=src_vm_ip)
            vxlan_exp_pkt = simple_vxlan_packet(eth_dst="00:00:00:00:00:00",
                                            eth_src="00:00:00:00:00:00",
                                            ip_dst=self.dst_pa_ip,
                                            ip_src=self.vip,
                                            udp_sport=0, # TODO: Fix sport in pipeline
                                            with_udp_chksum=False,
                                            vxlan_vni=self.vnet_vni,
                                            inner_frame=inner_exp_pkt)
            # TODO: Fix IP chksum
            vxlan_exp_pkt[IP].chksum = 0
            # TODO: Fix UDP length
            vxlan_exp_pkt[IP][UDP][VXLAN].flags = 0

            self.pkt_exp = vxlan_exp_pkt
            print("\nSending outbound packet...\n\n", vxlan_pkt.__repr__())
            send_packet(self, 0, vxlan_pkt)
            print("\nVerifying packet...\n", self.pkt_exp.__repr__())
            verify_packet(self, self.pkt_exp, 0)
            print ("test_sai_thrift_outbound_udp_pkt_test OK")
        except AssertionError as ae:
            self.failure_teardown()
            raise ae

    def failure_teardown(self):
        # Delete entries which might be lingering from previous failures etc.; ignore failures here
        if hasattr(self, "ocpe"):
            status = sai_thrift_remove_outbound_ca_to_pa_entry(self.client, self.ocpe)
        if hasattr(self, "ore"):
            status = sai_thrift_remove_outbound_routing_entry(self.client, self.ore)
        #if hasattr(self, "out_acl_rule_id"):
        #    sai_thrift_remove_dash_acl_rule(self.client, self.out_acl_rule_id)
        if hasattr(self, "e2v"):
            sai_thrift_remove_outbound_eni_to_vni_entry(self.client, self.e2v)
        if hasattr(self, "eam"):
            sai_thrift_remove_eni_ether_address_map_entry(self.client, self.eam)
        if hasattr(self, "eni"):
            sai_thrift_remove_eni(self.client, self.eni)
        if hasattr(self, "vnet"):
            sai_thrift_remove_vnet(self.client, self.vnet)
        assert(status == SAI_STATUS_SUCCESS)                        
        if hasattr(self, "out_acl_group_id") and self.out_acl_group_id != SAI_NULL_OBJECT_ID:
            sai_thrift_remove_dash_acl_group(self.client, self.out_acl_group_id)
        if hasattr(self, "in_acl_group_id") and self.in_acl_group_id != SAI_NULL_OBJECT_ID:
            sai_thrift_remove_dash_acl_group(self.client, self.in_acl_group_id)
        if hasattr(self, "dle"):
            sai_thrift_remove_direction_lookup_entry(self.client, self.dle)
        if hasattr(self, "vpe"):
            sai_thrift_remove_vip_entry(self.client, self.vpe)
        self.cleaned_up = True

    def tearDown(self):

        # Delete in reverse order
        if self.cleaned_up:
            return
        status = sai_thrift_remove_outbound_ca_to_pa_entry(self.client, self.ocpe)
        assert(status == SAI_STATUS_SUCCESS)                        

        status = sai_thrift_remove_outbound_routing_entry(self.client, self.ore)
        assert(status == SAI_STATUS_SUCCESS)

        #status = sai_thrift_remove_dash_acl_rule(self.client, self.out_acl_rule_id)
        #assert(status == SAI_STATUS_SUCCESS)

        status = sai_thrift_remove_eni_ether_address_map_entry(self.client, self.eam)
        assert(status == SAI_STATUS_SUCCESS)

        status = sai_thrift_remove_eni(self.client, self.eni)
        assert(status == SAI_STATUS_SUCCESS)

        status = sai_thrift_remove_vnet(self.client, self.vnet)
        assert(status == SAI_STATUS_SUCCESS)                        

        status = sai_thrift_remove_dash_acl_group(self.client, self.out_acl_group_id)
        assert(status == SAI_STATUS_SUCCESS)

        status = sai_thrift_remove_dash_acl_group(self.client, self.in_acl_group_id)
        assert(status == SAI_STATUS_SUCCESS)

        status = sai_thrift_remove_direction_lookup_entry(self.client, self.dle)
        assert(status == SAI_STATUS_SUCCESS)

        sai_thrift_remove_vip_entry(self.client, self.vpe)
        assert(status == SAI_STATUS_SUCCESS)

[mukeshmv]

the v6 test case just hangs

@mhanif seems to me to be a PTF thrift client side issue at setup time where it gets blocked before invoking SAI thrift API. As soon as we send a SIGINT (Ctrl-C) to the client, it seems to get unblocked and sends the request to the thrift server just before quitting. We can see that the thrift server correctly programs the P4 table. @reshmaintel @vmytnykx can you take a look ?

[mhanif]

Thanks @mukeshmv. Are we good to merge this PR and we can checkin the test case as a separate PR?

[mukeshmv]

LGTM

[chrispsommers]

As agreed in the WG meeting on Sept. 21 2022, we allowed the remainder of last week for reviews. None appeared, we will merge now.