diff --git a/ansible/doc/README.testbed.Routing.md b/ansible/doc/README.testbed.Routing.md new file mode 100644 index 00000000000..4f5cdcfee33 --- /dev/null +++ b/ansible/doc/README.testbed.Routing.md @@ -0,0 +1,167 @@ +# Testbed Routing Design + +This document discusses the BGP routing design in the Testbed. + +``` + +------+ + + VM +---------+ + +------+ | + | + +------+ | + + VM +---------+ + +------+ | ++-------+ +---+---+ +| PTF +------------------+ DUT | ++-------+ +---+---+ + +------+ | + + VM +---------+ + +------+ | + | + +------+ | + + VM +---------+ + +------+ +``` + +In this topology, VMs (vEOS) act as the BGP neighbors of the DUT. VMs generate and advertise BGP routes to the DUT. +This approach has several issues: +- Difficult to generate arbitary routes from the vEOS, e.g., write complex route-map and filter to generate the needed routes. +- Consume lots of memory in the vEOS +- Rules are specific the NOS. If we plan to switch VM to SONiC, we need to rewrite all the route-maps. + + +``` + +------+ + +---------+ VM +---------+ + | +------+ | + | | + | +------+ | + +---------+ VM +---------+ + | +------+ | ++---+---+ +---+---+ +| PTF | | DUT | ++---+---+ +---+---+ + | +------+ | + +---------+ VM +---------+ + | +------+ | + | | + | +------+ | + +---------+ VM +---------+ + +------+ +``` + +The new approach is use VM as a pass-through device. We run exabgp instances on PTF docker, +and exabgp advertise the routes to the VM, and VM then re-advertise the routes to DUT. +This approach has several advantages +- VM template becomes much more simplified. Only basic interface, lag, BGP configuration. +- VM memory consumption is smaller. +- Exabgp can generate complex routes. +- Easy to support different NOS as neigbhor devices, for example SONiC VM. + +## Implementation details: +- backplane bridge to physically connect ptf docker and vm. bridge ```the br-b-{{ testbed name }}``` is created +on the host to connect the eos and ptf via backplane network. + +``` +br-b-vms6-1 8000.72bb0a6ad08c no VM0100-back + VM0101-back + VM0102-back + VM0103-back + ptf-vms6-1-b +``` + +- eos vm backplane interface. ```Et5``` is created in eos as the backplane interface. + +``` +ARISTA01T1#show int status +Port Name Status Vlan Duplex Speed Type Flags Encapsulation +Et1 connected in Po1 full unconf EbraTestPhyPort +Et2 connected 1 full unconf EbraTestPhyPort +Et3 connected 1 full unconf EbraTestPhyPort +Et4 connected 1 full unconf EbraTestPhyPort +Et5 backplane connected routed full unconf EbraTestPhyPort +Ma0 connected routed full 10G 10/100/1000 +Po1 connected routed full unconf N/A +``` + +- ptf backplane interface. ```backplane``` interface is created in ptf docker. + +``` +root@6884a0fcd031:~# ip addr show backplane +1359: backplane@if1360: mtu 1500 qdisc noqueue state UP group default qlen 1000 + link/ether 96:1f:0e:1f:fc:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.10.246.254/24 scope global backplane + valid_lft forever preferred_lft forever + inet6 fc0a::ff/64 scope global + valid_lft forever preferred_lft forever +``` + +- eos bgp connection to exabgp. ```exabgp_v4``` and ```exabgp_v6``` (not shown below) is created in eos to connect with exabgp. +The bgp connection is iBGP connection. + +``` +ARISTA01T1#show ip bgp sum +BGP summary information for VRF default +Router identifier 100.1.0.29, local AS number 64600 +Neighbor Status Codes: m - Under maintenance + Description Neighbor V AS MsgRcvd MsgSent InQ OutQ Up/Down State PfxRcd PfxAcc + 65100 10.0.0.56 4 65100 5081 5080 0 0 1d07h Estab 2 2 + exabgp_v4 10.10.246.254 4 64600 8277 1880 0 0 1d07h Estab 6399 6399 +``` + +- exabgp ansible module to control exabgp service in PTF docker. You can use ```supervisorctl``` to check the exabgp service status. + - create, remove, restart exabgp service using supervisord. + - each exabgp instance listen to a http port for route announce/withdraw + +``` +root@6884a0fcd031:~# supervisorctl status +exabgp-ARISTA01T1 RUNNING pid 118, uptime 2 days, 8:16:45 +exabgp-ARISTA01T1-v6 RUNNING pid 131, uptime 2 days, 8:16:42 +exabgp-ARISTA02T1 RUNNING pid 92, uptime 2 days, 8:16:53 +exabgp-ARISTA02T1-v6 RUNNING pid 105, uptime 2 days, 8:16:49 +exabgp-ARISTA03T1 RUNNING pid 66, uptime 2 days, 8:17:00 +exabgp-ARISTA03T1-v6 RUNNING pid 79, uptime 2 days, 8:16:57 +exabgp-ARISTA04T1 RUNNING pid 40, uptime 2 days, 8:17:07 +exabgp-ARISTA04T1-v6 RUNNING pid 53, uptime 2 days, 8:17:04 +ptf_nn_agent RUNNING pid 14, uptime 2 days, 10:35:21 +sshd RUNNING pid 13, uptime 2 days, 10:35:21 +``` + +- pytest fib fixture to generate routes to exabgp + - currently support t0, t1-lag, extending to other topology is straighforward + - fib module generate http request to exabgp instance + +## Future applications: +- multi-path relax test. pytest will instruct multiple exabgp instances to advertise +same VIP to different T0 VM. +- add/remove routes test. pytest will control exabgp to advertise/withdraw routes + +## How to use exabgp module + +start an exabgp instance +```python + ptfhost.exabgp(name=k, + state="started", \ + router_id = 10.0.0.1, \ + local_ip = 10.0.0.1, \ + peer_ip = 10.0.0.2, \ + local_asn = 65100, \ + peer_asn = 65100, \ + port = 6000) +``` + +use a fib fixture. The fixture will detect your testbed type and then generate routes based your testbed type. + +```python +import pytest + +def test_announce_routes(fib): + """Simple test case that utilize fib to announce route in order to a newly setup test bed receive + BGP routes from remote devices + """ + assert True +``` + +## Q&A +Q: Why not use exabgp to advertise routes directly to the DUT? +A: Yes, we can. But, we could not simulate the BGP over LAG as there is no LAG protocol +running inside the PTF docker.