From d1cb0657fee50b59abaa19acdd8e32fc62ac10c5 Mon Sep 17 00:00:00 2001 From: Guohan Lu Date: Fri, 21 Jul 2017 06:33:41 +0000 Subject: [PATCH 1/2] split testbed into mulitple sections --- ansible/README.testbed.md | 421 +-------------------- ansible/doc/README.testbed.Cli.md | 50 +++ ansible/doc/README.testbed.Config.md | 43 +++ ansible/doc/README.testbed.FAQ.md | 13 + ansible/doc/README.testbed.Internal.md | 129 +++++++ ansible/doc/README.testbed.Overview.md | 56 +++ ansible/doc/README.testbed.Setup.md | 93 +++++ ansible/doc/README.testbed.Topology.md | 65 ++++ ansible/{ => doc}/img/testbed-direct.png | Bin ansible/{ => doc}/img/testbed-injected.png | Bin ansible/{ => doc}/img/testbed-ptf32.png | Bin ansible/{ => doc}/img/testbed-ptf64.png | Bin ansible/doc/img/testbed-server.png | Bin 0 -> 21083 bytes ansible/{ => doc}/img/testbed-t0.png | Bin ansible/{ => doc}/img/testbed-t1-lag.png | Bin ansible/{ => doc}/img/testbed-t1.png | Bin ansible/{ => doc}/img/testbed.png | Bin ansible/img/testbed-server.png | Bin 3291 -> 0 bytes 18 files changed, 458 insertions(+), 412 deletions(-) create mode 100644 ansible/doc/README.testbed.Cli.md create mode 100644 ansible/doc/README.testbed.Config.md create mode 100644 ansible/doc/README.testbed.FAQ.md create mode 100644 ansible/doc/README.testbed.Internal.md create mode 100644 ansible/doc/README.testbed.Overview.md create mode 100644 ansible/doc/README.testbed.Setup.md create mode 100644 ansible/doc/README.testbed.Topology.md rename ansible/{ => doc}/img/testbed-direct.png (100%) rename ansible/{ => doc}/img/testbed-injected.png (100%) rename ansible/{ => doc}/img/testbed-ptf32.png (100%) rename ansible/{ => doc}/img/testbed-ptf64.png (100%) create mode 100644 ansible/doc/img/testbed-server.png rename ansible/{ => doc}/img/testbed-t0.png (100%) rename ansible/{ => doc}/img/testbed-t1-lag.png (100%) rename ansible/{ => doc}/img/testbed-t1.png (100%) rename ansible/{ => doc}/img/testbed.png (100%) delete mode 100644 ansible/img/testbed-server.png diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index ec617a25d60..47c7f6fe9a2 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -1,412 +1,9 @@ -# Requirements for the Linux Host -1. Ubuntu 16.04 x64 -2. Installed docker-engine and python (ansible requires python 2.7) -3. Three network cards: - * first is used for the server management - * second is used to connect management interfaces of VMs and docker containers to a network. - * third is used to connect VMs and ptf containers to DUTs frontal panel ports - -# Hardware SKU -1. Server SKU (this is what we are using, not mandatory): Dell 730; 2 CPUs each has 18 cores; 192G memory; hard disk:2X500G -2. Fanout switch: Arista 7260 or similar -3. NIC: Mellanox MT27700 Family CX4 - -# Content of /etc/network/interfaces: -``` -root@STR-AZURE-SERV-01:~# cat /etc/network/interfaces -# The primary network interface - testbed server management -auto em1 -iface em1 inet static - address 10.250.0.245 - netmask 255.255.255.0 - network 10.250.0.0 - broadcast 10.250.0.255 - gateway 10.250.0.1 - dns-nameservers 10.250.0.1 10.250.0.2 - # dns-* options are implemented by the resolvconf package, if installed - dns-search SOMECOMPANY - -# VM and dockers management interfaces -auto br1 -iface br1 inet manual - bridge_ports em2 - bridge_stp off - bridge_maxwait 0 - bridge_fd 0 - -# DUTs frontpanel ports -auto p4p1 -iface p4p1 inet manual - mtu 9216 -up ip link set p4p1 up -``` - -# SONiC testbed physical topology - -![](img/testbed.png) - -1. Every DUT port is connected to one of leaf fanout switches -2. Every leaf fanout switch has unique vlan tag for every DUT port -3. Root fanout switch connects leaf fanout switches and testbed servers -4. Connections from root fanout switches are 802.1Q trunks -5. Any testbed server can access any DUT port by sending a packet with the port vlan tag (root fanout switch should have this vlan number enabled on the server trunk) - -# Fanout switch configuration -*Fanout switch*: A physical switch which enables VLAN trunking. (SKU: Arista 7260 or similar) - * Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. - * Et1-Et32 are vlan access ports and are connect to DUT. - * Enable LACP/LLDP passthrough - * Disable spanning tree protocol - -# Testbed server connections - -![](img/testbed-server.png) - -1. The testbed server has 3 network ports: - 1. Trunk port to root fanout switch - 2. Server management port to manage the server - 3. Testbed management port to manage VMs and PTFs containers on the server -2. VMs are created right after the server starts -3. VMs connections and PTF containers are created when a new topology is being added - -# VMs -The VMs use Arista vEOS. Each VM has 10 network interfaces: - -1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port). -2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above). -3. 1 management port. This port is used to connect to the VMs - -# Topologies - -1. Configuration of a testbed topology is defined in one file: testbed.csv -2. One script to operate all testbeds: testbed-cli.sh -3. Flexible topologies which allow to use vm_set and ptf container as one entity -4. All VM management ip information in one place: veos inventory file -5. ptf container is generalized and used in every topology -6. Automatic provisioning of fanout switch configuration (should be refactored) -7. Every VM uses 2G of RAM - -# Testbed topology configuration - -1. One entry in testbed.csv -2. Consist of: - 1. physical topology: How ports of VMs and ptf connected to DUT - 2. configuration templates for VMs -3. Defined in vars/topo_*.yml files -4. Current topologies are: - 1. t1: 32 VMs + ptf container for injected ports - 2. t1-lag: 24 VMs + ptf container for injected ports. 8 VMs has two ports each in LAG - 3. ptf32: classic ptf container with 32 ports connected directly to DUT ports - 4. ptf64: as ptf32, but with 64 ports - 5. t0: 4 VMs + ptf. ptf container has 4 injected ports + 28 directly connected ports - -# testbed.csv -``` -# conf-name,group-name,topo,ptf_image_name,ptf_mgmt_ip,server,vm_base,dut,comment -ptf1-m,ptf1,ptf32,docker-ptf-sai-mlnx,10.255.0.188/24,server_1,,str-msn2700-01,Tests ptf -vms-t1,vms1-1,t1,docker-ptf-sai-mlnx,10.255.0.178/24,server_1,VM0100,str-msn2700-01,Tests vms -vms-t1-lag,vms1-1,t1-lag,docker-ptf-sai-mlnx,10.255.0.178/24,server_1,VM0100,str-msn2700-01,Tests vms - -``` - -1. uniq-name - to address row in table -2. testbed-name – used in interface names, up to 8 characters -3. topo – name of topology -4. ptf_imagename – defines ptf image -5. ptf_mgmt_ip – ip address for mgmt interface of ptf container -6. server – server where the testbed resides -7. vm_base – first VM for the testbed. If empty, no VMs are used -8. DUT – target dut name -9. Comment – any text here - -# testbed-cli.sh - -1. Maintenance purposes only - - ./testbed-cli.sh start-vms {server_name} ~./password # after a server restarted - - ./testbed-cli.sh stop-vms {server_name} ~./password # before a server restarted -2. General usage - - ./testbed-cli.sh add-topo {topo_name} ~./password # create topo with name {topo_name} from testbed.csv - - ./testbed-cli.sh remove-topo {topo_name} ~./password # destroy topo with name {topo_name} from testbed.csv - - ./testbed-cli.sh renumber-topo {topo_name} ~./password # renumber topo with name {topo_name} from testbed.csv - -# Current topologies - -## t1 - -![](img/testbed-t1.png) - - - Requires 32 VMs - - All DUT ports are connected to VMs - - PTF container has injected ports only - -## t1-lag - -![](img/testbed-t1-lag.png) - - - Requires 24 VMs - - All DUT ports are connected to VMs - - PTF container has injected ports only - -## ptf32 - -![](img/testbed-ptf32.png) - - - Requires 0 VMs - - All DUT ports are directly connected to PTF container - - PTF container has no injected ports - -## ptf64 - -![](img/testbed-ptf64.png) - - - Requires 0 VMs - - All DUT ports are directly connected to PTF container - - PTF container has no injected ports - -## t0 - -![](img/testbed-t0.png) - - - Requires 4 VMs - - 4 DUT ports are connected to VMs - - PTF container has 4 injected ports and 28 directly connected ports - -# Direct interface vs injected interface - -## Direct -![](img/testbed-direct.png) - -DUT front panel port is directly connected to one of ptf container ports. Usually eth0 port of ptf container connects Ethernet0 port of DUT, eth1 port of ptf container connects Ethernet4 port of DUT and so on. This is usually used in ptf topologies to connect DUT ports to ptf container ports. - -## Injected -![](img/testbed-injected.png) - -DUT front panel port is directly connected to one of VMs interfaces. But also we have a tap into this connection. Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time. - -# testbed-cli.sh – Add/Remove topo -``` -# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner -vms1-1-t1,vms1-1,t1,docker-ptf-sai-mlnx,10.0.10.5/23,server_1,VM0100,str-msn2700-11,t1 tests -vms1-1-t1-lag,vms1-1,t1-lag,docker-ptf-sai-mlnx,10.0.10.5/23,server_1,VM0100,str-msn2700-11,t1-lag tests - -``` -Goal is to use one VM with different topologies - -1. To add a new testbed “vms1-1-t1”: - - ./testbed-cli add-topo vms1-1-t1 ~/.password - -2. To switch from testbed “vms1-1-t1” to testbed “vms1-1-lag” - - ./testbed-cli remove-topo vms1-1-t1 ~/.password - - ./testbed-cli add-topo vms1-1-t1-lag ~/.password - -Feature: The VMs configuration will be updated while switching from one topo to another -Feature: Might be used for renumbering too -Caveat: Have to remember what was the initial topology. Should be fixed in future - -# testbed-cli.sh – Renumber topo -``` -# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner -vms2-2-b,vms2-2,t1,docker-ptf-sai-brcm,10.0.10.7/23,server_1,VM0100,str-d6000-05,brcm test -vms2-2-m,vms2-2,t1,docker-ptf-sai-mlnx,10.0.10.7/23,server_1,VM0100,str-msn2700-5,mlnx test - -``` -Goal is to use one VM set against different DUTs - -1. To add a new testbed “vms2-2-b”: - - ./testbed-cli add-topo vms2-2-b ~/.password - -2. To switch from testbed “vms2-2-b” to testbed “vms2-2-m” - - ./testbed-cli renumber-topo vms2-2-m ~/.password - -Feature: The VMs configuration will NOT be updated while switching from one topo to another (faster). - -TODO: check topo field when renumbering between topologies - -# testbed.csv Consistency rules -``` -# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner -vms2-2-b,vms2-2,t1,docker-ptf-sai-brcm,10.0.10.7/23,server_1,VM0100,str-d6000-05,brcm test -vms2-2-m,vms2-2,t1,docker-ptf-sai-mlnx,10.0.10.7/23,server_1,VM0100,str-msn2700-5,mlnx test - -``` -Must be strictly checked in code reviews - - uniq-name must be unique - - All testbed records with the same testbed-name must have the same: - - ptf_ip - - server - - vm_base - - testbed-name must be up to 8 characters long - - topo name must be valid (topo registered in veos and topo file presented in vars/topo_*.yml - - ptf_imagename must be valid - - server name must be valid and presented in veos inventory file - - vm_base must not overlap with testbeds from different groups (different test-name) - -TODO: check this constraints in testbed-cli.sh - -# How to find IP addresses of VMs and PTF - - IP address of testbed ptf container could be found in testbed.csv - - To find some VM IP address: - - find vm_offset parameter for the VM in your topology file - - find vm_base parameter in testbed.csv - - Calculate physical VM name as vm_base + vm_offset - - Find physical VM entry in veos file - -TODO: Create ansible playbook for this - -# Example of setting up the testbed and deploying a topology -## Start VMs -1. clone sonic-mgmt repo (https://github.com/Azure/sonic-mgmt.git) to local directory -2. Add/Update your testbed server management IP in veos file. Example:'STR-ACS-SERV-01 ansible_host=10.0.0.5' where 10.0.0.5 your server mgmt ip -3. Add testbed server credentials in ansible/group_vars/vm_host/creds.yml -4. Check that ansible could reach this device by command 'ansible -m ping -i veos vm_host_1' -5. Put files: Aboot-veos-serial-8.0.0.iso and vEOS-lab-4.15.9M.vmdk to /home/{your_username from step 3}/veos-vm/images on your testbed server -6. Edit ansible/host_vars/STR-ACS-SERV-01.yml. You need to change external_iface, mgmt_gw and mgmt_prefixlen. These settings define network parameters for VM/ptf management interfaces. Example: -``` -external_iface: eno2   <--- your interface which will be used for -VM mgmt interfaces - -mgmt_gw: 10.64.246.1   <--- ip of gateway for VM mgmt interfaces - -mgmt_prefixlen: 23     <--- prefixlen for management -interfaces -``` -7. Add ip addresses for your VMs in veos inventory file ansible/veos inventory file. These ip addresses should be at the same network as parameters in step 6 -8. Update VM credentials in ansible/group_vars/eos/creds.yml. Use root:123456 as credentials -9. Start initial deploy VMs by: bash ./testbed-cli.sh start-vms server_1 ~/.password -10. Check that all VMs are up and running: ansible -m ping -i veos server_1 - -## Deploy topology -1. Add information about your docker registry here: vars/docker_registry.yml -2. Update testbed.csv with your data. At least update ptf mgmt interface settings -3. To deploy topology run: ./testbed-cli.sh add-topo ptf1-m ~/.password -4. To remove topology run: ./testbed-cli.sh remove-topo ptf1-m ~/.password - -# Testbed internals - -## Topology definition - - - List of currently defined topologies in veos inventory file -``` -[servers:vars] -topologies=[‘t1', ‘t1-lag', 't0', 'ptf32', 'ptf64'] -``` - - Topologies stored inside of vars/topo_*.yml, where * is a topology name - - Configuration templates for the topologies saved in roles/eos/templates/*.yml - -## Topology file - - - Topology file is a regular ansible yaml file with variables: - - topology – defines physical topology - - configuration – defines variables for VMs configuration templates - - configuration_properties – defines group variables for VMs configuration templates - - - topology dictionary is required - - configuration and configuration_properties are optional and used only for topologies with VMs - -## Topology file. topology dictionary - - - Two dictionaries: - - host_interface – defines a list of port offsets which would be inserted into the ptf container - - VMs – defines a list and a physical configuration of VMs used in topology - -``` -topology: - host_interfaces: - - 0 - - 1 - VMs: - ARISTA01T1: - vlans: - - 2 - - 3 - vm_offset: 0 - ARISTA02T1: - vlans: - - 4 - vm_offset: 1 - -``` - - ARISTA01T1 – hostname for a VM - - vlans - list of vlan offsets used in VM - - vm_offset – offset of VM with base configured as vm_base in testbed.csv - - - In this example: - - Let’s consider: vm_base == VM0100, vlan_base == ‘100’ - - First VM: - - hostname ARISTA01T1 - - Uses VM with physical name VM0100 - - Ethernet1 is connected to vlan 102 - - Ethernet2 is connected to vlan 103 - - Ethernet9 is connected to Ethernet9 of ARISTA02T1 - - Second VM: - - hostname ARISTA02T1 - - Uses VM with physical name VM0101 (vm_offset: 1 + vm_base: VM0100) - - Ethernet1 is connected to vlan 104 - - Ethernet9 is connected to Ethernet9 of ARISTA01T1 - - ptf container: - - 5 ethernet interfaces: - - eth0 is directly connected to DUT. vlan 100 - - eth1 is directly connected to DUT. vlan 101 - - eth2 is injected interface for Ethernet1 of VM ARISTA01T1 - - eth3 is injected interface for Ethernet2 of VM ARISTA01T1 - - eth4 is injected interface for Ethernet1 of VM ARISTA02T1 - -## Topology file. configuration_properties -``` -configuration_properties: - common: - nhipv4: 10.10.246.100 - nhipv6: FC0A::C9 - spine: - swrole: spine - podset_number: 200 - tor_number: 16 - tor_subnet_number: 2 - leaf_asn_start: 62001 - tor_asn_start: 65501 - failure_rate: 0 - tor: - swrole: tor - tor_subnet_number: 5 - -``` - - Configuration properties contains any number of dictionary entries - - You could have as many as you want - - Lately you can refer to these entries in your configuration dictionary. See entry “properties” - - You could use them as {{ props.property_name }} inside of jinja2 template. Example: {% for tor in range(0, props.tor_number) %} - -## Topology file. configuration -``` -configuration: - ARISTA01T2: - properties: - - common - - spine - bgp: - asn: 65200 - peers: - 65100: - - 10.0.0.0 - - FC00::1 - interfaces: - Loopback0: - ipv4: 100.1.0.1/32 - ipv6: 2064:100::1/128 - Ethernet1: - ipv4: 10.0.0.1/31 - ipv6: fc00::2/126 - Ethernet9: - ipv4: 10.10.246.1/24 - ipv6: fc0a::2/64 - -``` - - Configurations contains any number of dictionary entries - - You could have as many as you want - - You have to have entry properties when you want to bring some common property into the configuration - - You could use configuration as {{configuration[hostname].property }} inside of jinja2 template. - - Example: -``` -{% set host = configuration[hostname] %} -{% for name, iface in host['interfaces'].items() %} -``` +# SONiC Testbed + +- [Overview](doc/README.testbed.Overview.md) +- [Setup](doc/README.testbed.Setup.md) +- [Topology](doc/README.testbed.Topology.md) +- [Configuration](doc/README.testbed.Config.md) +- [Command Line](doc/README.testbed.Cli.md) +- [FAQ](doc/README.testbed.FAQ.md) +- [Internal](doc/README.testbed.Internal.md) diff --git a/ansible/doc/README.testbed.Cli.md b/ansible/doc/README.testbed.Cli.md new file mode 100644 index 00000000000..dd0dabba1bc --- /dev/null +++ b/ansible/doc/README.testbed.Cli.md @@ -0,0 +1,50 @@ +# Testbed Command Line + +```testbed-cli.sh``` is the command line to setup/teardown the testbed, as well as add/remove/switch topology. + +- Maintenance purposes only + - ```./testbed-cli.sh start-vms {server_name} ~./password``` # after a server restarted + - ```./testbed-cli.sh stop-vms {server_name} ~./password``` # before a server restarted +- General usage + - ```./testbed-cli.sh add-topo {topo_name} ~./password``` # create topo with name {topo_name} from testbed.csv + - ```./testbed-cli.sh remove-topo {topo_name} ~./password``` # destroy topo with name {topo_name} from testbed.csv + - ```./testbed-cli.sh renumber-topo {topo_name} ~./password``` # renumber topo with name {topo_name} from testbed.csv + +## Add/Remove topo +``` +# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner +vms1-1-t1,vms1-1,t1,docker-ptf-sai-mlnx,10.0.10.5/23,server_1,VM0100,str-msn2700-11,t1 tests +vms1-1-t1-lag,vms1-1,t1-lag,docker-ptf-sai-mlnx,10.0.10.5/23,server_1,VM0100,str-msn2700-11,t1-lag tests + +``` +Goal is to use one VM with different topologies + +- To add a new testbed “vms1-1-t1”: + - ./testbed-cli add-topo vms1-1-t1 ~/.password + +- To switch from testbed “vms1-1-t1” to testbed “vms1-1-lag” + - ./testbed-cli remove-topo vms1-1-t1 ~/.password + - ./testbed-cli add-topo vms1-1-t1-lag ~/.password + +Feature: The VMs configuration will be updated while switching from one topo to another +Feature: Might be used for renumbering too +Caveat: Have to remember what was the initial topology. Should be fixed in future + +# Renumber topo +``` +# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner +vms2-2-b,vms2-2,t1,docker-ptf-sai-brcm,10.0.10.7/23,server_1,VM0100,str-d6000-05,brcm test +vms2-2-m,vms2-2,t1,docker-ptf-sai-mlnx,10.0.10.7/23,server_1,VM0100,str-msn2700-5,mlnx test + +``` +Goal is to use one VM set against different DUTs + +- To add a new testbed “vms2-2-b”: + - ./testbed-cli add-topo vms2-2-b ~/.password + +- To switch from testbed “vms2-2-b” to testbed “vms2-2-m” + - ./testbed-cli renumber-topo vms2-2-m ~/.password + +Feature: The VMs configuration will NOT be updated while switching from one topo to another (faster). + +TODO: check topo field when renumbering between topologies diff --git a/ansible/doc/README.testbed.Config.md b/ansible/doc/README.testbed.Config.md new file mode 100644 index 00000000000..db6c8f9e91e --- /dev/null +++ b/ansible/doc/README.testbed.Config.md @@ -0,0 +1,43 @@ +# Testbed Configuration + +```testbed.csv``` is the topology configuration file for the testbed. + +# ```testbed.csv``` +``` +# conf-name,group-name,topo,ptf_image_name,ptf_mgmt_ip,server,vm_base,dut,comment +ptf1-m,ptf1,ptf32,docker-ptf-sai-mlnx,10.255.0.188/24,server_1,,str-msn2700-01,Tests ptf +vms-t1,vms1-1,t1,docker-ptf-sai-mlnx,10.255.0.178/24,server_1,VM0100,str-msn2700-01,Tests vms +vms-t1-lag,vms1-1,t1-lag,docker-ptf-sai-mlnx,10.255.0.178/24,server_1,VM0100,str-msn2700-01,Tests vms + +``` + +- conf-name - to address row in table +- group-name – used in interface names, up to 8 characters +- topo – name of topology +- ptf_imagename – defines PTF image +- ptf_mgmt_ip – ip address for mgmt interface of PTF container +- server – server where the testbed resides +- vm_base – first VM for the testbed. If empty, no VMs are used +- dut – target dut name +- comment – any text here + +# ```testbed.csv``` consistency rules +``` +# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner +vms2-2-b,vms2-2,t1,docker-ptf-sai-brcm,10.0.10.7/23,server_1,VM0100,str-d6000-05,brcm test +vms2-2-m,vms2-2,t1,docker-ptf-sai-mlnx,10.0.10.7/23,server_1,VM0100,str-msn2700-5,mlnx test + +``` +Must be strictly checked in code reviews + - uniq-name must be unique + - All testbed records with the same testbed-name must have the same: + - ptf_ip + - server + - vm_base + - testbed-name must be up to 8 characters long + - topo name must be valid (topo registered in ```veos``` and topo file presented in vars/topo_*.yml + - ptf_imagename must be valid + - server name must be valid and presented in veos inventory file + - vm_base must not overlap with testbeds from different groups (different test-name) + +TODO: check this constraints in testbed-cli.sh diff --git a/ansible/doc/README.testbed.FAQ.md b/ansible/doc/README.testbed.FAQ.md new file mode 100644 index 00000000000..a5a9d309af9 --- /dev/null +++ b/ansible/doc/README.testbed.FAQ.md @@ -0,0 +1,13 @@ +# FAQ + +## How to find IP addresses of VMs and PTF + - IP address of testbed PTF container could be found in testbed.csv + - To find some VM IP address: + - find vm_offset parameter for the VM in your topology file + - find vm_base parameter in testbed.csv + - Calculate physical VM name as vm_base + vm_offset + - Find physical VM entry in veos file + +TODO: Create ansible playbook for this + + diff --git a/ansible/doc/README.testbed.Internal.md b/ansible/doc/README.testbed.Internal.md new file mode 100644 index 00000000000..fb0c057c46a --- /dev/null +++ b/ansible/doc/README.testbed.Internal.md @@ -0,0 +1,129 @@ +# Testbed internals + +## Topology definition + + - List of currently defined topologies in veos inventory file +``` +[servers:vars] +topologies=[‘t1', ‘t1-lag', 't0', 'ptf32', 'ptf64'] +``` + - Topologies stored inside of vars/topo_*.yml, where * is a topology name + - Configuration templates for the topologies saved in roles/eos/templates/*.yml + +## Topology file + + - Topology file is a regular ansible yaml file with variables: + - topology – defines physical topology + - configuration – defines variables for VMs configuration templates + - configuration_properties – defines group variables for VMs configuration templates + + - topology dictionary is required + - configuration and configuration_properties are optional and used only for topologies with VMs + +## Topology file. topology dictionary + + - Two dictionaries: + - host_interface – defines a list of port offsets which would be inserted into the PTF container + - VMs – defines a list and a physical configuration of VMs used in topology + +``` +topology: + host_interfaces: + - 0 + - 1 + VMs: + ARISTA01T1: + vlans: + - 2 + - 3 + vm_offset: 0 + ARISTA02T1: + vlans: + - 4 + vm_offset: 1 + +``` + - ARISTA01T1 – hostname for a VM + - vlans - list of vlan offsets used in VM + - vm_offset – offset of VM with base configured as vm_base in testbed.csv + + - In this example: + - Let’s consider: vm_base == VM0100, vlan_base == ‘100’ + - First VM: + - hostname ARISTA01T1 + - Uses VM with physical name VM0100 + - Ethernet1 is connected to vlan 102 + - Ethernet2 is connected to vlan 103 + - Ethernet9 is connected to Ethernet9 of ARISTA02T1 + - Second VM: + - hostname ARISTA02T1 + - Uses VM with physical name VM0101 (vm_offset: 1 + vm_base: VM0100) + - Ethernet1 is connected to vlan 104 + - Ethernet9 is connected to Ethernet9 of ARISTA01T1 + - PTF container: + - 5 ethernet interfaces: + - eth0 is directly connected to DUT. vlan 100 + - eth1 is directly connected to DUT. vlan 101 + - eth2 is injected interface for Ethernet1 of VM ARISTA01T1 + - eth3 is injected interface for Ethernet2 of VM ARISTA01T1 + - eth4 is injected interface for Ethernet1 of VM ARISTA02T1 + +## Topology file. configuration_properties +``` +configuration_properties: + common: + nhipv4: 10.10.246.100 + nhipv6: FC0A::C9 + spine: + swrole: spine + podset_number: 200 + tor_number: 16 + tor_subnet_number: 2 + leaf_asn_start: 62001 + tor_asn_start: 65501 + failure_rate: 0 + tor: + swrole: tor + tor_subnet_number: 5 + +``` + - Configuration properties contains any number of dictionary entries + - You could have as many as you want + - Lately you can refer to these entries in your configuration dictionary. See entry “properties” + - You could use them as {{ props.property_name }} inside of jinja2 template. Example: {% for tor in range(0, props.tor_number) %} + +## Topology file. configuration +``` +configuration: + ARISTA01T2: + properties: + - common + - spine + bgp: + asn: 65200 + peers: + 65100: + - 10.0.0.0 + - FC00::1 + interfaces: + Loopback0: + ipv4: 100.1.0.1/32 + ipv6: 2064:100::1/128 + Ethernet1: + ipv4: 10.0.0.1/31 + ipv6: fc00::2/126 + Ethernet9: + ipv4: 10.10.246.1/24 + ipv6: fc0a::2/64 + +``` + - Configurations contains any number of dictionary entries + - You could have as many as you want + - You have to have entry properties when you want to bring some common property into the configuration + - You could use configuration as {{configuration[hostname].property }} inside of jinja2 template. + + Example: +``` +{% set host = configuration[hostname] %} +{% for name, iface in host['interfaces'].items() %} +``` diff --git a/ansible/doc/README.testbed.Overview.md b/ansible/doc/README.testbed.Overview.md new file mode 100644 index 00000000000..7ff5ce585ad --- /dev/null +++ b/ansible/doc/README.testbed.Overview.md @@ -0,0 +1,56 @@ +# SONiC Testbed Overview + +This document gives overview of the SONiC testbed. + +## Physical topology + +![](img/testbed.png) + +1. Every DUT port is connected to one of leaf fanout switches +2. Every leaf fanout switch has unique vlan tag for every DUT port +3. Root fanout switch connects leaf fanout switches and testbed servers +4. Connections from root fanout switches are 802.1Q trunks +5. Any testbed server can access any DUT port by sending a packet with the port vlan tag (root fanout switch should have this vlan number enabled on the server trunk) + +## Fanout switch +*Fanout switch*: A physical switch which enables VLAN trunking. + * Hardware SKU: Arista 7260 or similar + * Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. + * Et1-Et32 are vlan access ports and are connect to DUT. + * Enable LACP/LLDP passthrough + * Disable spanning tree protocol + +## Testbed server + +- Server SKU (this is what we are using, not mandatory): Dell 730; 2 CPUs each has 18 cores; 192G memory; hard disk:2X500G +- NIC: Mellanox MT27700 Family CX4 + +![](img/testbed-server.png) + +### Network connections + +- The testbed server has 2 network ports: + - Trunk port (Mellanox MT27700 Family CX4) to root fanout switch + - Server management port to manage the server, VMs and PTF containers on the server + +### VMs + +The VMs use Arista vEOS. They are using to setup protocol test such as BGP, LACP, LLDP. They are created using ```testbed-cli.sh start-vms``` command. Each VM uses 2G of RAM and has 10 network interfaces: + +1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port). +2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above). +3. 1 management port. This port is used to connect to the VMs + +### PTF + +PTF container is used to send and receive data packets to validate the DUT data plane. + +#### PTF with direct port +![](img/testbed-direct.png) + +DUT front panel port is directly connected to one of PTF container ports. Usually eth0 port of PTF container connects Ethernet0 port of DUT, eth1 port of PTF container connects Ethernet4 port of DUT and so on. This is usually used in PTF topologies to connect DUT ports to PTF container ports. + +#### PTF with injected port +![](img/testbed-injected.png) + +DUT front panel port is directly connected to one of VMs interfaces. But also we have a tap into this connection. Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time. diff --git a/ansible/doc/README.testbed.Setup.md b/ansible/doc/README.testbed.Setup.md new file mode 100644 index 00000000000..a741a09b9e5 --- /dev/null +++ b/ansible/doc/README.testbed.Setup.md @@ -0,0 +1,93 @@ +# Testbed Setup + +This document describes the steps to setup the testbed and deploy a topology. + +## Prepare testbed server + +- Install Ubuntu 16.04 or 17.04 amd64 server. +- Setup management port configuration using sample ```/etc/network/interfaces```. + +``` +root@server-1:~# cat /etc/network/interfaces +# The management network interface +auto ma0 +iface ma0 inet manual + +# Server, VM and PTF management interface +auto br1 +iface br1 inet static + bridge_ports ma0 + bridge_stp off + bridge_maxwait 0 + bridge_fd 0 + address 10.250.0.245 + netmask 255.255.255.0 + network 10.250.0.0 + broadcast 10.250.0.255 + gateway 10.250.0.1 + dns-nameservers 10.250.0.1 10.250.0.2 + # dns-* options are implemented by the resolvconf package, if installed +``` + +- Installed python 2.7 (required by ansible). +- Add Docker’s official GPG key +``` + $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - +``` + +## Build and run ```sonic-mgmt``` docker + +ansible playbook in ```sonic-mgmt``` repo requires to setup ansible and various dependencies. +We have built a ```sonic-mgmt``` docker that installs all dependencies, and you can simply +build that docker and ansible playbook inside the docker. + +- Build ```sonic-mgmt``` docker +``` +git clone --recursive https://github.com/Azure/sonic-buildimage.git +make target/docker-sonic-mgmt.gz +``` + +- Run ```sonic-mgmt``` docker +``` +docker load -i target/docker-sonic-mgmt.gz +docker run -it docker-sonic-mgmt bash +cd ~/sonic-mgmt +``` + +From now on, all steps are running inside the ```sonic-mgmt``` docker. + +## Prepare testbed configurations + +Prepare various configuration files for your testbed. + +- Add/Update your testbed server management IP in veos file. Example:'STR-ACS-SERV-01 ansible_host=10.0.0.5' where 10.0.0.5 your server mgmt ip +- Add testbed server credentials in ```ansible/group_vars/vm_host/creds.yml``` +- Check that ansible could reach this device by command ```ansible -m ping -i veos vm_host_1``` +- Put files: ```Aboot-veos-serial-8.0.0.iso``` and ```vEOS-lab-4.15.9M.vmdk``` to /home/{your_username from step 3}/veos-vm/images on your testbed server +- Edit ```ansible/host_vars/STR-ACS-SERV-01.yml```. You need to change ```external_iface```,```mgmt_gw``` and ```mgmt_prefixlen```. These settings define network parameters for VM/ptf management interfaces. Example: + +``` +external_iface: p4p1   <--- trunk port of the server (connected to the fanout switch) +mgmt_gw: 10.250.0.1    <--- ip of gateway for VM mgmt interfaces +mgmt_prefixlen: 24     <--- prefixlen for management interfaces +``` + +- Add ip addresses for your VMs in veos inventory file ```ansible/veos``` inventory file. These IP addresses should be in the management subnet defined in above file. +- Update VM credentials in ```ansible/group_vars/eos/creds.yml```. Use root:123456 as credentials +- Add information about your docker registry here: ```vars/docker_registry.yml``` + +## Setup VMs in the server + + +``` +./testbed-cli.sh start-vms server_1 password.txt +``` + +Check that all VMs are up and running: ```ansible -m ping -i veos server_1``` + +## Deploy topology + +- Update testbed.csv with your data. At least update PTF mgmt interface settings +- To deploy PTF topology run: ```./testbed-cli.sh add-topo ptf1-m ~/.password``` +- To remove PTF topology run: ```./testbed-cli.sh remove-topo ptf1-m ~/.password``` +- To deploy T1 topology run: ```./testbed-cli.sh add-topo vms-t1 ~/.password``` diff --git a/ansible/doc/README.testbed.Topology.md b/ansible/doc/README.testbed.Topology.md new file mode 100644 index 00000000000..75ab767ff29 --- /dev/null +++ b/ansible/doc/README.testbed.Topology.md @@ -0,0 +1,65 @@ +# Topologies + +1. Configuration of a testbed topology is defined in one file: ```testbed.csv``` +2. One script to operate all testbeds: ```testbed-cli.sh``` +3. Flexible topologies which allow to use VM_SET and PTF container as one entity +4. All VM management ip information in one place: ```veos``` inventory file +5. PTF container is generalized and used in every topology +6. Automatic provisioning of fanout switch configuration (should be refactored) + +## Testbed topology configuration + +- One entry in ```testbed.csv``` consist of: + - physical topology: How ports of VMs and PTF connected to DUT + - configuration templates for VMs +- Topologies are defined in vars/topo_*.yml files +- Current topologies are: + - t1: 32 VMs + PTF container for injected ports + - t1-lag: 24 VMs + PTF container for injected ports. 8 VMs has two ports each in LAG + - ptf32: classic PTF container with 32 ports connected directly to DUT ports + - ptf64: as ptf32, but with 64 ports + - t0: 4 VMs + ptf. PTF container has 4 injected ports + 28 directly connected ports + +## Current topologies + +### t1 + +![](img/testbed-t1.png) + + - Requires 32 VMs + - All DUT ports are connected to VMs + - PTF container has injected ports only + +### t1-lag + +![](img/testbed-t1-lag.png) + + - Requires 24 VMs + - All DUT ports are connected to VMs + - PTF container has injected ports only + +### ptf32 + +![](img/testbed-ptf32.png) + + - Requires 0 VMs + - All DUT ports are directly connected to PTF container + - PTF container has no injected ports + +### ptf64 + +![](img/testbed-ptf64.png) + + - Requires 0 VMs + - All DUT ports are directly connected to PTF container + - PTF container has no injected ports + +### t0 + +![](img/testbed-t0.png) + + - Requires 4 VMs + - 4 DUT ports are connected to VMs + - PTF container has 4 injected ports and 28 directly connected ports + + diff --git a/ansible/img/testbed-direct.png b/ansible/doc/img/testbed-direct.png similarity index 100% rename from ansible/img/testbed-direct.png rename to ansible/doc/img/testbed-direct.png diff --git a/ansible/img/testbed-injected.png b/ansible/doc/img/testbed-injected.png similarity index 100% rename from ansible/img/testbed-injected.png rename to ansible/doc/img/testbed-injected.png diff --git a/ansible/img/testbed-ptf32.png b/ansible/doc/img/testbed-ptf32.png similarity index 100% rename from ansible/img/testbed-ptf32.png rename to ansible/doc/img/testbed-ptf32.png diff --git a/ansible/img/testbed-ptf64.png b/ansible/doc/img/testbed-ptf64.png similarity index 100% rename from ansible/img/testbed-ptf64.png rename to ansible/doc/img/testbed-ptf64.png diff --git a/ansible/doc/img/testbed-server.png b/ansible/doc/img/testbed-server.png new file mode 100644 index 0000000000000000000000000000000000000000..d03020609bdf88e0e08db452cf427d3290b965d9 GIT binary patch literal 21083 zcmd^nc|6qJ`?pqUQ4$pmHA*35Y%!LSB@D7>8HzFXov~)_7THONA+*@nWH*)y84Sw4 z4M}!}vhUBC(S3jW_FbOe@7aFO>v{g@zKhQ}pL4ErE${1nUFQ>UOI6{>L6(CwG&DyP zZ^~-W(CkKoKbQl%!B0G6dgQ_Xb~$S($k608pP2yv*lT`Wa9D(FYhiN%rSay z_#Sq1e>L)Q@R=FH>sHMPgIw6W6nUDmL=x`~5=q|`BD_VymIP=5ps&4Xh&u*!3=T{H(|P`hc4 zJtFL-IeQ?Sj^^4iC?m}uXJt8QXs-V!9<*umJIYRf^k`UU==2=g65qR47sftwg$4nI zvAQ@qzC3jR72WgHZFYd9pI6PAR1~LT%|3mJCPo$|a#L58^RSGKsBi#+hUpx5+{Fd; z;jAo~AR#5C^C&SszGr9(tOWe(ZW@ zdKwD9qL^=qhJP|Gg7d(fr;|R%>*Y43jLDC>c;QFM3B-6Tky-SJP%y#p(Huc5T~bO) zM39x1mX=-0`%MJAxw%=WzS(7}b6_BWxu8u)^QbPdeRK1k;3bm(r2OdWs>ibCZC+Rj zl5SA3x=(xNVNvppyZiTWIG0r*hwyHmo*C(A$=K^tLi5(;`xj+QKb8?~WUPy-c@#x# z(KMeFP<%3XP#EP@qe}%uYk|TJN1hW*X2p>~iR8m<7oOo1`fWJRt(DIxt*EkGym%4w z!w9eO_k*tUo^RAmQZU$L;f|DYUKF4huOM zH<6RPz}`3LZ{S|xO1-y78-08`s4IIxV3(gK$mx- z6M8$=@T^g+H=u7SU~bFAgpy}gx0Ef}gpD;-MlCaNrHv!cEhCdiLVsKbf{t0cL^7HD zyw4AP>hdS_Q|^+}S~#QZX&upxk1=9}doU~F#iqSEft3*a*P_;nCkF==Z>;9&*YVnq zkd!m??m2(xgS#yo&L<~oAG{)1Kk~|ye8){R64OSrp8*&2Wpem7s_anu`wfR(rVr{^ zpQA_51YIaLeVFNV-+aIMOwwT^h&bPqk4sH-HnaUC#;%Qs2u65@smXT5_j=JIcaDXZ z9>|djl}pLYcqnLra?9ZPG$8rzI=cD22A^Hs^qec%^k#mP9-NCjdFxQD0Mc#?tY|l7 zMX0h>6CaDY9LPFrL!Kqgz4E)v?>p+=KjpUD-!Wdz=2ke?n!Jt(L7yU?uO<#NFCb3Z z*%yzIlZs2~_65XYWo~+nkrPVfrX9i>KaeZ*oOO%5UJKWy>{|H#SZ}W`yL~Xxl2ohR zq+K}kNPFKb_8TYKs`$2>%3rf6a5S~&-J9?49w^0Mv}1rv)DS;yUh4{;7`W9`U`fL^ zaD^hAJMM%K!C>_#Dq`^wZS&SiIPoI;hn&4&KQ+82ett35 zN}^$Sa``FGxvMiNyWHKYFMkNR&>?w<(a0HJ&^|EP#k8ho6Wd`$3q;fA+ee}oZpkK+ z>H{vS%JB$j)-Bn7xZ~Az_3^pV5ts(Qkj3$ZFKeQwP`yGlN9|+e92On3lFBy52gq@? z+8BFA^taoWOCeK6f?}gLuAWqOc60LdDXunM7s(zR5DvY2#}4N%vW#xpXt*g@Pbjve zK~iO`pgXuuq;=hFthvS~Q$AR8h2toEaHVYoL&V!0Ke>lwoEL+S;Ivtrx4zhPJ|LKA z2Lx?&$Ii7X^j-P*_M>piHkvZ9FPWRDGN-JCA81YP@gKFw_l8ow9`4=o3QYZm3>gl6 z>bgXCfzPkR@FN2=F=_pHc$$7`P=Ot%?`+agXYMKqQ+PbzX+C#UV7Ts4vD*wOznY_f zTQII?i2VARYKqF9@vlenfo6Huwh!l^WjlIgeIaB+t} z4KcEg`He;0k#62h>nxuSi~M$Hkb~VgoiYY94AD?zqEG}i$}scsiz)p0+X&>q0Jj9u zhEx5-B)UkzxsSEdD`hnnw0H_0kN4a~WAS*^dABTDMg-=%X{SI#6FXP6y659)jcUfPu#a|0o zRa2V@4SmfaayE$rIx{=l7t+D6sLs7nlVBF|jRO|Z=Y*P6u(w4@@3#^S{4v>)IamEk ztd>vjDT##4H^nH_eP+<}b(1~6_(EjiYwe+Yl2X&;f@N(&cm5f6_Dm*OG+m%Sv8g{l z)~xgH=jSH0O0PD%LeqIM?q?w3+U^Rnu<%vm?L}iui7H)?M&8Ax%hng!DW@rBV;L12 zO5iT%fHL3`Ou$zBiB89G)z5U&C;YQMz@f*(C%Bes5_}1lY5yx9)=&;F#p5$ih9^jP zywYaCVNQMaC*IAc4;noIzjYj%e@u4tvi(50-XaQSjNL3my2kJ9!=ET~43}Tv5>=X? z|9Y5N(ekCX2E{IkVkGR%Zyp5YPgXq=@qvZsaV?k2hPRZAakHHqGe( z$wT3K^ib#SR&L=PaYFxBJ{-!zG2g{bCao=9ggZ>6+S`-Xhq@Ydqa*B)M+`rsN{ebj zV7ikkLf1d=2t)4L;;Bbfs5<=>tTjylwkf_m#_c9HGEh7?fY%x0yEr(1aQN2|HWkzn zi>lMDlo2pkZda*lltEM}yU(QROc>o$yoUV?y5mOR#G___jzQb zu9?rdtg0|6OY(agjMC>tsmpLo|0C4DT^?0=ZynPSQ9m~$4NinRgsAOmz?5UH-KPoSdj?Kb{{^&yi*&O z-!W@tg^Ln*W<&vJQ%W7uAG0X?ynEj{SfjcQhyiEnq;&?D@)VhCgynP2Z0)>op52q&+v|e`6ou1g* zT-Wbd9>m(~A}@mDkR5%KtZgL@<$*ds6SW`6@`T|4-tX1!3ZPnNFyZpvaIr(WI5y{~ z)s0o>9qIn1%?!Bw^Ou$Wi*)>dh4oy5!h(b|HBRzKz=8^&|EYeRXY#bzqT z4bL*cXQ~b+LUNuABw9Tf$i$SOJ4l_@956=a6Wexo#Qb*8DwFHUJ86^VNeheXmy}j?vacokJlxcgMP+LD5`GIG-b{~VmgocI9 zgsOUe|LTiOk(lgxBK_jJ!Z+q_icuX6FCBPrDu09^bkQYm_tBtuyw;=%{`K&<|B(X| z{x|_e^WuCjcILJ_s+9rvi0Y~Sw>#&*Q(P-9*edrPznxh4Gth$L zen5Ec%TJu&2^|bL-?K1t%L(p?Tr;=dgK46fr&D70$&)8Rrt9uDGF%sikXQYg>ijl z<#uKiP{3|=9(HLTq2cA_<(ae-zb`C&qTa!mm<|5@KR`uQh&u+Ey0*!+v{}H0xuo8z1UrvNS$U2o{6G z<5q$N_fy02sc^jmz#~4(fgRco?U|o-xrmY8qIg%jQzmS=T8)&xDRA9Mck;+iNY8t_Nj=d{iM%}}@Vwc93PIEU z+-U!0CfaatPVJ|0B=N3nry?puo`bqC zuDg}i&zgy(w-OAsi?xT!bVh8n@gj>qwpY#gu*uB>JCSe|`*X2aldv|Xq)x7f(ND?C zh~qvvK#N}Y5(=26GZOT4&6;(pGSkg;CW$If=0G?+(MpYmH35=JdTFkK5OUApV3!us zu6M`+h(Ubs>RGsySFS?IjdbS?(7hNq@NyKaG^jEMpu(ChocSj12GjPC08^{1^SX= zIRsOWXipL;aEJ?>A@> zF6K`tsP>mf(f<@uh~EDYG79uZRJ0Vb%6TkKYOhf8mFUa$WO7cA0M+~dOgnZWR?!(8 zzj0-IXVBu8_j^D}W;a-Ad-9S7!5E(mlBtF2Yl3|}?! z4|-MPQDC?!P{EnmQ#J$kKJ#%^8;wJ0C9;yHKW7yQ@e)}Sj}UCD1n%+;2{NhP4W5*$Z~$7r@*e9 z;Zlkc?lTMM3>FhL4TswJ1E4c8xo%+)NK zZQNXb=H9?P1#NQgS>BYwl+?MWsl-jEHf)EtJ}Q|Ea}xp{Z36giT>5 z!2`yIW;c497-!k6@ z2oAsqty8B`Q&TOspS+EDb7=&ghZJ9eu(S8!j)N@i@6Z4as@bJximi6SVvWCBjwzqN ztKQ&gk~uU=pZ)sv%f$23-(38O!rm)$*QGnpdmm{0D_a3%N79+_^wzc``;#AVhr5MD z&7I^1qkG>4WGmYuy_Hj}?Rrd5lc^Wl%8p@huyISb6WNa|24lEGT?p(?%#et^GVq)aox9BkB`ZsFTRvZvRCWrXOA^%YPaTQlDk-z5d8GDsh0MBEC zI=}vlOws5i2pMm14s3Q;&WYXlD;u*u`|ARdLx>$ihu5BHM88Cjo?k`}R^GGje5TZE zzui;5A1bI&shS%>=)bhXdC+laE?T%=wJ+hZzQol;%+SPWJ(Av>P($0MOZaSb1D>Np zE~PZt-bcD{Jve6l$eDS#@D{Cs@G4HA1*8IOdt^sH25Qq*B`J)r#AXFI(<;0R~r zsi5}4T~ygo*bR+OBi2s|h})5Yl6mIoe6S{ogCfiMkwdCSqdm}Xtmh0_3|=~8Flh6N zuS;$8xh3U>>&|Y!vL#BVa5-OHc)CtO1v8a6+K`4;w*;sBuDA`A7@=aBCXJ6wnrIQ- z0hB1tH*$=ab{I1v^m=gpdXm7$Q(rCw3n6c6n^zD{j3m3==(xupRV;1CAQ*EVV6^ zE3D(YYtO+_buBnu$4(aeuiw;8Er@4+`H`?N)t{KzKN9IR=;Jl3ACG_T-NF+r`pcAg zi_|G=?=cR96n@l7-r_hlrFc8C6P~Ah1Ere#2{GBsZ^y9wVIx#2iJN2IGk!{|R2#QG zluB;jiaA@#v$#K&imh(<_$bH%-6aQXOnBv!dQl3s0`WOAxs0&Z&GqF%?us{=IgH_A z6*7YL@XfNIlBQa$!!2_ZX3DU*0rzUj+T6-s_QiCRIF|9e7i!eLwpAPoG`qbv8J{k^ z(*L@ImO(fb!qh|>RT&wo_QQ0zLjT;ZJ%bykf)D1FB`PPQFT(+u=~R?~jPqRG-hf3{ zX_(tT_!dT-&c3ZrUEbH@>L5CmRb9#HB1t<7s9zg{#V5>h&r`#ps}(JEgBjWa?zy|C zC26T+q(#T&L>1b$B-GI;RS5;t(a-1JR&$iMx2p?J33bJirWPlda-Ok;61Sk_fmzQg zM!A2;`?7gnzQ5(7du2euB|I>Y_|0js;AYUTmg$dZd8}sR*7FVdUDgT?o09qaDa@-c z7L}g9B#X}cJ8qV+V6#$#sj?%4^yo1BJvrSo7ce8$XtVM?(U;>F-Zn`DIzQ7_oSA?a z#m`peXExVwVq}T-4M%^)_~7N98!LH8S5@+H=dUN3y8vSYh`$}@=gXR(5E9>gAG>I_ z!}I*Y6@&GR)?kWJ5gVj!sejCpMI1~LjW+{=7b{gbe~+JlrYu#nS)Q?8bNhPe8z7>7 zr<)@dYX%TT2Y(B@VA!habjS$sT>mg&wr1e=KO2BkoqqUmPqq%IJv6a@9na5Cx%OLv z(khR3pAdCUS$kFjG@&ja`D#}*@F{Mj zjvd>s#a2?TH69J*b?O zHp1Ry&q7<2s;^tQLhcpgJpr}y?&h2%m6$YSVlo|4NSUBp<%56qOuzMuePF4uPE z3r`?9vb_YCW%R{L(GF6S%Hrqv^WOZIgbTSht^KCHkAHsM;aK}v!fT1hwx%eH=Avd6 zk5@n0eLdI;9$z^L4M1>hXg*%@UhL3a`lJShDPN~BkV1fYE0j5n7quogq-=^iK{Y|@ zbkCzdj|)KWLO-A5;d%$5`Ilr<95I*_riL#SJ!jm^dSFZ5g>fEQ-Vjpm!T zEG(s|r_^CZ>Rqn?sKXXPSBIrp=Y~OAwO+N2$H4ER8ftzHKjBk>s$N*i^YhVwhg_l_Sp44>5ZYIn3DZKKQ@Z7>!KC$Z2*kDWt|7@OLltomMC^!G?OEw=EO4q@K=6}1WERT zMw}NA9JDR@b7h2=(npY*;RNU|Y=0;p+$6h!fCtKL#TyqQc2lwtm|RMXbwfpkq@YC(oODM& zxsc3uvw&!QY6g^32N_7Blikw2ujwEZWyR-(tg ze7VLm*w@$SsjEXtjPyc5zAX0gHw*U6w5XHGu@}l8P2zyj0Xk2Qqm2|NTyETk@TX1- zNcexj#wKNjb7k(aS*$vIDQX8bGymoqBjGR-d%Ws;sA#~m zz5j4^5vITiVF)BVzp#38RMLCp(@u@lDi~4Kyjj|SR4rYeDPGR$C~d4?F8|>Eni%ED zsPcpuuBh+`3Nsh@8#40%>_ny4ax*;&gqzn5gmR-Xacw*8QJ}e^sA6 zUk)n7zAmx=k){R;zBpk#can5fmI8H4;=9h`#V_?7l0TBB^fU-ugn`h zJTnmm|9mFPh;XjioGr3;?@?h;7yKtp6RnRq+U`ZOh2xlhgK@nMD4&(NyRHPrZGnh? z;Uqa=YO1OqE0LX>Z_%MGzcSnZSku*pfOgBo>gsAT`>Hq`?(E?K+BWMre~dP<=8tOm ztoIH6RvlIjCAbVe6TLJzYu4!+m44`Zp7nS4U%jxlLdy8s+jU=$k6T+(gIgSBUocfR z5nRFnf6o_bkn6C-!bn3y!*2D{AgKkIcJ{cT!bjDcfxl5tBca5mo3|rDxe8Dv632x{U3|;D&<9p}nv6zO9a;)P+Yq z51AbP39v?)cPg=eVmAZSz#&DcB=;ZR0(9h1y?13fV9KAbDICKYbjhQ(Ce*MRpyo<7 zC^oSC0%jZ?`*zP8MA){C2EbrH!58QZ*vW9LYJuP;HO^}?CrI71(v7DLFFi9Oi>|M_ zLu2!i(*HFzC}KzC!&9P8LCTtnYA zJy79g)hN9+EZsZA0#M*MrEh2v!wG{xoh$YUq{I{Uerg?-mQA51OM82-|D@kfK^ZM* zDH8f;b!H$v?{E(vpPWC0`i+GFS}T106n&9m9MnPJtQ(ZGG5RZsj3o@e{uFo`{TWb) zR0Py+=rd^kEBozwDz`~j@H&LxJwF4CXQJ7D2 z94oap^DkDbA)g`!A8C$lN$Za_`k2^J6Ws z1&`UW8=tdmU%47c>;|1$Joy2_C7l4$#Y7>3do~!J#B_4W5PRF7Ezb^G6SJ1=z8pw% z&9L*Nlm=rS=3-|YD1I_27{L_TS8Q)>3lfTyV+(bc3WQ3QVrY-DS==%_eUROBGG50l zr8ybqVmejizL&ThHa$BVm+oBJIYBaOSgVN%=rj}G9iHvAxmIYRW_`etG#d5}q#apZ z*2OtN;7!E=TZPqtcxhx{#jW9xwP}IH<1F@UOiG2S7h32~B9JdS(@lDZXqt*ubzQaF z8-(>mAOFdt=Y`|uNC|Oouj4b==Vl!K2-G*v(K0G`zw5WBfK>L#OjI-n=s0L2xLo%D zI^Q_CHIUT;o3bK-Hbl#KO?6aYGs1aR6hN09Wia8;M7{;hTkH`t4hCAaK%@p(4(Y%g2IJ+yNxU?ENpD_Q+;3E1Ga7TBb473lElH#oF5$XU%#46 z=M8`g5Ty*iO5w=!UJ$+o^mu-+u{jHE1I?6C{vhb+XTJ=E8|GE>uoO6_0_+6d=V4NG zY;t#@T_edRd4zzKrZ%d;?xNElpV{Bo=k|T0Kx>z|G=1!td$=H#8)RL1w6DLbGzV4 zDKP@!e=5Dkdac_)8W* zTvCk-kshXefXDj#-pXkbLotRHmAO>rC!AvYC0 z6CRLAah<9h6zMG!2FHwdEjv#FjT4JCw-hF_S#O{`JKej&9NAHilU_9HLh}Xs?rn^{ z{p!6Q;j!6PT--m^NEg4`<(>tS6>EZdo!z&>n4etCN3I!=nJQ^;X$VI6ChA;}9eoMP@k4ik_tJvNR^3m@5G_HlJ()-%PFCtNPX|9S=W*2{Ye}oMxHkGgNI9x(4^$N0b z!x@lgDUA^=4!$;fz9w)byS^>#;?(gTnW=#)z*}$thx#a7Z0b9Ebgt)J1v$_(k+(@Wv(Y1= za92ZHLph?{dOwoniPHO3^@mrv@V~zIPNEstzx(FVE255=UNwi9Z(PTPs7qk zv4d?oi7^$N3R6XEHPgbr1v^r6H|gXOfssz0Kn%vkFTS{S0BC1l2vJ4T)Lhb4Pj_lm zO0yR;b^ICW$gL^xtiZigncTD>m+h^4*^^Xy*U`n%W4P-ro?qamYXBDjC?*_&0{?^ou`?}1S^%#; zny##`%h4vD5aasP*vtpCQDHi`Tr{0OynlcFsX6u$)tNaDs}b!YJ!I@3wnW;sAJ>om zM(?2%L4lQQV(fW99Y$ErUP4IURY`Qm>5IaSszvPfGmLj zRK4|pjg8G}>M;aQNudt~BGiA%&E2FZ^A+I)FefVx*b(GCK&Z$`Nh<&8D~u!r`T!e} zzeqqlpzMcj8+ag3-j~4ZO&z1QgV8|+>yIQKxjh@8hYak|H4{8K(X=U%HPhMQgpTdJ zP~U2?l!fQ{2u7WI4zd9by73hwpdmAh()-0OG3}kgY3^`0)O^t=>dHJ z?*SfyuF=j8{7v9q@oz1=%qMWKJ~n$!!iFIEM`G3ymO#xU@pX2?a>|VlkBUrxGv4(x^TE}W6jT?f{N;2^QkQZdnwW3!k2&^Dd4D1 zPy!Y;UVgqEG+#3f4B;hAjmT^gLNx;_s6%|M!}@Hkn#ad~CBrIwHhY5|tI{Q{gYhR* z6KhOAj<FLkVz(X9#j8m(+FC+jy-9R1=6XCG+SXG*1YN=Lp zy`2{EZYI{|#9mJQlQ7OYh9{*|Z{_l6hy4Tb4Fm!0!on5M9dWw9r|0tvR#5&h7rZnT z;;}HU{y`w;(LPv|XO`F ze?(a58=d5zaOb@H2JN+Tf7at~ps2O{pD6k73-|!#Is4gfmr{Fh_x664NBw~0QyLkm z2rvicwyMmjOD$esg9GZI+Wg;E{{P*Z{%RFC?@tLX))#0@=Asd2AR&ZT-*);5^1@f= zCy(CTzHmAcU0xl~dl~l5owtI&t z{n2M(_jx&f0xr<+KzRaqWlGXUo#7}|ZLgzt8^{COr7(M-XYnOy1#ob8cL(X9Umg{! z?R2M#4sC2~n1;m{T>!CFvn~Mgafx#HlwQ?`1fr@(`VAoqEOr|-4GMPaiTYD=xj$b< zoOZbmem(Q;TTOEX#U)eTMX{LcDD)UqlgmsY+roP}=soC2oPHESZR7JCsJw?47me`v zg&*h%kxOv|nZO$%L7N}?5r6-N6b?;LFI`$%B1(w*gyuq$)#RmmU;>CkFB*7OBqFx+%_?iVTWorl#D^Q)@t{c>G2Q*+t% zK|DdGM!9G@T!A5tJlV4_%h&k}B1Q`r1P8KKLF-Ci#Kb87-%f=%y&I(euPps^PdIUA zF4&7gTx&;RSzdi@w#wAM@jfrozrtp*N8Mpo?oE99bnxm34Ri_U3`+CzP% zd6#8&B*n4roLc^&V1ye_A*(XyWD@phXSU9Kw>oIF8C_Fz?ASigCL0YImm9rp6xc(X z-A3;gqc{+iXOMp^f@*L-Pw3<+9G|7(8?Gf{%vO_H^}Y zfUH%Fh6e*c@s~0+k2JWTJfnlZzBRwRyF%Da^AqvP`=)6rm+8SqCegUv+Fs+IZwhuy z|1rdzS6xE)i=F}T)rLXn2?==wA6zMTv!|%s1lzmsKH=58 zSC_`q*DFp4x`RXAKPDx>(@QIO)s214pI8d~AD0ry?K~MK>LV`dGl`;SuDoKv^{6PM zq?GZLZ`li!U78uRcFmme**`H_Tn*9pSg!B$c~g3*f!uW0v1-cF&t3-AEQj)h25Fp3 zG~4NE>T%6i$%V)Td3{sZ&p&LqVWG-vQkV0?{)!6iSQk@LDO-n@p#1)aO~@k5n2Y-4 z%tMQ&UR(PX+OA-_U$jY+IDSlJ+@};dSTiQx(75%0bZQA=6bZ@xcHpiO-wHp6klJO91Namli;gA!wX+RW$~?-^I91U?0ZEg z(uDTQEjg-Fk^7#tpSCHT^a+YdiOLf;J^Z(0v$}Ma$?GvhxW1W1#Q6nTDtMm==15rGa`FPvR3!@*8z`bj-egNL!@M;Ir{v0bjO8s_^Qf-VZahLHVrykUJT> zWPXmu+GF*{NUB=mbR72s9SWI%Z*55L@6(Z60S#&zZ8>>Pvx8N|*$3rOoF`dvG=ISU zKfR#$f8V7v{Z#I7;k2Q!`{-jhaQ{A27PY(SER076r33EY$6iP2JR)Qr52q7)NXP=Mr56CD2KwT` zjR7S9nLrf)N|}(z5+5I*-UTA-^p!VYBz1`1>+0$oi@jT0m5}{9J8#M0qf}PZ)YJyE z$4weTkxxMd!xcAvwOjoiXi1<<$|N@mI)h!2$rX#>>S3SQv+IhMEIHE!um51$d6!iH zVtyk2QJIR8AD!A3E1EzFg zM>Ur7W%e(qm z@Li-=&nA7&d%L)NW8-Q)bflLqGfb6Ik4j>_F!)GkBs4fpEVd_4@Ir4+=!{i1MsN9j zX(W5%&VhoJW;pgLs*?_1?kBB$4kEuUN=d?`pRg*odZl<)VVGp@x0!rcQ2RnReQIAp z{OcP*)=xV@pDw%Yk<~cv3GVd$g^{D`BOO}H-@rAy#wfH~ zPc){DO)$(2Bs0rNc)3mMB?bIUE1}}kb0rQ(_+ZEY9i8aQ$kvfC=FzvH5+)-mnH=|^n3<}(g{xTF|BYNfQZY-VYU2uGB?H;p+v zYj;A`ns2cbbuC;&(B*_Yt%q`1|M?*2E%9bP^DN^);%} zYRG{OClB=-u5XM#8%jbhW4NQi4gRDu4fu*)y1`4ntJ7&b11sO+$gqISMAqrKs5z2g zKXb(`h9*l+kC#w8E~HJ5`mlzr41c*m=;>U!={cp|6kPZK=Ru$nz$i43;ao}wbhi%k z%NfFsFfYI7o(@a_7Zy&IC|SVETL*82s<85%Y-*BM`$9g=Xz)BHN%^**@V}kR6ufL?#&=yf~RY^flFM zzO8?0dcdM7&$60yPrG_1E%Y5pkhAEjfF-nIO~dxX=8G4}#8cHx>5u!=c`W15@}6;g zXLd|6NJ!IoN!eaRr4+Z8nC`DN zNqL5+BOMqyS2`NWKlG)UV^o)OGA`J*5pVKZ{3YOyaITyaWv!zhADwVBeU%`dg28+A z`y!X$^0%cdw*7&t!>3Ii?2$Az31W=+=wx>5gNnQ85M;z`dru=}--RzAIlFe2SE!JZ z>6TMa>J?+O@A(yVHONUB&SHct#BlqRa9fIs%>sW$nGEJ{E3;xm2eahJr^}6g)d*=# zp8s-W@*igmI_Ngsto@EP@wzMLq-8W$PX@`;cGQhk7-rjRqx;^{PoclWOTHIu)QV9& zm__37*6#bi&7v)vqBM3p8Pn44V&8^S+d6R$v%<-^SkOp1@1h&AX&wJ94%6a{VV^mD z;;q+H5#9Yj8}a3pc{B19RT;XE+`Xge8U6_8&7`kpWN{Ym*L?g<$x*|w8-nhxx9T%! znBnCxbv3-9?Tk^2$5VTKgU{_-4|DIG!(X&?HoKLfGVYHnau?FL_)y0T?c6HTv~G!z z#va}}pmE}_xfxkawR)$$A>^mZ2wx_6Ki(xKEtL&?y1-=~&oh(Kq{VMh_h{OPvQ-C0 zyRGZur*2b&XSOnX&qdGvc6@ z??`uhE%ax|yZo$_xAfahhcJ~1U%tYoBN|ATgy+v=%G;|NH#{g+H)`lZ=ee9qetz5i zybHU?l7+?>O?8l`?u5O74_qe)E45Y%n% zTZC6j0OD+vuRH~)%w#46C??E_;nBQ>1DrCTV4aosBS7v$*E8=n`HU|(ynU=O@7q1Q z)~3`7#{gY>Hfo83(uTG86>uQA+S#D~th;Zr_4DU&9*pP9>+qpY-X@HXj~%D0<&juh zd;6AVx2b1FaZ$^zqS=M6qW2seuIR*?+sX)~!}B|{j)-AWc&t)Ji}Q0VwkXw~dWdCM zTvqV}Ygk`TPmdWqcNDa}i#~KRJ>bDQDIz=dSxe9$(NcN&O%U^JaRr`ExRdwQcL6V{ zLtw|PKB$bfG^4b`>36b~+Jw39#wye{@?c*H81186oow?YdP=m#J=zt&b1|70(OQ@jA8HZKBKF z>r|#r2r5nqJ$xlMeTJiq^?3fdklDwtI!)0c?aaldBo{5l{U;~vWi*BJ^_HCYyL#vK zDCd(bo04YZ${}lbj&b(uJLLYJA*K%4I=DrY|KlCKF|NBn`H+45P~0tTVe7diZ=3Q) zU3mXpP7krjxsnV03*C;4f<}#?@X5&9y4IAm6{dnN7p*Fh6AFWL^gME}YY@;hpIN;O zjHuxOG~amGaC4>$R@{F3v1{pK_Yey9Cn{mfNad$ftz()plr(eDV zn{lYCNdm<=!F2YU_e`g7x2s|l@B6;3%Fqj+cxw4Z>|thX_iBLR=I}F2n+d0@fq8I9 z;+2YX%cV|LyqiZ~9x0X)tzO_fELfc-nF#D#1Lt|M`0gvT3WDYGNT^ckvySA)Iozl@ zyCg)kvY=(wt$~t&yiC#JG5v>Nzl=yR922IrF`b;z55<(2nrLj7B$EnaKqq}j-gm|n z(QheBeK2<&f9uM=kO`Gj=>5ev!6#pyG_=D@k1z^bn`l%murjI@1jbJ^xReS-tog#c z8Aqd`OF0Qm49NbNqbwCi=}8HQDG`wlM`gUFKm-tTB}~NU!@}eKb{_-cokYsatS1=+ z9~ztopPadAHW}alI3QM7Al-OPPFD_{^j4|MAr`eDZ6TD>n2h_z)SwsVz#+Nmr`Mlg zp^a%*(SQ^e#4!sh&V$1-9lnULPpcZUBYrVY{2DupG4*78jC;@fl;Y)Ji>h|jvQ4ou z?fjc$%`zWLR)dzBwOYhvj%_C=k&PP?mDQl@?C9a(@FGZSc{<>AaN=@HBb{yOE$QKGh@n&sf>1dHK(S} z8epyHncy~QJ4;bs^wZJ2YF#j;1MB4WlXIie-j?-nZFvKB**4OOm}CZWxZ-A}{i!ZtU}0g;o!**eHN+&lsA35aa#>x|mEk?u z$5;9vySUG!@sIv}@<1$5`viU$>PW=L`=eI_xd(LeS4m&3%CtnGVfcU@xEn>DmM zPnRO{B#-GiqKZX1*C1Pn*7up5PKV{gg++D=0XoTzoU;vF9gY!?C?XoX<`!9?dW5@m zEq&CH^*BVd*q+SAEnw~34>dAqcSAF5_L0ag2EoF=iOllN0=^Y(Qhy}&kW}7062})! zl@3q7$WW*BP3yUa3WI={#UKb7IdWdY?x_{90*^Ws!z=IstM&fJW%#b-vjzE>;HJw~ zv2LHGNzA4pe=~&l?Uy@X@ctpCCFT&k5^G2Gn_kAPy0tT#TS4SUqeN-&}UkSV3;xB?wIeeNw5KFJqXV_iYK8d zoy+zWfxA2UP0u_(=ttOl1$>f*XL%*q2y|Pq1;p3fZ=2VY`Sjs7{sAv@)#sBCQ(W$c1j+-rp)$O5fYfp#|84{|BK%38)})!4f6C o(0;qJf+$~Be7C(hxJ@f0Y~#?)pN0em7amMxuOUkqLo=kvH6%NeZN@fCl(EZ5BHP$2F1gA&i+xL>WJ#N)t~D`C z2qD?U-@Y6B`nvahzq}vLem|lG3I85+#KgQKp+q|2Cat!ftV9dB#?>aB(uOo z=|3?{M4aJuP}z6Uxsw3whB8HgK<`PM2hOKXVpek#OMP>5^W)=V8_X@;lMK5L8czg) z#Gd>kOr)tbR}hHH0i%bq3?5t>=}4Tc6a)FFapcLp6Y7u@6z3vFiiQS|<2Z!Af=`o; zBWfCX_Lx7;HnV+QzZ2UBjof^Z2IXka?@f#>egfKOX1eqyw*9^T%saa%3FBC5%X<5- zWD_@ri*0=h`e@g6kagHUFSWb?A?J=!gj0^4TTjO@*^zts=a!<|O{f|Z`zVK1 zMtsfG+gd{xW;?HV(*UYDr*wz*8y2z4(E?iShq0dhPMt}4(y8Kv<` z`s`R{yZ`}C$8wX4WY@AyPSGo++|76%nLUNMpwtE`3ELkpZVrWocj+?jrZ%^mP7hJI zHnR0-J7-{vaI<#%;lV`U`ONKZNJKZ=>1t~-bRZFUNP}@B!)VfW!%4LwL%;6+>W)T; zpJ%T$-ZyT1<43aoLH*yzjBQR_J95v%;gN}$^MA7GjYt-IO<86q=M(*})&=#GCvVHv zUvES}#xzxBkOp#&UE?{S>;+8nxz7Zj&@1-+9;s9oAXYew?(MU{?;}FBTR)wf0v?8D zCc)QOFvw$&GF)EE0;mp`p&l7h4tNvj*9_fMF%D3f^%7lutzDn(g20V~l_JTxF@E}y zH97pg$wXnI&vv>W~K3F*p+e&Q*Bl*f0pUkj_V~o+m7#TZ&Q=&=E-y#${Da& zWs$IQK=!L(Y-t$g_#`0_=^6MhPHI+JngIhx0is0S>hWBwjW0Z_8IZQ7lvlWngofP9 zZc_Q9D!#DQpJ?^AIzFwtVIy;hr?r<5Uowk%=L^a5jEobyHfG^N8d9JjmdGhXRzIt4 zXw~XC7haRc6$$G^1xAW(n8V>Q)juIgq(yI&o}d`3Rxo|hx&T~?eRJuv?~Y@xi7B-i z6wUgoKL%-h6i68GQ#>`gfZR{GvY5EUjWzAfKASpNboz(gd{NnRrCc*rw5x9CYtAY&@V#AlCs|w zw37Q(Jh22bFH4Ro{+^JOK4vXXTz^N;6k+_h3RFP)ZHUBY{Sp-dt*Mz#({f4nD2I}H z8=$w^UCx?JIyG<+ri!|yN$`jPbjS&r2^uMYe}-s1Qpe$@luR?1TO0l3ZV(O%!|P!~ z>**7X7Z`-rZ6~?(3~7cphr)q@AQH<;!{~Znd^cX)V+x#pRP({oBAiF2-oky&jKh4g zajN%E!s|g&&pBilp5|f&`MyclD6#r>)2mGoAlL{ukk3K-q<4AH@`_l4idGrxT)^B3 zhbUq~#`jZEe8^I=N6Fs?)DU58asGH;csY}>Y4K#+d`d_Hnv^vzN=D548T`gJe!MKW@C7^6Ga}J;hQp7sSh^+c#7${?)OiQz zv0&#{k%5%7ElVW2)&DgQsv6_Mk%l1j%k3aBCbxvW;81~ zNZ^o=Uu-7WnQWNsc-um=%>bBq5j@B5L8)xoG^&AJd&GP*$2H?;T5!U$NmWz8-ZO*d zBrdW$(zVM!#!!t=D?|eBisW0sNKr0i7tjpUuIPS?e0H<@dvrric~ z^#kXVBsDubaRor9d5X$hQ*l$q(75arK7|m--lrQyD{d0LmztKO@8FQytlKv6IU`<0 z$t*M8_nf9;tVv6--#_YTA^wg_HmEbWa6xT)e@&5b7<{Kj{MIe!58`hz^T|7*>&u2- z3A8jdg);ro-`0nkm(~QA_|~=szbE!;wmL!|?aw9&F1_8&GMpIgu6mOR-&NN3l0^sb zP8IAhjD8fEPR|Hj!VOO%VvK$~(7wgJoLz2=O&$I_c=@6{`>%9S2En6>@CLfRr8K4v z>BINF2d<#+k_m~v)M7Qa)F>Lgt}3QfM3evXG+i;#8nk;V`~fF^ztJMfei6mnC7(dS zPr$vZIHTo=fTuF2`k~pFXoi8ie>XI7uS-;Tf-Fw8V9P+*Msk@XHNY}VSDXG2E~??0 zy-zrtrPH?>;KI?wo&u4VYx;%Z>iGKF4Z?=M`beEB!~dL3r=$m(@?U5*7y5>e6#)@`ZAW z37GxkMH7V1DU4wod^VU0J~hxUv8LeTe4b<*D_TmmcZ*r9d~zPHdIqS0?`z@^EJt=! zrkn6Ci2+0M*AO8;{YVeO-+F}(xryia?Kz7 zSq8=zw20#1E4&DG6#aA>&8+W#djKL9{(Sx=Jl9*Ywg`4MFg8>M{Rt+V=}Tf2&Mgu< zP$K8Ad9uhTeYTG%rl`~Rf9_O9=jMir2sbvLqp>jZs;#6H;58NwW^sas;weaj;B+<9iveTtUD&R_}A5->i?H)QG zLgC#KIRWtd=(^O)h_rRiTo!><=fC_#$Q#&F{b8B&arus+G21LYC4egDmEBW@*>tRk z2Su9S$&Kqv!*7?Md9%y#n4-tEzFm?~?Wh|&kl-wg-}O)o0QEOh)Y4K{&w}1;eOe9? z6lGcT8&9ZxFQ6*@bpujI)#eBi_^Z1Vr=ij){ZWujOgkZ@EnsAg!V~lUATv&o%B3iu za0$@(x{{Y`!S*tb`H4gDtE9B_i$C~!YBBRKeN#+sKye$^)KP6`{&_Od?-F`k4Lqvz zim2bH*EGHTN9NreL@><-_032L^P3~X857AOc;OL}sc`pJG(3X%6)y#wAZkl*BUsW#clqgT z2H9H)J(?od9T9(mgJ#c+jo0!nH7t;GlYA5In`FLdA6VRM%0YsW?;T_KLR;$V9wSR* zr56shW>vwy6;Gm+(Cd=(;C1Px=S=J0%IdyRJupYMNyGnlpq}h{C5o^>XI$;{-3zb8 X-geoy{q^ Date: Tue, 25 Jul 2017 17:32:52 +0000 Subject: [PATCH 2/2] add explaination for back plane port --- ansible/doc/README.testbed.Internal.md | 4 ++-- ansible/doc/README.testbed.Overview.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/ansible/doc/README.testbed.Internal.md b/ansible/doc/README.testbed.Internal.md index fb0c057c46a..78521f80879 100644 --- a/ansible/doc/README.testbed.Internal.md +++ b/ansible/doc/README.testbed.Internal.md @@ -54,12 +54,12 @@ topology: - Uses VM with physical name VM0100 - Ethernet1 is connected to vlan 102 - Ethernet2 is connected to vlan 103 - - Ethernet9 is connected to Ethernet9 of ARISTA02T1 + - Ethernet9 is connected to backplane network (implicitly configured) - Second VM: - hostname ARISTA02T1 - Uses VM with physical name VM0101 (vm_offset: 1 + vm_base: VM0100) - Ethernet1 is connected to vlan 104 - - Ethernet9 is connected to Ethernet9 of ARISTA01T1 + - Ethernet9 is connected to backplane network (implicitly configured) - PTF container: - 5 ethernet interfaces: - eth0 is directly connected to DUT. vlan 100 diff --git a/ansible/doc/README.testbed.Overview.md b/ansible/doc/README.testbed.Overview.md index 7ff5ce585ad..4fcf3aa1381 100644 --- a/ansible/doc/README.testbed.Overview.md +++ b/ansible/doc/README.testbed.Overview.md @@ -37,9 +37,9 @@ This document gives overview of the SONiC testbed. The VMs use Arista vEOS. They are using to setup protocol test such as BGP, LACP, LLDP. They are created using ```testbed-cli.sh start-vms``` command. Each VM uses 2G of RAM and has 10 network interfaces: -1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port). -2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above). -3. 1 management port. This port is used to connect to the VMs +- 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port). +- 1 back panel port. All VMs in one topology have their 9th port connected to this backplane network. +- 1 management port. This port is used to connect to the VMs ### PTF