diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index e1c613d1538..5928a8131d5 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -1,53 +1,110 @@ -# Testbed topology +# Requirements for the Linux Host +1. Ubuntu 16.04 x64 +2. Installed docker-engine +3. Three network cards: + 1. first is used for the server management + 2. second is used to connect management interfaces of VMs and docker containers to network. + 3. third is used to connect VMs and ptf containers to DUTs + +Content of /etc/network/interfaces: +``` +root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces +# The primary network interface +auto em1 +iface em1 inet static + address 10.250.0.245 + netmask 255.255.255.0 + network 10.250.0.0 + broadcast 10.250.0.255 + mtu 9216 + gateway 10.250.0.1 + dns-nameservers 10.250.0.1 10.250.0.2 + # dns-* options are implemented by the resolvconf package, if installed + dns-search SOMECOMPANY + +auto br1 +iface br1 inet manual + bridge_ports em2 + bridge_stp on + bridge_maxwait 0 + bridge_fd 0 + +auto p4p1 +iface p4p1 inet manual +up ip link set p4p1 up +``` + + +# PTF Testbed topology ``` Linux Host Fanout Switch DUT - +---------------------------------------------------------------+ +--------------+ +---------------+ - | PTF Docker Openvswitch | | | | | - | +----------------------+ +-------------+ | | | | | - | | eth0 +------+ +--vlan101--+ | | Et1 +-----+ Ethernet0 | - | | eth1 +------+ +--vlan102--| | | Et2 +-----+ Ethernet4 | - | | eth2 +------+ +--vlan103--| | | Et3 +-----+ Ethernet8 | - | | eth3 +------+ +--vlan104--| | | Et4 +-----+ Ethernet12 | - | | eth4 +------+ +--vlan105--| | | Et5 +-----+ Ethernet16 | - | | eth5 +------+ +--vlan106--| | | Et6 +-----+ Ethernet20 | - | | eth6 +------+ +--vlan107--| | | Et7 +-----+ Ethernet24 | - | | eth7 +------+ +--vlan108--| | | Et8 +-----+ Ethernet28 | - | | eth8 +------+ +--vlan109--| | | Et9 +-----+ Etherent32 | - | | eth9 +------+ +--vlan110--| | | Et10 +-----+ Ethernet36 | - | | eth10 +------+ +--vlan111--| | | Et11 +-----+ Ethernet40 | - | | eth11 +------+ +--vlan112--| | | Et12 +-----+ Ethernet44 | - | | eth12 +------+ +--vlan113--| | | Et13 +-----+ Ethernet48 | - | | eth13 +------+ +--vlan114--| | | Et14 +-----+ Ethernet52 | - | | eth14 +------+ +--vlan115--| | | Et15 +-----+ Ethernet56 | - | | eth15 +------+ +--vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 | - | | eth16 +------+ +--vlan117--| | | Et17 +-----+ Ethernet64 | - | | eth17 +------+ +--vlan118--| | | Et18 +-----+ Ethernet68 | - | | eth18 +------+ +--vlan119--| | | Et19 +-----+ Ethernet72 | - | | eth19 +------+ +--vlan120--| | | Et20 +-----+ Ethernet76 | - | | eth20 +------+ +--vlan121--| | | Et21 +-----+ Ethernet80 | - | | eth21 +------+ +--vlan122--| | | Et22 +-----+ Ethernet84 | - | | eth22 +------+ +--vlan123--| | | Et23 +-----+ Ethernet88 | - | | eth23 +------+ +--vlan124--| | | Et24 +-----+ Ethernet92 | - | | eth24 +------+ +--vlan125--| | | Et25 +-----+ Ethernet96 | - | | eth25 +------+ +--vlan126--| | | Et26 +-----+ Ethernet100 | - | | eth26 +------+ +--vlan127--| | | Et27 +-----+ Ethernet104 | - | | eth27 +------+ +--vlan128--| | | Et28 +-----+ Ethernet108 | - | | eth28 +------+ +--vlan129--| | | Et29 +-----+ Ethernet112 | - | | eth29 +------+ +--vlan130--| | | Et30 +-----+ Ethernet116 | - | | eth30 +------+ +--vlan131--| | | Et31 +-----+ Ethernet120 | - | | eth31 +------+ +--vlan132--+ | | Et32 +-----+ Ethernet124 | - | +----------------------+ +-------------+ | | | | | - | | | | | | - +---------------------------------------------------------------+ +--------------+ +---------------+ + +----------------------------------------------+ +--------------+ +---------------+ + | PTF Docker | | | | | + | +----------------------+ | | | | | + | | eth0 +------vlan101--+ | | Et1 +-----+ Ethernet0 | + | | eth1 +------vlan102--| | | Et2 +-----+ Ethernet4 | + | | eth2 +------vlan103--| | | Et3 +-----+ Ethernet8 | + | | eth3 +------vlan104--| | | Et4 +-----+ Ethernet12 | + | | eth4 +------vlan105--| | | Et5 +-----+ Ethernet16 | + | | eth5 +------vlan106--| | | Et6 +-----+ Ethernet20 | + | | eth6 +------vlan107--| | | Et7 +-----+ Ethernet24 | + | | eth7 +------vlan108--| | | Et8 +-----+ Ethernet28 | + | | eth8 +------vlan109--| | | Et9 +-----+ Etherent32 | + | | eth9 +------vlan110--| | | Et10 +-----+ Ethernet36 | + | | eth10 +------vlan111--| | | Et11 +-----+ Ethernet40 | + | | eth11 +------vlan112--| | | Et12 +-----+ Ethernet44 | + | | eth12 +------vlan113--| | | Et13 +-----+ Ethernet48 | + | | eth13 +------vlan114--| | | Et14 +-----+ Ethernet52 | + | | eth14 +------vlan115--| | | Et15 +-----+ Ethernet56 | + | | eth15 +------vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 | + | | eth16 +------vlan117--| | | Et17 +-----+ Ethernet64 | + | | eth17 +------vlan118--| | | Et18 +-----+ Ethernet68 | + | | eth18 +------vlan119--| | | Et19 +-----+ Ethernet72 | + | | eth19 +------vlan120--| | | Et20 +-----+ Ethernet76 | + | | eth20 +------vlan121--| | | Et21 +-----+ Ethernet80 | + | | eth21 +------vlan122--| | | Et22 +-----+ Ethernet84 | + | | eth22 +------vlan123--| | | Et23 +-----+ Ethernet88 | + | | eth23 +------vlan124--| | | Et24 +-----+ Ethernet92 | + | | eth24 +------vlan125--| | | Et25 +-----+ Ethernet96 | + | | eth25 +------vlan126--| | | Et26 +-----+ Ethernet100 | + | | eth26 +------vlan127--| | | Et27 +-----+ Ethernet104 | + | | eth27 +------vlan128--| | | Et28 +-----+ Ethernet108 | + | | eth28 +------vlan129--| | | Et29 +-----+ Ethernet112 | + | | eth29 +------vlan130--| | | Et30 +-----+ Ethernet116 | + | | eth30 +------vlan131--| | | Et31 +-----+ Ethernet120 | + | | eth31 +------vlan132--+ | | Et32 +-----+ Ethernet124 | + | +----------------------+ | | | | | + | | | | | | + +----------------------------------------------+ +--------------+ +---------------+ ``` Figure 1: PTF container testbed - *PTF docker*: A docker container that has 32 ports with pre-installed PTF tools. See https://github.com/Azure/sonic-buildimage/tree/master/dockers/docker-ptf -- *Vlan ports*: 32 vlan ports are created on top of physical port, e.g., eth0, inside the Linux host -- *Openvswitch*: Connect 32 vlan ports to the 32 docker ports inside the Linux host. Openvswitch forwards the packet to the corresponding output port based on the incoming port. No L2/L3 forwarding is enabled in the Openvswitch. +- *Vlan ports*: 32 vlan ports are created on top of a physical port, e.g., eth0, inside the Linux host. After creation the vlan ports are injected directly to a ptf docker host. - *Fanout switch*: A physical switch which enables VLAN trunking. Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. Et1-Et32 are vlan access ports and are connect to DUT. +### Deploy testbed with one ptf container +1. clone sonic-mgmt repo to local directory +2. Edit 'ansible/group_vars/vm_host'. Put your credentials to reach the server +3. Check, that you can reach the server by running command 'ansible -i veos -m ping vm_host_1' from ansible directory. The output should contain 'pong' +4. Edit 'ansible/group_vars/vm_host/main.yml'. + * 'http_proxy': your http_proxy + * 'http_proxy': your https_proxy +5. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02 + * 'mgmt_gw': ip address of gateway for management interfaces of ptf_container + * 'mgmt_bridge': the bridge which is used to connect the management network + * 'externel_iface': the interface which is connected to the fanout switch + * 'ptf_X_enabled': true, if you want to run X ptf container + * 'ptf_X_mgmt_ip': which ip is used inside of the container for the management network + * 'ptf_X_vlan_base': vlan number which is used for connection to first port of DUT +7. Edit 'ansible/vars/docker_registry.yml'. You need put your docker registry server here +8. Start ptf container with command 'ansible-playbook -i veos start_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See start_ptf_containers.yml for more examples +9. Stop ptf container with command 'ansible-playbook -i veos stop_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See stop_ptf_containers.yml for more examples + + +# VM set testbed topology + ``` Linux Host Fanout Switch DUT +-------------------------------------------------------------------+ +--------------+ +---------------+ @@ -186,44 +243,18 @@ Figure 1: PTF container testbed ``` Figure 2: VM set testbed with injected PTF docker -In this testbed, we have 32 VMs and 1 PTF docker connected to 32 openvswitch. Each openvswitch connected to a vlan interface. -Packets coming from the vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are -sent to the vlan interface. - -## Requirenments for the Linux Host -1. Ubuntu 16.04 x64 -2. Installed docker-engine -3. Three network cards: - 1. first is used for the server management - 2. second is used to connect management interfaces of VMs and docker containers to network. - 3. third is used to connect VMs and ptf containers to DUTs - -Content of /etc/network/interfaces: -``` -root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces -# The primary network interface -auto em1 -iface em1 inet static - address 10.250.0.245 - netmask 255.255.255.0 - network 10.250.0.0 - broadcast 10.250.0.255 - gateway 10.250.0.1 - dns-nameservers 10.250.0.1 10.250.0.2 - # dns-* options are implemented by the resolvconf package, if installed - dns-search SOMECOMPANY +In this testbed, we have 32 VMs and 1 PTF docker. The VMs use Arista vEOS. Each VM has 10 network interfaces: + 1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port). + 2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above). + 3. 1 management port. This port is used to connect to the VMs -auto br1 -iface br1 inet manual - bridge_ports em2 - bridge_stp on - bridge_maxwait 0 - bridge_fd 0 +The ptf docker container connects to the bridges which connect the VMs frontpanel ports and physical vlans. Each bridge has three ports: + 1. Frontpanel port from a VM + 2. Physical vlan port + 3. PTF container port -auto p4p1 -iface p4p1 inet manual -up ip link set p4p1 up -``` +Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are +sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time. ### Deploy testbed with one VM set 1. clone sonic-mgmt repo to local directory @@ -240,8 +271,10 @@ up ip link set p4p1 up 6. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02 * 'mgmt_gw': ip address of gateway for management interfaces of VM. See 3.2 * 'vm_X_enabled': true, if you want to run X vm set - * 'vm_X_external_iface': name of interface which connected to DUT. See 3.3 * 'vm_X_vlan_base': vlan number which is used for connection to first port of DUT. -7. Edit 'ansible/vars/configurations/*.yml' files. You need to adjust 'minigraph_mgmt_interface' to settings of your network See 3.2 + * 'vlans': list of vlan offsets for the VM FP ports. For example: if vlans equal to "5,6" it means that the VM frontpanel port 0 will be connected to vlan {{ vm_X_vlan_base + 5 - 1 }} and VM frontpanel port 1 will be connected to vlan {{ vm_X_vlan_base + 6 - 1 }} +7. Edit 'ansible/minigraph/*.xml' files. You need to adjust following xml nodes to settings of your network: + * DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/Prefix/IPPrefix + * DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/PrefixStr 8. Start testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos start_vm_sets.yml --limit server_1 -e vm_set_1=true' 9. Stop testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos stop_vm_sets.yml --limit server_1 -e vm_set_1=true'