From 576d7585510c94602c5af350a69e0f2f34ef4cc6 Mon Sep 17 00:00:00 2001 From: Pavel Shirshov Date: Mon, 14 Nov 2016 12:10:32 -0800 Subject: [PATCH 1/6] Remove ovs from the ptf_host --- ansible/README.testbed.md | 79 +++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 40 deletions(-) diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index e1c613d1538..b7a469acf94 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -2,50 +2,49 @@ ``` Linux Host Fanout Switch DUT - +---------------------------------------------------------------+ +--------------+ +---------------+ - | PTF Docker Openvswitch | | | | | - | +----------------------+ +-------------+ | | | | | - | | eth0 +------+ +--vlan101--+ | | Et1 +-----+ Ethernet0 | - | | eth1 +------+ +--vlan102--| | | Et2 +-----+ Ethernet4 | - | | eth2 +------+ +--vlan103--| | | Et3 +-----+ Ethernet8 | - | | eth3 +------+ +--vlan104--| | | Et4 +-----+ Ethernet12 | - | | eth4 +------+ +--vlan105--| | | Et5 +-----+ Ethernet16 | - | | eth5 +------+ +--vlan106--| | | Et6 +-----+ Ethernet20 | - | | eth6 +------+ +--vlan107--| | | Et7 +-----+ Ethernet24 | - | | eth7 +------+ +--vlan108--| | | Et8 +-----+ Ethernet28 | - | | eth8 +------+ +--vlan109--| | | Et9 +-----+ Etherent32 | - | | eth9 +------+ +--vlan110--| | | Et10 +-----+ Ethernet36 | - | | eth10 +------+ +--vlan111--| | | Et11 +-----+ Ethernet40 | - | | eth11 +------+ +--vlan112--| | | Et12 +-----+ Ethernet44 | - | | eth12 +------+ +--vlan113--| | | Et13 +-----+ Ethernet48 | - | | eth13 +------+ +--vlan114--| | | Et14 +-----+ Ethernet52 | - | | eth14 +------+ +--vlan115--| | | Et15 +-----+ Ethernet56 | - | | eth15 +------+ +--vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 | - | | eth16 +------+ +--vlan117--| | | Et17 +-----+ Ethernet64 | - | | eth17 +------+ +--vlan118--| | | Et18 +-----+ Ethernet68 | - | | eth18 +------+ +--vlan119--| | | Et19 +-----+ Ethernet72 | - | | eth19 +------+ +--vlan120--| | | Et20 +-----+ Ethernet76 | - | | eth20 +------+ +--vlan121--| | | Et21 +-----+ Ethernet80 | - | | eth21 +------+ +--vlan122--| | | Et22 +-----+ Ethernet84 | - | | eth22 +------+ +--vlan123--| | | Et23 +-----+ Ethernet88 | - | | eth23 +------+ +--vlan124--| | | Et24 +-----+ Ethernet92 | - | | eth24 +------+ +--vlan125--| | | Et25 +-----+ Ethernet96 | - | | eth25 +------+ +--vlan126--| | | Et26 +-----+ Ethernet100 | - | | eth26 +------+ +--vlan127--| | | Et27 +-----+ Ethernet104 | - | | eth27 +------+ +--vlan128--| | | Et28 +-----+ Ethernet108 | - | | eth28 +------+ +--vlan129--| | | Et29 +-----+ Ethernet112 | - | | eth29 +------+ +--vlan130--| | | Et30 +-----+ Ethernet116 | - | | eth30 +------+ +--vlan131--| | | Et31 +-----+ Ethernet120 | - | | eth31 +------+ +--vlan132--+ | | Et32 +-----+ Ethernet124 | - | +----------------------+ +-------------+ | | | | | - | | | | | | - +---------------------------------------------------------------+ +--------------+ +---------------+ + +----------------------------------------------+ +--------------+ +---------------+ + | PTF Docker | | | | | + | +----------------------+ | | | | | + | | eth0 +------vlan101--+ | | Et1 +-----+ Ethernet0 | + | | eth1 +------vlan102--| | | Et2 +-----+ Ethernet4 | + | | eth2 +------vlan103--| | | Et3 +-----+ Ethernet8 | + | | eth3 +------vlan104--| | | Et4 +-----+ Ethernet12 | + | | eth4 +------vlan105--| | | Et5 +-----+ Ethernet16 | + | | eth5 +------vlan106--| | | Et6 +-----+ Ethernet20 | + | | eth6 +------vlan107--| | | Et7 +-----+ Ethernet24 | + | | eth7 +------vlan108--| | | Et8 +-----+ Ethernet28 | + | | eth8 +------vlan109--| | | Et9 +-----+ Etherent32 | + | | eth9 +------vlan110--| | | Et10 +-----+ Ethernet36 | + | | eth10 +------vlan111--| | | Et11 +-----+ Ethernet40 | + | | eth11 +------vlan112--| | | Et12 +-----+ Ethernet44 | + | | eth12 +------vlan113--| | | Et13 +-----+ Ethernet48 | + | | eth13 +------vlan114--| | | Et14 +-----+ Ethernet52 | + | | eth14 +------vlan115--| | | Et15 +-----+ Ethernet56 | + | | eth15 +------vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 | + | | eth16 +------vlan117--| | | Et17 +-----+ Ethernet64 | + | | eth17 +------vlan118--| | | Et18 +-----+ Ethernet68 | + | | eth18 +------vlan119--| | | Et19 +-----+ Ethernet72 | + | | eth19 +------vlan120--| | | Et20 +-----+ Ethernet76 | + | | eth20 +------vlan121--| | | Et21 +-----+ Ethernet80 | + | | eth21 +------vlan122--| | | Et22 +-----+ Ethernet84 | + | | eth22 +------vlan123--| | | Et23 +-----+ Ethernet88 | + | | eth23 +------vlan124--| | | Et24 +-----+ Ethernet92 | + | | eth24 +------vlan125--| | | Et25 +-----+ Ethernet96 | + | | eth25 +------vlan126--| | | Et26 +-----+ Ethernet100 | + | | eth26 +------vlan127--| | | Et27 +-----+ Ethernet104 | + | | eth27 +------vlan128--| | | Et28 +-----+ Ethernet108 | + | | eth28 +------vlan129--| | | Et29 +-----+ Ethernet112 | + | | eth29 +------vlan130--| | | Et30 +-----+ Ethernet116 | + | | eth30 +------vlan131--| | | Et31 +-----+ Ethernet120 | + | | eth31 +------vlan132--+ | | Et32 +-----+ Ethernet124 | + | +----------------------+ | | | | | + | | | | | | + +----------------------------------------------+ +--------------+ +---------------+ ``` Figure 1: PTF container testbed - *PTF docker*: A docker container that has 32 ports with pre-installed PTF tools. See https://github.com/Azure/sonic-buildimage/tree/master/dockers/docker-ptf -- *Vlan ports*: 32 vlan ports are created on top of physical port, e.g., eth0, inside the Linux host -- *Openvswitch*: Connect 32 vlan ports to the 32 docker ports inside the Linux host. Openvswitch forwards the packet to the corresponding output port based on the incoming port. No L2/L3 forwarding is enabled in the Openvswitch. +- *Vlan ports*: 32 vlan ports are created on top of a physical port, e.g., eth0, inside the Linux host. After creation the vlan ports are injected directly to a ptf docker host. - *Fanout switch*: A physical switch which enables VLAN trunking. Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. Et1-Et32 are vlan access ports and are connect to DUT. ``` From 410c973c7d56ca2c765623f6c6228935c33a1ff9 Mon Sep 17 00:00:00 2001 From: Pavel Shirshov Date: Mon, 14 Nov 2016 12:27:32 -0800 Subject: [PATCH 2/6] Adjust documentation about vm_test --- ansible/README.testbed.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index b7a469acf94..5dd64edd2a9 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -185,9 +185,18 @@ Figure 1: PTF container testbed ``` Figure 2: VM set testbed with injected PTF docker -In this testbed, we have 32 VMs and 1 PTF docker connected to 32 openvswitch. Each openvswitch connected to a vlan interface. -Packets coming from the vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are -sent to the vlan interface. +In this testbed, we have 32 VMs and 1 PTF docker. The VMs use Arista vEOS. Each VM has 10 network interfaces: + 1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port). + 2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above). + 3. 1 management port. This port is used to connect to the VMs + +The ptf docker container connects to the bridges which connect the VMs frontpanel ports and physical vlans. Each bridge has three ports: + 1. Frontpanel port from a VM + 2. Physical vlan port + 3. PTF container port + +Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are +sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time. ## Requirenments for the Linux Host 1. Ubuntu 16.04 x64 @@ -241,6 +250,7 @@ up ip link set p4p1 up * 'vm_X_enabled': true, if you want to run X vm set * 'vm_X_external_iface': name of interface which connected to DUT. See 3.3 * 'vm_X_vlan_base': vlan number which is used for connection to first port of DUT. + * 'vlans': list of vlan offsets for the VM FP ports. For example: if vlans equal to "5,6" it means that the VM frontpanel port 0 will be connected to vlan {{ vm_X_vlan_base + 5 - 1 }} and VM frontpanel port 1 will be connected to vlan {{ vm_X_vlan_base + 6 - 1 }} 7. Edit 'ansible/vars/configurations/*.yml' files. You need to adjust 'minigraph_mgmt_interface' to settings of your network See 3.2 8. Start testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos start_vm_sets.yml --limit server_1 -e vm_set_1=true' 9. Stop testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos stop_vm_sets.yml --limit server_1 -e vm_set_1=true' From 897d5e14cf80cb501d74a388f0e4a19de6e410d1 Mon Sep 17 00:00:00 2001 From: Pavel Shirshov Date: Mon, 14 Nov 2016 12:46:28 -0800 Subject: [PATCH 3/6] Add documentation for ptf-container --- ansible/README.testbed.md | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index 5dd64edd2a9..3a2bd844d3f 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -48,6 +48,25 @@ Figure 1: PTF container testbed - *Fanout switch*: A physical switch which enables VLAN trunking. Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. Et1-Et32 are vlan access ports and are connect to DUT. ``` + +### Deploy testbed with one ptf container +1. clone sonic-mgmt repo to local directory +2. Edit 'ansible/group_vars/vm_host'. Put your credentials to reach the server +3. Check, that you can reach the server by running command 'ansible -i veos -m ping vm_host_1' from ansible directory. The output should contain 'pong' +4. Edit 'ansible/group_vars/vm_host/main.yml'. + * 'http_proxy': your http_proxy + * 'http_proxy': your https_proxy +5. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02 + * 'mgmt_gw': ip address of gateway for management interfaces of ptf_container + * 'mgmt_bridge': the bridge which is used to connect the management network + * 'externel_iface': the interface which is connected to the fanout switch + * 'ptf_X_enabled': true, if you want to run X ptf container + * 'ptf_X_mgmt_ip': which ip is used inside of the container for the management network + * 'ptf_X_vlan_base': vlan number which is used for connection to first port of DUT +7. Edit 'ansible/vars/docker_registry.yml'. You need put your docker registry server here +8. Start ptf container with command 'ansible-playbook -i veos start_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See start_ptf_containers.yml for more examples +9. Stop ptf container with command 'ansible-playbook -i veos stop_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See stop_ptf_containers.yml for more examples + Linux Host Fanout Switch DUT +-------------------------------------------------------------------+ +--------------+ +---------------+ | PTF Docker VM sets Openvswitch | | | | | @@ -216,6 +235,7 @@ iface em1 inet static netmask 255.255.255.0 network 10.250.0.0 broadcast 10.250.0.255 + mtu 9216 gateway 10.250.0.1 dns-nameservers 10.250.0.1 10.250.0.2 # dns-* options are implemented by the resolvconf package, if installed @@ -248,9 +268,10 @@ up ip link set p4p1 up 6. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02 * 'mgmt_gw': ip address of gateway for management interfaces of VM. See 3.2 * 'vm_X_enabled': true, if you want to run X vm set - * 'vm_X_external_iface': name of interface which connected to DUT. See 3.3 * 'vm_X_vlan_base': vlan number which is used for connection to first port of DUT. * 'vlans': list of vlan offsets for the VM FP ports. For example: if vlans equal to "5,6" it means that the VM frontpanel port 0 will be connected to vlan {{ vm_X_vlan_base + 5 - 1 }} and VM frontpanel port 1 will be connected to vlan {{ vm_X_vlan_base + 6 - 1 }} -7. Edit 'ansible/vars/configurations/*.yml' files. You need to adjust 'minigraph_mgmt_interface' to settings of your network See 3.2 +7. Edit 'ansible/minigraph/*.xml' files. You need to adjust following xml nodes to settings of your network: + * DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/Prefix/IPPrefix + * DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/PrefixStr 8. Start testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos start_vm_sets.yml --limit server_1 -e vm_set_1=true' 9. Stop testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos stop_vm_sets.yml --limit server_1 -e vm_set_1=true' From bad5a86f5fdf071803bab5f685c112065338f260 Mon Sep 17 00:00:00 2001 From: Pavel Shirshov Date: Mon, 14 Nov 2016 12:49:59 -0800 Subject: [PATCH 4/6] Fix --- ansible/README.testbed.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index 3a2bd844d3f..e2182125598 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -47,8 +47,6 @@ Figure 1: PTF container testbed - *Vlan ports*: 32 vlan ports are created on top of a physical port, e.g., eth0, inside the Linux host. After creation the vlan ports are injected directly to a ptf docker host. - *Fanout switch*: A physical switch which enables VLAN trunking. Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. Et1-Et32 are vlan access ports and are connect to DUT. -``` - ### Deploy testbed with one ptf container 1. clone sonic-mgmt repo to local directory 2. Edit 'ansible/group_vars/vm_host'. Put your credentials to reach the server @@ -67,6 +65,7 @@ Figure 1: PTF container testbed 8. Start ptf container with command 'ansible-playbook -i veos start_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See start_ptf_containers.yml for more examples 9. Stop ptf container with command 'ansible-playbook -i veos stop_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See stop_ptf_containers.yml for more examples +``` Linux Host Fanout Switch DUT +-------------------------------------------------------------------+ +--------------+ +---------------+ | PTF Docker VM sets Openvswitch | | | | | From ed0c46dd538e9a89aaf0c64c1c8ad2bfe2e05d9e Mon Sep 17 00:00:00 2001 From: Pavel Shirshov Date: Mon, 14 Nov 2016 12:51:57 -0800 Subject: [PATCH 5/6] Add better titles. Put linux server requirenmets on top. --- ansible/README.testbed.md | 78 ++++++++++++++++++++------------------- 1 file changed, 41 insertions(+), 37 deletions(-) diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index e2182125598..3fbfcc15842 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -1,4 +1,41 @@ -# Testbed topology +# Requirenments for the Linux Host +1. Ubuntu 16.04 x64 +2. Installed docker-engine +3. Three network cards: + 1. first is used for the server management + 2. second is used to connect management interfaces of VMs and docker containers to network. + 3. third is used to connect VMs and ptf containers to DUTs + +Content of /etc/network/interfaces: +``` +root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces +# The primary network interface +auto em1 +iface em1 inet static + address 10.250.0.245 + netmask 255.255.255.0 + network 10.250.0.0 + broadcast 10.250.0.255 + mtu 9216 + gateway 10.250.0.1 + dns-nameservers 10.250.0.1 10.250.0.2 + # dns-* options are implemented by the resolvconf package, if installed + dns-search SOMECOMPANY + +auto br1 +iface br1 inet manual + bridge_ports em2 + bridge_stp on + bridge_maxwait 0 + bridge_fd 0 + +auto p4p1 +iface p4p1 inet manual +up ip link set p4p1 up +``` + + +# PTF Testbed topology ``` Linux Host Fanout Switch DUT @@ -65,6 +102,9 @@ Figure 1: PTF container testbed 8. Start ptf container with command 'ansible-playbook -i veos start_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See start_ptf_containers.yml for more examples 9. Stop ptf container with command 'ansible-playbook -i veos stop_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See stop_ptf_containers.yml for more examples + +# VM set testbed topology + ``` Linux Host Fanout Switch DUT +-------------------------------------------------------------------+ +--------------+ +---------------+ @@ -216,42 +256,6 @@ The ptf docker container connects to the bridges which connect the VMs frontpane Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time. -## Requirenments for the Linux Host -1. Ubuntu 16.04 x64 -2. Installed docker-engine -3. Three network cards: - 1. first is used for the server management - 2. second is used to connect management interfaces of VMs and docker containers to network. - 3. third is used to connect VMs and ptf containers to DUTs - -Content of /etc/network/interfaces: -``` -root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces -# The primary network interface -auto em1 -iface em1 inet static - address 10.250.0.245 - netmask 255.255.255.0 - network 10.250.0.0 - broadcast 10.250.0.255 - mtu 9216 - gateway 10.250.0.1 - dns-nameservers 10.250.0.1 10.250.0.2 - # dns-* options are implemented by the resolvconf package, if installed - dns-search SOMECOMPANY - -auto br1 -iface br1 inet manual - bridge_ports em2 - bridge_stp on - bridge_maxwait 0 - bridge_fd 0 - -auto p4p1 -iface p4p1 inet manual -up ip link set p4p1 up -``` - ### Deploy testbed with one VM set 1. clone sonic-mgmt repo to local directory 2. Edit 'ansible/veos' file. Put ip address of your server after 'ansible_host=' From d6330814455111ab19e686bf40af0fa7ba836c4a Mon Sep 17 00:00:00 2001 From: Pavel Shirshov Date: Mon, 14 Nov 2016 15:47:39 -0800 Subject: [PATCH 6/6] Fix a typo --- ansible/README.testbed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ansible/README.testbed.md b/ansible/README.testbed.md index 3fbfcc15842..5928a8131d5 100644 --- a/ansible/README.testbed.md +++ b/ansible/README.testbed.md @@ -1,4 +1,4 @@ -# Requirenments for the Linux Host +# Requirements for the Linux Host 1. Ubuntu 16.04 x64 2. Installed docker-engine 3. Three network cards: