Skip to content

Update docs for our testbed #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pavel-shirshov merged 6 commits into from
Nov 14, 2016
Merged

Update docs for our testbed #40

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
576d758
Remove ovs from the ptf_host
pavel-shirshov Nov 14, 2016
410c973
Adjust documentation about vm_test
pavel-shirshov Nov 14, 2016
897d5e1
Add documentation for ptf-container
pavel-shirshov Nov 14, 2016
bad5a86
Fix
pavel-shirshov Nov 14, 2016
ed0c46d
Add better titles. Put linux server requirenmets on top.
pavel-shirshov Nov 14, 2016
d633081
Fix a typo
pavel-shirshov Nov 14, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 112 additions & 79 deletions ansible/README.testbed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,53 +1,110 @@
# Testbed topology
# Requirements for the Linux Host
1. Ubuntu 16.04 x64
2. Installed docker-engine
3. Three network cards:
1. first is used for the server management
2. second is used to connect management interfaces of VMs and docker containers to network.
3. third is used to connect VMs and ptf containers to DUTs

Content of /etc/network/interfaces:
```
root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces
# The primary network interface
auto em1
iface em1 inet static
address 10.250.0.245
netmask 255.255.255.0
network 10.250.0.0
broadcast 10.250.0.255
mtu 9216
gateway 10.250.0.1
dns-nameservers 10.250.0.1 10.250.0.2
# dns-* options are implemented by the resolvconf package, if installed
dns-search SOMECOMPANY

auto br1
iface br1 inet manual
bridge_ports em2
bridge_stp on
bridge_maxwait 0
bridge_fd 0

auto p4p1
iface p4p1 inet manual
up ip link set p4p1 up
```


# PTF Testbed topology

```
Linux Host Fanout Switch DUT
+---------------------------------------------------------------+ +--------------+ +---------------+
| PTF Docker Openvswitch | | | | |
| +----------------------+ +-------------+ | | | | |
| | eth0 +------+ +--vlan101--+ | | Et1 +-----+ Ethernet0 |
| | eth1 +------+ +--vlan102--| | | Et2 +-----+ Ethernet4 |
| | eth2 +------+ +--vlan103--| | | Et3 +-----+ Ethernet8 |
| | eth3 +------+ +--vlan104--| | | Et4 +-----+ Ethernet12 |
| | eth4 +------+ +--vlan105--| | | Et5 +-----+ Ethernet16 |
| | eth5 +------+ +--vlan106--| | | Et6 +-----+ Ethernet20 |
| | eth6 +------+ +--vlan107--| | | Et7 +-----+ Ethernet24 |
| | eth7 +------+ +--vlan108--| | | Et8 +-----+ Ethernet28 |
| | eth8 +------+ +--vlan109--| | | Et9 +-----+ Etherent32 |
| | eth9 +------+ +--vlan110--| | | Et10 +-----+ Ethernet36 |
| | eth10 +------+ +--vlan111--| | | Et11 +-----+ Ethernet40 |
| | eth11 +------+ +--vlan112--| | | Et12 +-----+ Ethernet44 |
| | eth12 +------+ +--vlan113--| | | Et13 +-----+ Ethernet48 |
| | eth13 +------+ +--vlan114--| | | Et14 +-----+ Ethernet52 |
| | eth14 +------+ +--vlan115--| | | Et15 +-----+ Ethernet56 |
| | eth15 +------+ +--vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 |
| | eth16 +------+ +--vlan117--| | | Et17 +-----+ Ethernet64 |
| | eth17 +------+ +--vlan118--| | | Et18 +-----+ Ethernet68 |
| | eth18 +------+ +--vlan119--| | | Et19 +-----+ Ethernet72 |
| | eth19 +------+ +--vlan120--| | | Et20 +-----+ Ethernet76 |
| | eth20 +------+ +--vlan121--| | | Et21 +-----+ Ethernet80 |
| | eth21 +------+ +--vlan122--| | | Et22 +-----+ Ethernet84 |
| | eth22 +------+ +--vlan123--| | | Et23 +-----+ Ethernet88 |
| | eth23 +------+ +--vlan124--| | | Et24 +-----+ Ethernet92 |
| | eth24 +------+ +--vlan125--| | | Et25 +-----+ Ethernet96 |
| | eth25 +------+ +--vlan126--| | | Et26 +-----+ Ethernet100 |
| | eth26 +------+ +--vlan127--| | | Et27 +-----+ Ethernet104 |
| | eth27 +------+ +--vlan128--| | | Et28 +-----+ Ethernet108 |
| | eth28 +------+ +--vlan129--| | | Et29 +-----+ Ethernet112 |
| | eth29 +------+ +--vlan130--| | | Et30 +-----+ Ethernet116 |
| | eth30 +------+ +--vlan131--| | | Et31 +-----+ Ethernet120 |
| | eth31 +------+ +--vlan132--+ | | Et32 +-----+ Ethernet124 |
| +----------------------+ +-------------+ | | | | |
| | | | | |
+---------------------------------------------------------------+ +--------------+ +---------------+
+----------------------------------------------+ +--------------+ +---------------+
| PTF Docker | | | | |
| +----------------------+ | | | | |
| | eth0 +------vlan101--+ | | Et1 +-----+ Ethernet0 |
| | eth1 +------vlan102--| | | Et2 +-----+ Ethernet4 |
| | eth2 +------vlan103--| | | Et3 +-----+ Ethernet8 |
| | eth3 +------vlan104--| | | Et4 +-----+ Ethernet12 |
| | eth4 +------vlan105--| | | Et5 +-----+ Ethernet16 |
| | eth5 +------vlan106--| | | Et6 +-----+ Ethernet20 |
| | eth6 +------vlan107--| | | Et7 +-----+ Ethernet24 |
| | eth7 +------vlan108--| | | Et8 +-----+ Ethernet28 |
| | eth8 +------vlan109--| | | Et9 +-----+ Etherent32 |
| | eth9 +------vlan110--| | | Et10 +-----+ Ethernet36 |
| | eth10 +------vlan111--| | | Et11 +-----+ Ethernet40 |
| | eth11 +------vlan112--| | | Et12 +-----+ Ethernet44 |
| | eth12 +------vlan113--| | | Et13 +-----+ Ethernet48 |
| | eth13 +------vlan114--| | | Et14 +-----+ Ethernet52 |
| | eth14 +------vlan115--| | | Et15 +-----+ Ethernet56 |
| | eth15 +------vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 |
| | eth16 +------vlan117--| | | Et17 +-----+ Ethernet64 |
| | eth17 +------vlan118--| | | Et18 +-----+ Ethernet68 |
| | eth18 +------vlan119--| | | Et19 +-----+ Ethernet72 |
| | eth19 +------vlan120--| | | Et20 +-----+ Ethernet76 |
| | eth20 +------vlan121--| | | Et21 +-----+ Ethernet80 |
| | eth21 +------vlan122--| | | Et22 +-----+ Ethernet84 |
| | eth22 +------vlan123--| | | Et23 +-----+ Ethernet88 |
| | eth23 +------vlan124--| | | Et24 +-----+ Ethernet92 |
| | eth24 +------vlan125--| | | Et25 +-----+ Ethernet96 |
| | eth25 +------vlan126--| | | Et26 +-----+ Ethernet100 |
| | eth26 +------vlan127--| | | Et27 +-----+ Ethernet104 |
| | eth27 +------vlan128--| | | Et28 +-----+ Ethernet108 |
| | eth28 +------vlan129--| | | Et29 +-----+ Ethernet112 |
| | eth29 +------vlan130--| | | Et30 +-----+ Ethernet116 |
| | eth30 +------vlan131--| | | Et31 +-----+ Ethernet120 |
| | eth31 +------vlan132--+ | | Et32 +-----+ Ethernet124 |
| +----------------------+ | | | | |
| | | | | |
+----------------------------------------------+ +--------------+ +---------------+
```
Figure 1: PTF container testbed

- *PTF docker*: A docker container that has 32 ports with pre-installed PTF tools. See https://github.com/Azure/sonic-buildimage/tree/master/dockers/docker-ptf
- *Vlan ports*: 32 vlan ports are created on top of physical port, e.g., eth0, inside the Linux host
- *Openvswitch*: Connect 32 vlan ports to the 32 docker ports inside the Linux host. Openvswitch forwards the packet to the corresponding output port based on the incoming port. No L2/L3 forwarding is enabled in the Openvswitch.
- *Vlan ports*: 32 vlan ports are created on top of a physical port, e.g., eth0, inside the Linux host. After creation the vlan ports are injected directly to a ptf docker host.
- *Fanout switch*: A physical switch which enables VLAN trunking. Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. Et1-Et32 are vlan access ports and are connect to DUT.

### Deploy testbed with one ptf container
1. clone sonic-mgmt repo to local directory
2. Edit 'ansible/group_vars/vm_host'. Put your credentials to reach the server
3. Check, that you can reach the server by running command 'ansible -i veos -m ping vm_host_1' from ansible directory. The output should contain 'pong'
4. Edit 'ansible/group_vars/vm_host/main.yml'.
* 'http_proxy': your http_proxy
* 'http_proxy': your https_proxy
5. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02
* 'mgmt_gw': ip address of gateway for management interfaces of ptf_container
* 'mgmt_bridge': the bridge which is used to connect the management network
* 'externel_iface': the interface which is connected to the fanout switch
* 'ptf_X_enabled': true, if you want to run X ptf container
* 'ptf_X_mgmt_ip': which ip is used inside of the container for the management network
* 'ptf_X_vlan_base': vlan number which is used for connection to first port of DUT
7. Edit 'ansible/vars/docker_registry.yml'. You need put your docker registry server here
8. Start ptf container with command 'ansible-playbook -i veos start_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See start_ptf_containers.yml for more examples
9. Stop ptf container with command 'ansible-playbook -i veos stop_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See stop_ptf_containers.yml for more examples


# VM set testbed topology

```
Linux Host Fanout Switch DUT
+-------------------------------------------------------------------+ +--------------+ +---------------+
Expand Down Expand Up @@ -186,44 +243,18 @@ Figure 1: PTF container testbed
```
Figure 2: VM set testbed with injected PTF docker

In this testbed, we have 32 VMs and 1 PTF docker connected to 32 openvswitch. Each openvswitch connected to a vlan interface.
Packets coming from the vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are
sent to the vlan interface.

## Requirenments for the Linux Host
1. Ubuntu 16.04 x64
2. Installed docker-engine
3. Three network cards:
1. first is used for the server management
2. second is used to connect management interfaces of VMs and docker containers to network.
3. third is used to connect VMs and ptf containers to DUTs

Content of /etc/network/interfaces:
```
root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces
# The primary network interface
auto em1
iface em1 inet static
address 10.250.0.245
netmask 255.255.255.0
network 10.250.0.0
broadcast 10.250.0.255
gateway 10.250.0.1
dns-nameservers 10.250.0.1 10.250.0.2
# dns-* options are implemented by the resolvconf package, if installed
dns-search SOMECOMPANY
In this testbed, we have 32 VMs and 1 PTF docker. The VMs use Arista vEOS. Each VM has 10 network interfaces:
1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port).
2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above).
3. 1 management port. This port is used to connect to the VMs

auto br1
iface br1 inet manual
bridge_ports em2
bridge_stp on
bridge_maxwait 0
bridge_fd 0
The ptf docker container connects to the bridges which connect the VMs frontpanel ports and physical vlans. Each bridge has three ports:
1. Frontpanel port from a VM
2. Physical vlan port
3. PTF container port

auto p4p1
iface p4p1 inet manual
up ip link set p4p1 up
```
Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are
sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time.

### Deploy testbed with one VM set
1. clone sonic-mgmt repo to local directory
Expand All @@ -240,8 +271,10 @@ up ip link set p4p1 up
6. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02
* 'mgmt_gw': ip address of gateway for management interfaces of VM. See 3.2
* 'vm_X_enabled': true, if you want to run X vm set
* 'vm_X_external_iface': name of interface which connected to DUT. See 3.3
* 'vm_X_vlan_base': vlan number which is used for connection to first port of DUT.
7. Edit 'ansible/vars/configurations/*.yml' files. You need to adjust 'minigraph_mgmt_interface' to settings of your network See 3.2
* 'vlans': list of vlan offsets for the VM FP ports. For example: if vlans equal to "5,6" it means that the VM frontpanel port 0 will be connected to vlan {{ vm_X_vlan_base + 5 - 1 }} and VM frontpanel port 1 will be connected to vlan {{ vm_X_vlan_base + 6 - 1 }}
7. Edit 'ansible/minigraph/*.xml' files. You need to adjust following xml nodes to settings of your network:
* DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/Prefix/IPPrefix
* DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/PrefixStr
8. Start testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos start_vm_sets.yml --limit server_1 -e vm_set_1=true'
9. Stop testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos stop_vm_sets.yml --limit server_1 -e vm_set_1=true'