Skip to content

Commit a73e8dd

Browse files
pavel-shirshovShuotian Cheng
pavel-shirshov
authored and
Shuotian Cheng
committed
doc: Update testbed README (#40)
* Remove OVS from the ptf_host * Add documentation for ptf_container
1 parent 005ddfc commit a73e8dd

File tree

1 file changed

+112
-79
lines changed

1 file changed

+112
-79
lines changed

‎ansible/README.testbed.md‎

Lines changed: 112 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,110 @@
1-
# Testbed topology
1+
# Requirements for the Linux Host
2+
1. Ubuntu 16.04 x64
3+
2. Installed docker-engine
4+
3. Three network cards:
5+
1. first is used for the server management
6+
2. second is used to connect management interfaces of VMs and docker containers to network.
7+
3. third is used to connect VMs and ptf containers to DUTs
8+
9+
Content of /etc/network/interfaces:
10+
```
11+
root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces
12+
# The primary network interface
13+
auto em1
14+
iface em1 inet static
15+
address 10.250.0.245
16+
netmask 255.255.255.0
17+
network 10.250.0.0
18+
broadcast 10.250.0.255
19+
mtu 9216
20+
gateway 10.250.0.1
21+
dns-nameservers 10.250.0.1 10.250.0.2
22+
# dns-* options are implemented by the resolvconf package, if installed
23+
dns-search SOMECOMPANY
24+
25+
auto br1
26+
iface br1 inet manual
27+
bridge_ports em2
28+
bridge_stp on
29+
bridge_maxwait 0
30+
bridge_fd 0
31+
32+
auto p4p1
33+
iface p4p1 inet manual
34+
up ip link set p4p1 up
35+
```
36+
37+
38+
# PTF Testbed topology
239

340
```
441
Linux Host Fanout Switch DUT
5-
+---------------------------------------------------------------+ +--------------+ +---------------+
6-
| PTF Docker Openvswitch | | | | |
7-
| +----------------------+ +-------------+ | | | | |
8-
| | eth0 +------+ +--vlan101--+ | | Et1 +-----+ Ethernet0 |
9-
| | eth1 +------+ +--vlan102--| | | Et2 +-----+ Ethernet4 |
10-
| | eth2 +------+ +--vlan103--| | | Et3 +-----+ Ethernet8 |
11-
| | eth3 +------+ +--vlan104--| | | Et4 +-----+ Ethernet12 |
12-
| | eth4 +------+ +--vlan105--| | | Et5 +-----+ Ethernet16 |
13-
| | eth5 +------+ +--vlan106--| | | Et6 +-----+ Ethernet20 |
14-
| | eth6 +------+ +--vlan107--| | | Et7 +-----+ Ethernet24 |
15-
| | eth7 +------+ +--vlan108--| | | Et8 +-----+ Ethernet28 |
16-
| | eth8 +------+ +--vlan109--| | | Et9 +-----+ Etherent32 |
17-
| | eth9 +------+ +--vlan110--| | | Et10 +-----+ Ethernet36 |
18-
| | eth10 +------+ +--vlan111--| | | Et11 +-----+ Ethernet40 |
19-
| | eth11 +------+ +--vlan112--| | | Et12 +-----+ Ethernet44 |
20-
| | eth12 +------+ +--vlan113--| | | Et13 +-----+ Ethernet48 |
21-
| | eth13 +------+ +--vlan114--| | | Et14 +-----+ Ethernet52 |
22-
| | eth14 +------+ +--vlan115--| | | Et15 +-----+ Ethernet56 |
23-
| | eth15 +------+ +--vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 |
24-
| | eth16 +------+ +--vlan117--| | | Et17 +-----+ Ethernet64 |
25-
| | eth17 +------+ +--vlan118--| | | Et18 +-----+ Ethernet68 |
26-
| | eth18 +------+ +--vlan119--| | | Et19 +-----+ Ethernet72 |
27-
| | eth19 +------+ +--vlan120--| | | Et20 +-----+ Ethernet76 |
28-
| | eth20 +------+ +--vlan121--| | | Et21 +-----+ Ethernet80 |
29-
| | eth21 +------+ +--vlan122--| | | Et22 +-----+ Ethernet84 |
30-
| | eth22 +------+ +--vlan123--| | | Et23 +-----+ Ethernet88 |
31-
| | eth23 +------+ +--vlan124--| | | Et24 +-----+ Ethernet92 |
32-
| | eth24 +------+ +--vlan125--| | | Et25 +-----+ Ethernet96 |
33-
| | eth25 +------+ +--vlan126--| | | Et26 +-----+ Ethernet100 |
34-
| | eth26 +------+ +--vlan127--| | | Et27 +-----+ Ethernet104 |
35-
| | eth27 +------+ +--vlan128--| | | Et28 +-----+ Ethernet108 |
36-
| | eth28 +------+ +--vlan129--| | | Et29 +-----+ Ethernet112 |
37-
| | eth29 +------+ +--vlan130--| | | Et30 +-----+ Ethernet116 |
38-
| | eth30 +------+ +--vlan131--| | | Et31 +-----+ Ethernet120 |
39-
| | eth31 +------+ +--vlan132--+ | | Et32 +-----+ Ethernet124 |
40-
| +----------------------+ +-------------+ | | | | |
41-
| | | | | |
42-
+---------------------------------------------------------------+ +--------------+ +---------------+
42+
+----------------------------------------------+ +--------------+ +---------------+
43+
| PTF Docker | | | | |
44+
| +----------------------+ | | | | |
45+
| | eth0 +------vlan101--+ | | Et1 +-----+ Ethernet0 |
46+
| | eth1 +------vlan102--| | | Et2 +-----+ Ethernet4 |
47+
| | eth2 +------vlan103--| | | Et3 +-----+ Ethernet8 |
48+
| | eth3 +------vlan104--| | | Et4 +-----+ Ethernet12 |
49+
| | eth4 +------vlan105--| | | Et5 +-----+ Ethernet16 |
50+
| | eth5 +------vlan106--| | | Et6 +-----+ Ethernet20 |
51+
| | eth6 +------vlan107--| | | Et7 +-----+ Ethernet24 |
52+
| | eth7 +------vlan108--| | | Et8 +-----+ Ethernet28 |
53+
| | eth8 +------vlan109--| | | Et9 +-----+ Etherent32 |
54+
| | eth9 +------vlan110--| | | Et10 +-----+ Ethernet36 |
55+
| | eth10 +------vlan111--| | | Et11 +-----+ Ethernet40 |
56+
| | eth11 +------vlan112--| | | Et12 +-----+ Ethernet44 |
57+
| | eth12 +------vlan113--| | | Et13 +-----+ Ethernet48 |
58+
| | eth13 +------vlan114--| | | Et14 +-----+ Ethernet52 |
59+
| | eth14 +------vlan115--| | | Et15 +-----+ Ethernet56 |
60+
| | eth15 +------vlan116--+---+-- eth0 --+ Et33 Et16 +-----+ Ethernet60 |
61+
| | eth16 +------vlan117--| | | Et17 +-----+ Ethernet64 |
62+
| | eth17 +------vlan118--| | | Et18 +-----+ Ethernet68 |
63+
| | eth18 +------vlan119--| | | Et19 +-----+ Ethernet72 |
64+
| | eth19 +------vlan120--| | | Et20 +-----+ Ethernet76 |
65+
| | eth20 +------vlan121--| | | Et21 +-----+ Ethernet80 |
66+
| | eth21 +------vlan122--| | | Et22 +-----+ Ethernet84 |
67+
| | eth22 +------vlan123--| | | Et23 +-----+ Ethernet88 |
68+
| | eth23 +------vlan124--| | | Et24 +-----+ Ethernet92 |
69+
| | eth24 +------vlan125--| | | Et25 +-----+ Ethernet96 |
70+
| | eth25 +------vlan126--| | | Et26 +-----+ Ethernet100 |
71+
| | eth26 +------vlan127--| | | Et27 +-----+ Ethernet104 |
72+
| | eth27 +------vlan128--| | | Et28 +-----+ Ethernet108 |
73+
| | eth28 +------vlan129--| | | Et29 +-----+ Ethernet112 |
74+
| | eth29 +------vlan130--| | | Et30 +-----+ Ethernet116 |
75+
| | eth30 +------vlan131--| | | Et31 +-----+ Ethernet120 |
76+
| | eth31 +------vlan132--+ | | Et32 +-----+ Ethernet124 |
77+
| +----------------------+ | | | | |
78+
| | | | | |
79+
+----------------------------------------------+ +--------------+ +---------------+
4380
```
4481
Figure 1: PTF container testbed
4582

4683
- *PTF docker*: A docker container that has 32 ports with pre-installed PTF tools. See https://github.com/Azure/sonic-buildimage/tree/master/dockers/docker-ptf
47-
- *Vlan ports*: 32 vlan ports are created on top of physical port, e.g., eth0, inside the Linux host
48-
- *Openvswitch*: Connect 32 vlan ports to the 32 docker ports inside the Linux host. Openvswitch forwards the packet to the corresponding output port based on the incoming port. No L2/L3 forwarding is enabled in the Openvswitch.
84+
- *Vlan ports*: 32 vlan ports are created on top of a physical port, e.g., eth0, inside the Linux host. After creation the vlan ports are injected directly to a ptf docker host.
4985
- *Fanout switch*: A physical switch which enables VLAN trunking. Et33 is a vlan trunking port and is connected to the eth0 port of the linux host. Et1-Et32 are vlan access ports and are connect to DUT.
5086

87+
### Deploy testbed with one ptf container
88+
1. clone sonic-mgmt repo to local directory
89+
2. Edit 'ansible/group_vars/vm_host'. Put your credentials to reach the server
90+
3. Check, that you can reach the server by running command 'ansible -i veos -m ping vm_host_1' from ansible directory. The output should contain 'pong'
91+
4. Edit 'ansible/group_vars/vm_host/main.yml'.
92+
* 'http_proxy': your http_proxy
93+
* 'http_proxy': your https_proxy
94+
5. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02
95+
* 'mgmt_gw': ip address of gateway for management interfaces of ptf_container
96+
* 'mgmt_bridge': the bridge which is used to connect the management network
97+
* 'externel_iface': the interface which is connected to the fanout switch
98+
* 'ptf_X_enabled': true, if you want to run X ptf container
99+
* 'ptf_X_mgmt_ip': which ip is used inside of the container for the management network
100+
* 'ptf_X_vlan_base': vlan number which is used for connection to first port of DUT
101+
7. Edit 'ansible/vars/docker_registry.yml'. You need put your docker registry server here
102+
8. Start ptf container with command 'ansible-playbook -i veos start_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See start_ptf_containers.yml for more examples
103+
9. Stop ptf container with command 'ansible-playbook -i veos stop_ptf_containers.yml --vault-password-file=~/.password --limit server_1 -e ptf_1=true'. See stop_ptf_containers.yml for more examples
104+
105+
106+
# VM set testbed topology
107+
51108
```
52109
Linux Host Fanout Switch DUT
53110
+-------------------------------------------------------------------+ +--------------+ +---------------+
@@ -186,44 +243,18 @@ Figure 1: PTF container testbed
186243
```
187244
Figure 2: VM set testbed with injected PTF docker
188245

189-
In this testbed, we have 32 VMs and 1 PTF docker connected to 32 openvswitch. Each openvswitch connected to a vlan interface.
190-
Packets coming from the vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are
191-
sent to the vlan interface.
192-
193-
## Requirenments for the Linux Host
194-
1. Ubuntu 16.04 x64
195-
2. Installed docker-engine
196-
3. Three network cards:
197-
1. first is used for the server management
198-
2. second is used to connect management interfaces of VMs and docker containers to network.
199-
3. third is used to connect VMs and ptf containers to DUTs
200-
201-
Content of /etc/network/interfaces:
202-
```
203-
root@STR-AZURE-SERV-02:~# cat /etc/network/interfaces
204-
# The primary network interface
205-
auto em1
206-
iface em1 inet static
207-
address 10.250.0.245
208-
netmask 255.255.255.0
209-
network 10.250.0.0
210-
broadcast 10.250.0.255
211-
gateway 10.250.0.1
212-
dns-nameservers 10.250.0.1 10.250.0.2
213-
# dns-* options are implemented by the resolvconf package, if installed
214-
dns-search SOMECOMPANY
246+
In this testbed, we have 32 VMs and 1 PTF docker. The VMs use Arista vEOS. Each VM has 10 network interfaces:
247+
1. 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port).
248+
2. 1 back panel port. All testbed VMs connected to each other using this port (it isn't shown on the figure above).
249+
3. 1 management port. This port is used to connect to the VMs
215250

216-
auto br1
217-
iface br1 inet manual
218-
bridge_ports em2
219-
bridge_stp on
220-
bridge_maxwait 0
221-
bridge_fd 0
251+
The ptf docker container connects to the bridges which connect the VMs frontpanel ports and physical vlans. Each bridge has three ports:
252+
1. Frontpanel port from a VM
253+
2. Physical vlan port
254+
3. PTF container port
222255

223-
auto p4p1
224-
iface p4p1 inet manual
225-
up ip link set p4p1 up
226-
```
256+
Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are
257+
sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time.
227258

228259
### Deploy testbed with one VM set
229260
1. clone sonic-mgmt repo to local directory
@@ -240,8 +271,10 @@ up ip link set p4p1 up
240271
6. Edit 'ansible/host_vars/STR-ACS-SERV-01.yml'. It contains settings for STR-ACS-SERV-01. STR-ACS-SERV-02 contains similar settings which are applied to STR-ACS-SERV-02
241272
* 'mgmt_gw': ip address of gateway for management interfaces of VM. See 3.2
242273
* 'vm_X_enabled': true, if you want to run X vm set
243-
* 'vm_X_external_iface': name of interface which connected to DUT. See 3.3
244274
* 'vm_X_vlan_base': vlan number which is used for connection to first port of DUT.
245-
7. Edit 'ansible/vars/configurations/*.yml' files. You need to adjust 'minigraph_mgmt_interface' to settings of your network See 3.2
275+
* 'vlans': list of vlan offsets for the VM FP ports. For example: if vlans equal to "5,6" it means that the VM frontpanel port 0 will be connected to vlan {{ vm_X_vlan_base + 5 - 1 }} and VM frontpanel port 1 will be connected to vlan {{ vm_X_vlan_base + 6 - 1 }}
276+
7. Edit 'ansible/minigraph/*.xml' files. You need to adjust following xml nodes to settings of your network:
277+
* DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/Prefix/IPPrefix
278+
* DeviceMiniGraph/DpgDec/DeviceDataPlaneInfo/ManagementIPInterfaces/ManagementIPInterface/PrefixStr
246279
8. Start testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos start_vm_sets.yml --limit server_1 -e vm_set_1=true'
247280
9. Stop testbed with command 'ANSIBLE_SCP_IF_SSH=y ansible-playbook -i veos stop_vm_sets.yml --limit server_1 -e vm_set_1=true'

0 commit comments

Comments
 (0)