split testbed readme into multiple sections

ansible/doc/README.testbed.Cli.md

@@ -0,0 +1,50 @@
+# Testbed Command Line
+
+```testbed-cli.sh``` is the command line to setup/teardown the testbed, as well as add/remove/switch topology.
+
+- Maintenance purposes only
+  - ```./testbed-cli.sh start-vms {server_name} ~./password```   # after a server restarted
+  - ```./testbed-cli.sh stop-vms {server_name} ~./password```    # before a server restarted
+- General usage
+  - ```./testbed-cli.sh add-topo {topo_name} ~./password```      # create topo with name {topo_name} from testbed.csv
+  - ```./testbed-cli.sh remove-topo {topo_name} ~./password```   # destroy topo with name {topo_name} from testbed.csv
+  - ```./testbed-cli.sh renumber-topo {topo_name} ~./password``` # renumber topo with name {topo_name} from testbed.csv
+
+## Add/Remove topo
+```
+# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner
+vms1-1-t1,vms1-1,t1,docker-ptf-sai-mlnx,10.0.10.5/23,server_1,VM0100,str-msn2700-11,t1 tests
+vms1-1-t1-lag,vms1-1,t1-lag,docker-ptf-sai-mlnx,10.0.10.5/23,server_1,VM0100,str-msn2700-11,t1-lag tests
+
+```
+Goal is to use one VM with different topologies
+
+- To add a new testbed “vms1-1-t1”:
+  - ./testbed-cli add-topo vms1-1-t1 ~/.password
+
+- To switch from testbed “vms1-1-t1” to testbed “vms1-1-lag”
+  - ./testbed-cli remove-topo vms1-1-t1 ~/.password
+  - ./testbed-cli add-topo vms1-1-t1-lag ~/.password
+
+Feature: The VMs configuration will be updated while switching from one topo to another
+Feature: Might be used for renumbering too
+Caveat: Have to remember what was the initial topology. Should be fixed in future
+
+# Renumber topo
+```
+# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner
+vms2-2-b,vms2-2,t1,docker-ptf-sai-brcm,10.0.10.7/23,server_1,VM0100,str-d6000-05,brcm test
+vms2-2-m,vms2-2,t1,docker-ptf-sai-mlnx,10.0.10.7/23,server_1,VM0100,str-msn2700-5,mlnx test
+
+```
+Goal is to use one VM set against different DUTs
+
+- To add a new testbed “vms2-2-b”:
+  - ./testbed-cli add-topo vms2-2-b ~/.password
+
+- To switch from testbed “vms2-2-b” to testbed “vms2-2-m”
+  - ./testbed-cli renumber-topo vms2-2-m ~/.password
+
+Feature: The VMs configuration will NOT be updated while switching from one topo to another (faster).
+
+TODO: check topo field when renumbering between topologies

ansible/doc/README.testbed.Config.md

@@ -0,0 +1,43 @@
+# Testbed Configuration
+
+```testbed.csv``` is the topology configuration file for the testbed.
+
+# ```testbed.csv```
+```
+# conf-name,group-name,topo,ptf_image_name,ptf_mgmt_ip,server,vm_base,dut,comment
+ptf1-m,ptf1,ptf32,docker-ptf-sai-mlnx,10.255.0.188/24,server_1,,str-msn2700-01,Tests ptf
+vms-t1,vms1-1,t1,docker-ptf-sai-mlnx,10.255.0.178/24,server_1,VM0100,str-msn2700-01,Tests vms
+vms-t1-lag,vms1-1,t1-lag,docker-ptf-sai-mlnx,10.255.0.178/24,server_1,VM0100,str-msn2700-01,Tests vms
+
+```
+
+- conf-name - to address row in table
+- group-name – used in interface names, up to 8 characters
+- topo – name of topology
+- ptf_imagename – defines PTF image
+- ptf_mgmt_ip – ip address for mgmt interface of PTF container
+- server – server where the testbed resides
+- vm_base – first VM for the testbed. If empty, no VMs are used
+- dut – target dut name
+- comment – any text here
+
+# ```testbed.csv``` consistency rules
+```
+# uniq-name,testbed-name,topo,ptf_image_name,ptf_ip,server,vm_base,dut,owner
+vms2-2-b,vms2-2,t1,docker-ptf-sai-brcm,10.0.10.7/23,server_1,VM0100,str-d6000-05,brcm test
+vms2-2-m,vms2-2,t1,docker-ptf-sai-mlnx,10.0.10.7/23,server_1,VM0100,str-msn2700-5,mlnx test
+
+```
+Must be strictly checked in code reviews
+ - uniq-name must be unique
+ - All testbed records with the same testbed-name must have the same:
+   - ptf_ip
+   - server
+   - vm_base
+ - testbed-name must be up to 8 characters long
+ - topo name must be valid (topo registered in ```veos``` and topo file presented in vars/topo_*.yml
+ - ptf_imagename must be valid
+ - server name must be valid and presented in veos inventory file
+ - vm_base must not overlap with testbeds from different groups (different test-name)
+
+TODO: check this constraints in testbed-cli.sh

ansible/doc/README.testbed.FAQ.md

@@ -0,0 +1,13 @@
+# FAQ
+
+## How to find IP addresses of VMs and PTF
+ - IP address of testbed PTF container could be found in testbed.csv
+ - To find some VM IP address:
+   - find vm_offset parameter for the VM in your topology file
+   - find vm_base parameter in testbed.csv
+   - Calculate physical VM name as vm_base + vm_offset
+   - Find physical VM entry in veos file
+
+TODO: Create ansible playbook for this
+
+

ansible/doc/README.testbed.Internal.md

@@ -0,0 +1,129 @@
+# Testbed internals
+
+## Topology definition
+
+ - List of currently defined topologies in veos inventory file
+```
+[servers:vars]
+topologies=[‘t1', ‘t1-lag', 't0', 'ptf32', 'ptf64']
+```
+ - Topologies stored inside of vars/topo_*.yml, where * is a topology name
+ - Configuration templates for the topologies saved in roles/eos/templates/*.yml
+
+## Topology file
+
+ - Topology file is a regular ansible yaml file with variables:
+   - topology – defines physical topology
+   - configuration – defines variables for VMs configuration templates
+   - configuration_properties – defines group variables for VMs configuration templates
+
+ - topology dictionary is required
+ - configuration and configuration_properties are optional and used only for topologies with VMs
+
+## Topology file. topology dictionary
+
+ - Two dictionaries:
+   - host_interface – defines a list of port offsets which would be inserted into the PTF container
+   - VMs – defines a list and a physical configuration of VMs used in topology
+
+```
+topology:
+  host_interfaces:
+    - 0
+    - 1
+  VMs:
+    ARISTA01T1:
+      vlans:
+        - 2
+        - 3
+      vm_offset: 0
+    ARISTA02T1:
+      vlans:
+        - 4
+      vm_offset: 1
+
+```
+ - ARISTA01T1 – hostname for a VM
+ - vlans - list of vlan offsets used in VM
+ - vm_offset – offset of VM with base configured as vm_base in testbed.csv
+
+ - In this example:
+   - Let’s consider: vm_base == VM0100, vlan_base == ‘100’
+   - First VM:
+     - hostname ARISTA01T1
+     - Uses VM with physical name VM0100
+     - Ethernet1 is connected to vlan 102
+     - Ethernet2 is connected to vlan 103
+     - Ethernet9 is connected to backplane network (implicitly configured)
+   - Second VM:
+     - hostname ARISTA02T1
+     - Uses VM with physical name VM0101 (vm_offset: 1 + vm_base: VM0100)
+     - Ethernet1 is connected to vlan 104
+     - Ethernet9 is connected to backplane network (implicitly configured)
+   - PTF container:
+     - 5 ethernet interfaces:
+     - eth0 is directly connected to DUT. vlan 100
+     - eth1 is directly connected to DUT. vlan 101
+     - eth2 is injected interface for Ethernet1 of VM ARISTA01T1
+     - eth3 is injected interface for Ethernet2 of VM ARISTA01T1
+     - eth4 is injected interface for Ethernet1 of VM ARISTA02T1
+
+## Topology file. configuration_properties
+```
+configuration_properties:
+  common:
+    nhipv4: 10.10.246.100
+    nhipv6: FC0A::C9
+  spine:
+    swrole: spine
+    podset_number: 200
+    tor_number: 16
+    tor_subnet_number: 2
+    leaf_asn_start: 62001
+    tor_asn_start: 65501
+    failure_rate: 0
+  tor:
+    swrole: tor
+    tor_subnet_number: 5
+
+```
+ - Configuration properties contains any number of dictionary entries
+ - You could have as many as you want
+ - Lately you can refer to these entries in your configuration dictionary. See entry “properties”
+ - You could use them as {{ props.property_name }} inside of jinja2 template. Example: {% for tor in range(0, props.tor_number) %}
+
+## Topology file. configuration
+```
+configuration:
+  ARISTA01T2:
+    properties:
+    - common
+    - spine
+    bgp:
+      asn: 65200
+      peers:
+        65100:
+        - 10.0.0.0
+        - FC00::1
+    interfaces:
+      Loopback0:
+        ipv4: 100.1.0.1/32
+        ipv6: 2064:100::1/128
+      Ethernet1:
+        ipv4: 10.0.0.1/31
+        ipv6: fc00::2/126
+      Ethernet9:
+        ipv4: 10.10.246.1/24
+        ipv6: fc0a::2/64
+
+```
+ - Configurations contains any number of dictionary entries
+ - You could have as many as you want
+ - You have to have entry properties when you want to bring some common property into the configuration
+ - You could use configuration as {{configuration[hostname].property }} inside of jinja2 template.
+
+   Example:
+```
+{% set host = configuration[hostname] %}
+{% for name, iface in host['interfaces'].items() %}
+```

ansible/doc/README.testbed.Overview.md

@@ -0,0 +1,56 @@
+# SONiC Testbed Overview
+
+This document gives overview of the SONiC testbed. 
+
+## Physical topology
+
+![](img/testbed.png)
+
+1. Every DUT port is connected to one of leaf fanout switches
+2. Every leaf fanout switch has unique vlan tag for every DUT port
+3. Root fanout switch connects leaf fanout switches and testbed servers
+4. Connections from root fanout switches are 802.1Q trunks
+5. Any testbed server can access any DUT port by sending a packet with the port vlan tag (root fanout switch should have this vlan number enabled on the server trunk)
+
+## Fanout switch
+*Fanout switch*: A physical switch which enables VLAN trunking. 
+   * Hardware SKU: Arista 7260 or similar
+   * Et33 is a vlan trunking port and is connected to the eth0 port of the linux host.
+   * Et1-Et32 are vlan access ports and are connect to DUT.
+   * Enable LACP/LLDP passthrough
+   * Disable spanning tree protocol
+
+## Testbed server
+
+- Server SKU (this is what we are using, not mandatory): Dell 730; 2 CPUs each has 18 cores; 192G memory; hard disk:2X500G
+- NIC: Mellanox MT27700 Family CX4
+
+![](img/testbed-server.png)
+
+### Network connections
+
+- The testbed server has 2 network ports:
+  - Trunk port (Mellanox MT27700 Family CX4) to root fanout switch
+  - Server management port to manage the server, VMs and PTF containers on the server
+
+### VMs
+
+The VMs use Arista vEOS. They are using to setup protocol test such as BGP, LACP, LLDP. They are created using ```testbed-cli.sh start-vms``` command. Each VM uses 2G of RAM and has 10 network interfaces:
+
+- 8 front panel ports. These ports are connected to openvswitch bridges, which are connected to vlan interfaces. The vlan interfaces are connected to the fanout switch (through physical port).
+- 1 back panel port. All VMs in one topology have their 9th port connected to this backplane network.
+- 1 management port. This port is used to connect to the VMs
+
+### PTF
+
+PTF container is used to send and receive data packets to validate the DUT data plane.
+
+#### PTF with direct port
+![](img/testbed-direct.png)
+
+DUT front panel port is directly connected to one of PTF container ports. Usually eth0 port of PTF container connects Ethernet0 port of DUT, eth1 port of PTF container connects Ethernet4 port of DUT and so on. This is usually used in PTF topologies to connect DUT ports to PTF container ports.
+
+#### PTF with injected port
+![](img/testbed-injected.png)
+
+DUT front panel port is directly connected to one of VMs interfaces. But also we have a tap into this connection. Packets coming from the physical vlan interface are sent to both the VMs and the PTF docker. Packets from the VM and PTF docker are sent to the vlan interface. It allows us to inject packets from the PTF host to DUT and maintain a BGP session between VM and DUT at the same time.

ansible/doc/README.testbed.Setup.md

@@ -0,0 +1,93 @@
+# Testbed Setup
+
+This document describes the steps to setup the testbed and deploy a topology.
+
+## Prepare testbed server
+
+- Install Ubuntu 16.04 or 17.04 amd64 server.
+- Setup management port configuration using sample ```/etc/network/interfaces```.
+
+```
+root@server-1:~# cat /etc/network/interfaces
+# The management network interface
+auto ma0
+iface ma0 inet manual
+
+# Server, VM and PTF management interface
+auto br1
+iface br1 inet static
+    bridge_ports ma0
+    bridge_stp off
+    bridge_maxwait 0
+    bridge_fd 0
+    address 10.250.0.245
+    netmask 255.255.255.0
+    network 10.250.0.0
+    broadcast 10.250.0.255
+    gateway 10.250.0.1
+    dns-nameservers 10.250.0.1 10.250.0.2
+    # dns-* options are implemented by the resolvconf package, if installed
+```
+
+- Installed python 2.7 (required by ansible).
+- Add Docker’s official GPG key
+```
+   $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
+```
+
+## Build and run ```sonic-mgmt``` docker
+
+ansible playbook in ```sonic-mgmt``` repo requires to setup ansible and various dependencies.
+We have built a ```sonic-mgmt``` docker that installs all dependencies, and you can simply
+build that docker and ansible playbook inside the docker.
+
+- Build ```sonic-mgmt``` docker
+```
+git clone --recursive https://github.com/Azure/sonic-buildimage.git
+make target/docker-sonic-mgmt.gz
+```
+
+- Run ```sonic-mgmt``` docker
+```
+docker load -i target/docker-sonic-mgmt.gz
+docker run -it docker-sonic-mgmt bash
+cd ~/sonic-mgmt
+```
+
+From now on, all steps are running inside the ```sonic-mgmt``` docker.
+
+## Prepare testbed configurations
+
+Prepare various configuration files for your testbed.
+
+- Add/Update your testbed server management IP in veos file. Example:'STR-ACS-SERV-01 ansible_host=10.0.0.5' where 10.0.0.5 your server mgmt ip
+- Add testbed server credentials in ```ansible/group_vars/vm_host/creds.yml```
+- Check that ansible could reach this device by command ```ansible -m ping -i veos vm_host_1```
+- Put files: ```Aboot-veos-serial-8.0.0.iso``` and ```vEOS-lab-4.15.9M.vmdk``` to /home/{your_username from step 3}/veos-vm/images on your testbed server
+- Edit ```ansible/host_vars/STR-ACS-SERV-01.yml```. You need to change ```external_iface```,```mgmt_gw``` and ```mgmt_prefixlen```. These settings define network parameters for VM/ptf management interfaces. Example:
+
+```
+external_iface: p4p1   <--- trunk port of the server (connected to the fanout switch)
+mgmt_gw: 10.250.0.1    <--- ip of gateway for VM mgmt interfaces
+mgmt_prefixlen: 24     <--- prefixlen for management interfaces
+```
+
+- Add ip addresses for your VMs in veos inventory file ```ansible/veos``` inventory file. These IP addresses should be in the management subnet defined in above file.
+- Update VM credentials in ```ansible/group_vars/eos/creds.yml```. Use root:123456 as credentials
+- Add information about your docker registry here: ```vars/docker_registry.yml```
+
+## Setup VMs in the server
+
+
+```
+./testbed-cli.sh start-vms server_1 password.txt
+```
+
+Check that all VMs are up and running: ```ansible -m ping -i veos server_1```
+
+## Deploy topology
+
+- Update testbed.csv with your data. At least update PTF mgmt interface settings
+- To deploy PTF topology run: ```./testbed-cli.sh add-topo ptf1-m ~/.password```
+- To remove PTF topology run: ```./testbed-cli.sh remove-topo ptf1-m ~/.password```
+- To deploy T1 topology run: ```./testbed-cli.sh add-topo vms-t1 ~/.password```