Skip to content

[kvm/docs] Improve KVM setup instructions #2476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daall merged 2 commits into from
Nov 6, 2020
Merged

[kvm/docs] Improve KVM setup instructions #2476

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d84f810
[kvm/docs] Improve KVM setup instructions
daall Nov 6, 2020
f51be58
Add j2 install and docker setup instructions
daall Nov 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 90 additions & 47 deletions ansible/doc/README.testbed.VsSetup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,77 +1,102 @@
# Testbed Setup
# KVM Testbed Setup
This document describes the steps to setup a virtual switch based testbed, deploy a T0 topology to it, and run a quick test to verify it is working as expected.

This document describes the steps to setup the virtual switch based testbed and deploy a topology.
## Prepare testbed host
First, we need to prepare the host where we will be configuring the virtual testbed and running the tests.

## Prepare testbed server
1. Install Ubuntu 20.04 AMD64 on your host or VM
- To setup a T0 topology, the server needs to have at least 10GB of memory free
- If the testbed host is a VM, then it must support nested virtualization
- [Instructions for Hyper-V based VMs](https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization#configure-nested-virtualization)

2. Run the host setup script to install required packages and initialize the management bridge network

- Install Ubuntu 20.04 amd64 server. To setup a T0 topology, the server needs to have 10GB free memory.
- Setup internal management network:
```
$ git clone https://github.com/Azure/sonic-mgmt
$ cd sonic-mgmt/ansible
$ sudo ./setup-management-network.sh
git clone https://github.com/Azure/sonic-mgmt
cd sonic-mgmt/ansible
sudo ./setup-management-network.sh
```

### Use vEOS image
3. [Install Docker CE](https://docs.docker.com/install/linux/docker-ce/ubuntu/). Be sure to follow the [post-install instructions](https://docs.docker.com/install/linux/linux-postinstall/) so that you don't need sudo privileges to run docker commands.

## Download an EOS VM image
We currently use EOS-based VMs to simulate neighboring devices in the virtual testbed, much like we do for physical testbeds. To do so, we need to download the image to our testbed host.

- Download vEOS image from [arista](https://www.arista.com/en/support/software-download).
- Copy below image files to `~/veos-vm/images` on your testbed server.
### Option 1: vEOS (KVM-based) image
1. Download the [vEOS image from Arista](https://www.arista.com/en/support/software-download)
2. Copy below image files to `~/veos-vm/images` on your testbed host:
- `Aboot-veos-serial-8.0.0.iso`
- `vEOS-lab-4.20.15M.vmdk`

### Use cEOS image (experimental)
#### Option 1, download and import cEOS image manually
- Download cEOS image from [arista](https://www.arista.com/en/support/software-download) onto your testbed server
- Import cEOS image (It will take several minutes to import, so please be patient)
### Option 2: cEOS (container-based) image (experimental)
#### Option 2.1: Download and import cEOS image manually
1. Download the [cEOS image from Arista](https://www.arista.com/en/support/software-download)
2. Import the cEOS image (it will take several minutes to import, so please be patient!)

```
$ docker import cEOS64-lab-4.23.2F.tar.xz ceosimage:4.23.2F
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ceosimage 4.23.2F d53c28e38448 2 hours ago 1.82GB
```
#### Option 2, download and image cEOS image automatically
Alternatively, you can host the cEOS image on a http server. Specify `vm_images_url` for downloading the image [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/group_vars/vm_host/main.yml#L2). If a saskey is required for downloading cEOS image, specify `ceosimage_saskey` in `sonic-mgmt/ansible/vars/azure_storage.yml`.

If you want to skip image downloading when the cEOS image is not imported locally, set `skip_ceos_image_downloading` to `true` in `sonic-mgmt/ansible/group_vars/all/ceos.yml`. Then when cEOS image is not locally imported, the scripts will not try to download it and will fail with an error message. Please use option 1 to download and import the cEOS image manually.
#### Option 2.2: Pull cEOS image automatically
1. Alternatively, you can host the cEOS image on a http server. Specify `vm_images_url` for downloading the image [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/group_vars/vm_host/main.yml#L2).

2. If a SAS key is required for downloading the cEOS image, specify `ceosimage_saskey` in `sonic-mgmt/ansible/vars/azure_storage.yml`.

If you want to skip downloading the image when the cEOS image is not imported locally, set `skip_ceos_image_downloading` to `true` in `sonic-mgmt/ansible/group_vars/all/ceos.yml`. Then, when the cEOS image is not locally available, the scripts will not try to download it and will fail with an error message. Please use option 1 to download and import the cEOS image manually.

## Download sonic-vs image
## Download the sonic-vs image
To run the tests with a virtual SONiC device, we need a virtual SONiC image. The simplest way to do so is to download a public build from Jenkins.

1. Download the sonic-vs image from [here](https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.img.gz)

- Download sonic-vs image from [here](https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.img.gz)
```
$ wget https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/lastSuccessfulBuild/artifact/target/sonic-vs.img.gz
```

- unzip the image and move it into `~/sonic-vm/images/`
2. Unzip the image and move it into `~/sonic-vm/images/`

```
$ gzip -d sonic-vs.img.gz
$ mkdir -p ~/sonic-vm/images
$ mv sonic-vs.img ~/sonic-vm/images
```

## Setup sonic-mgmt docker
All testbed configuration steps and tests are run from a `sonic-mgmt` docker container. This container has all the necessary packages and tools for SONiC testing so that test behavior is consistent between different developers and lab setups.

Run the `setup-container.sh` in the root directory of the sonic-mgmt repository:
1. Run the `setup-container.sh` in the root directory of the sonic-mgmt repository:

```
$ cd sonic-mgmt
$ ./setup-container.sh -n <container name> -d /data
```

From now on, all steps are running inside the *sonic-mgmt* docker except where otherwise specified.
2. From now on, **all steps are running inside the sonic-mgmt docker**, unless otherwise specified.


You can enter your sonic-mgmt container with the following command:

```
$ docker exec -it <container name> bash
```

### Setup public key to login into the linux host from sonic-mgmt docker
You will find your sonic-mgmt directory mounted at `/data/sonic-mgmt`:

- Modify `veos_vtb` to use the user name, e.g., `foo` to login linux host (this can be your username on the host).
```
$ ls /data/sonic-mgmt/
LICENSE README.md __pycache__ ansible docs lgtm.yml setup-container.sh spytest test_reporting tests
```

## Setup host public key in sonic-mgmt docker
In order to configure the testbed on your host automatically, Ansible needs to be able to SSH into it without a password prompt. The `setup-container` script from the previous step will setup all the necessary SSH keys for you, but there are a few more modifications needed to make Ansible work:

1. Modify `veos_vtb` to use the user name (e.g. `foo`) you want to use to login to the host machine (this can be your username on the host)

```
lgh@gulv-vm2:/data/sonic-mgmt/ansible$ git diff
foo@sonic:/data/sonic-mgmt/ansible$ git diff
diff --git a/ansible/veos_vtb b/ansible/veos_vtb
index 3e7b3c4e..edabfc40 100644
--- a/ansible/veos_vtb
Expand All @@ -87,29 +112,35 @@ index 3e7b3c4e..edabfc40 100644
hosts:
```

- Create dummy `password.txt` under `/data/sonic-mgmt/ansible`

Please note: Here "password.txt" is the Ansible Vault password file name/path. Ansible allows user to use Ansible Vault to encrypt password files. By default, this shell script requires a password file. If you are not using Ansible Vault, you can create a file with a dummy password, e.g., `abc` and pass the filename to the command line. The file name and location is created and maintained by user.
2. Create a dummy `password.txt` file under `/data/sonic-mgmt/ansible`
- **Note**: Here, `password.txt` is the Ansible Vault password file. Ansible allows users to use Ansible Vault to encrypt password files.

By default, the testbed scripts require a password file. If you are not using Ansible Vault, you can create a file with a dummy password (e.g. `abc`) and pass the filename to the command line. The file name and location is created and maintained by the user.

- On the host, run `sudo visudo` and add the following line at the end:
3. **On the host,** run `sudo visudo` and add the following line at the end:

```
foo ALL=(ALL) NOPASSWD:ALL
```

- Test you can login into the host `ssh foo@172.17.0.1` without any password prompt
from the `sonic-mgmt` container. Then, test you can sudo without password prompt in the host.
4. Verify that you can login into the host (e.g. `ssh foo@172.17.0.1`) from the `sonic-mgmt` container without any password prompt.

5. Verify that you can use `sudo` without a password prompt inside the host (e.g. `sudo bash`).

## Setup Arista VMs in the server
## Setup Arista VMs on the server
**(Skip this step if you are using cEOS - the containers will be automatically setup in a later step.)**

(skip this step if you use cEOS image)
Now we need to spin up some VMs on the host to act as neighboring devices to our virtual SONiC switch.

1. Start the VMs:
```
$ ./testbed-cli.sh -m veos_vtb -n 4 start-vms server_1 password.txt
```
- Please note: Here "password.txt" is the Ansible Vault password file name/path. Ansible allows user to use Ansible Vault to encrypt password files. By default, this shell script requires a password file. If you are not using Ansible Vault, just create a file with a dummy password and pass the filename to the command line. The file name and location is created and maintained by user.

Check that all VMs are up and running, and the passwd is `123456`
- **Reminder:** By default, this shell script requires a password file. If you are not using Ansible Vault, just create a file with a dummy password and pass the filename to the command line.


2. Check that all VMs are up and running. **Note:** The passwd is `123456`.
```
$ ansible -m ping -i veos_vtb server_1 -u root -k
VM0102 | SUCCESS => {
Expand All @@ -136,6 +167,7 @@ VM0100 | SUCCESS => {


## Deploy T0 topology
Now we're finally ready to deploy the topology for our testbed! Run the following command, depending on what type of EOS image you are using for your setup:

### vEOS
```
Expand All @@ -149,7 +181,7 @@ $ cd /data/sonic-mgmt/ansible
$ ./testbed-cli.sh -t vtestbed.csv -m veos_vtb -k ceos add-topo vms-kvm-t0 password.txt
```

Verify topology setup successfully.
Verify that the cEOS neighbors were created properly:

```
$ docker ps
Expand All @@ -166,13 +198,17 @@ c929c622232a sonicdev-microsoft.azurecr.io:443/docker-ptf:latest "/usr/
```

## Deploy minigraph on the DUT
Once the topology has been created, we need to give the DUT an initial configuration.

1. Deploy the `minigraph.xml` to the DUT and save the configuration:

```
$ ./testbed-cli.sh -t vtestbed.csv -m veos_vtb deploy-mg vms-kvm-t0 lab password.txt
```

You should be login into the sonic kvm using IP: 10.250.0.101 using admin:password.
You should see BGP sessions up in sonic.
2. Verify that you can login to the SONiC KVM using Mgmt IP = 10.250.0.101 and admin:password.

3. You should see BGP sessions up in SONiC:

```
admin@vlab-01:~$ show ip bgp sum
Expand All @@ -182,18 +218,25 @@ Peers 8, using 36 KiB of memory
Peer groups 2, using 112 bytes of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.57 4 64600 3208 12 0 0 0 00:00:22 6400
10.0.0.59 4 64600 3208 593 0 0 0 00:00:22 6400
10.0.0.61 4 64600 3205 950 0 0 0 00:00:21 6400
10.0.0.63 4 64600 3204 950 0 0 0 00:00:21 6400
10.0.0.57 4 64600 3208 12 0 0 0 00:00:22 1
10.0.0.59 4 64600 3208 593 0 0 0 00:00:22 1
10.0.0.61 4 64600 3205 950 0 0 0 00:00:21 1
10.0.0.63 4 64600 3204 950 0 0 0 00:00:21 1
```

## Run a Pytest
Now that the testbed has been fully setup and configured, let's run a simple test to make sure everything is functioning as expected.

1. Switch over to the `tests` directory:

```
cd sonic-mgmt/tests
```

2. Run the following command to execute the `bgp_fact` test (including the pre/post setup steps):

```
./run_tests.sh -n vms-kvm-t0 -d vlab01 -c bgp/test_bgp_fact.py -f vtestbed.csv -i veos_vtb
```





You should see three sets of tests run and pass. You're now set up and ready to use the KVM testbed!
38 changes: 28 additions & 10 deletions ansible/setup-management-network.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,53 @@
#!/bin/bash

if [[ $(id -u) -ne 0 ]]; then
echo "Root privilege required"
exit
fi

echo "STEP 1: Checking for bridge-utils package..."
echo "Refreshing apt package lists..."
apt-get update
echo

echo "STEP 1: Checking for j2cli package..."
if ! command -v j2; then
echo "j2cli not found, installing j2cli"
apt-get install -y j2cli
fi
echo

echo "STEP 2: Checking for bridge-utils package..."
if ! command -v brctl; then
echo "brctl not found, installing bridge-utils"
apt-get install -y bridge-utils
fi
echo

echo "STEP 2: Checking for net-tools package..."
echo "STEP 3: Checking for net-tools package..."
if ! command -v ifconfig; then
echo "ifconfig not found, install net-tools"
apt-get install -y net-tools
fi
echo

echo "STEP 3: Checking if bridge br1 already exists..."
if brctl show br1; then
echo "br1 already exists, deleting first"
ifconfig br1 down
brctl delbr br1
echo "STEP 4: Checking for ethtool package..."
if ! command -v ethtool; then
echo "ethtool not found, install ethtool"
apt-get install -y ethtool
fi
echo

echo "STEP 5: Checking if bridge br1 already exists..."
if ! brctl show br1; then
Copy link
Copy Markdown
Contributor

@prsunny prsunny Mar 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it returns the output as below, then this step fails. May be we should just add br1 even if it exists..

STEP 5: Checking if bridge br1 already exists...
bridge name	bridge id		STP enabled	interfaces
br1		can't get info No such device

echo "br1 not found, creating bridge network"
brctl addbr br1
brctl show br1
fi
echo

echo "STEP 4: Creating management bridge br1..."
brctl addbr br1
echo "STEP 6: Configuring br1 interface..."
echo "Assigning 10.250.0.1/24 to br1"
ifconfig br1 10.250.0.1/24
echo "Bringing up br1"
ifconfig br1 up
echo

Expand Down