[Kubernetes] Add ability to Setup HA Kubernetes Master for Virtual Switch Testbed

ansible/doc/README.testbed.k8s.Setup.md

@@ -54,13 +54,14 @@ Our setup meets Kubernetes Minimum Requirements to setup a High Available cluste
 - sudo privileges on all machines
 - SSH access from one device to all nodes in the system
 
-## How to Setup High Availability Kubernetes Master
+## How to Setup High Availability Kubernetes Master for Physical DUT
 
+#### To create a HA Kubernetes master for Virtual DUT: 
 1. Prepare Testbed Server and build and run `docker-sonic-mgmt` container as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.Setup.md) 
 2. Allocate 4 available IPs reachable from SONiC DUT.
-3. Update [`ansible/k8s-ubuntu`](../k8s-ubuntu) to include your 4 newly allocated IP addresses for the HA Kubernetes master and IP address of testbed server.
+3. Update [`ansible/k8s_ubuntu`](../k8s_ubuntu) to include your 4 newly allocated IP addresses for the HA Kubernetes master and IP address of testbed server.
 
-  - We will walk through an example of setting up HA Kubernetes master set 1 on server 19 (STR-ACS-SERV-19). The following snippets are the relevant portions from [`ansible/k8s-ubuntu`](../k8s-ubuntu).
+  - We will walk through an example of setting up HA Kubernetes master set 1 on server 19 (STR-ACS-SERV-19). The following snippets are the relevant portions from [`ansible/k8s_ubuntu`](../k8s_ubuntu).
 
    ```
    k8s_vm_host19:
@@ -92,7 +93,7 @@ Our setup meets Kubernetes Minimum Requirements to setup a High Available cluste
 
   - Replace each `ansible_host` value with an IP address allocated in step 2. 
 
-  - Take note of the group name `k8s_vms1_19`. At the bottom of [`ansible/k8s-ubuntu`](../k8s-ubuntu), make sure that `k8s_server_19` has its `host_var_file` and two `children` properly set: 
+  - Take note of the group name `k8s_vms1_19`. At the top of [`ansible/k8s_ubuntu`](../k8s_ubuntu), make sure that `k8s_server_19` has its `host_var_file` and two `children` properly set: 
 
 ```
 k8s_server_19:
@@ -110,23 +111,47 @@ k8s_server_19:
 6. Update the testbed server credentials in [`ansible/group_vars/k8s_vm_host/creds.yml`](../group_vars/k8s_vm_host/creds.yml).   
 7. If using Azure Storage to source Ubuntu 18.04 KVM image, set `k8s_vmimage_saskey` in [`ansible/vars/azure_storage.yml`](../vars/azure_storage.yml). 
    - To source image from public URL: download from  [here](https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img). Then, convert img to qcow2 by running `qemu-img convert -f qcow2 bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.qcow2`. Store qcow2 image at the path `/home/azure/ubuntu-vm/images/bionic-server-cloudimg-amd64.qcow2` on your testbed server. 
-8. From `docker-sonic-mgmt` container, `cd` into `sonic-mgmt/ansible` directory and run `./testbed-cli.sh -m k8s-ubuntu [additional OPTIONS] create-master <k8s-server-name> ~/.password`
-   - `k8s-server-name` corresponds to the group name used to describe the testbed server in the [`ansible/k8s-ubuntu`](../k8s-ubuntu) inventory file, of the form `k8s_server_{unit}`. 
+8. From `docker-sonic-mgmt` container, `cd` into `sonic-mgmt/ansible` directory and run `./testbed-cli.sh -m k8s_ubuntu [additional OPTIONS] create-master <k8s-server-name> ~/.password`
+   - `k8s-server-name` corresponds to the group name used to describe the testbed server in the [`ansible/k8s_ubuntu`](../k8s_ubuntu) inventory file, of the form `k8s_server_{unit}`. 
    - Please note: `~/.password` is the ansible vault password file name/path. Ansible allows users to use ansible-vault to encrypt password files. By default, this shell script requires a password file. If you are not using ansible-vault, just create an empty file and pass the file name to the command line. The file name and location are created and maintained by the user.
    - For HA Kubernetes master set 1 running on server 19 shown above, the proper command would be: 
-`./testbed-cli.sh -m k8s-ubuntu create-master k8s_server_19 ~/.password` 
+`./testbed-cli.sh -m k8s_ubuntu create-master k8s_server_19 ~/.password` 
   - OPTIONAL: We offer the functionality to run multiple master sets on one server. 
     - Each master set is one HA Kubernetes master composed of 4 Linux KVMs. 
-    - Should an additional HA master set be necessary on an occupied server, add the option `-s <msetnumber>`, where `msetnumber` would be 2 if this is the 2nd master set running on `<k8s-server-name>`. Make sure that [`ansible/k8s-ubuntu`](../k8s-ubuntu) is updated accordingly. Specifically, make sure that the IPS are set in the correct group `k8s_vms{msetnumber}_{servernumber}` and the `children` are properly updated for `k8s_server_{servernumber}` at the bottom of the inventory file. `msetnumber` is 1 by default. 
+    - Should an additional HA master set be necessary on an occupied server, add the option `-s <msetnumber>`, where `msetnumber` would be 2 if this is the 2nd master set running on `<k8s-server-name>`. Make sure that [`ansible/k8s_ubuntu`](../k8s_ubuntu) is updated accordingly. Specifically, make sure that the IPS are set in the correct group `k8s_vms{msetnumber}_{servernumber}` and the `children` are properly updated for `k8s_server_{servernumber}` at the bottom of the inventory file. `msetnumber` is 1 by default. 
 
 
 9. Join Kubernetes-enabled SONiC DUT to cluster (kube_join function to be written).
 
 
-#### To remove a HA Kubernetes master:
-- Run `./testbed-cli.sh -m k8s-ubuntu [additional OPTIONS] destroy-master <k8s-server-name> ~/.password`
+#### To remove a HA Kubernetes master for Physical DUT:
+- Run `./testbed-cli.sh -m k8s_ubuntu [additional OPTIONS] destroy-master <k8s-server-name> ~/.password`
 - For HA Kubernetes master set 1 running on server 19 shown above, the proper command would be: 
-`./testbed-cli.sh -m k8s-ubuntu destroy-master k8s_server_19 ~/.password` 
+`./testbed-cli.sh -m k8s_ubuntu destroy-master k8s_server_19 ~/.password` 
+
+## How to Setup High Availability Kubernetes Master for Virtual DUT
+
+#### To create a HA Kubernetes master for Virtual DUT: 
+1. Setup internal management network:
+```
+$ git clone https://github.com/Azure/sonic-mgmt
+$ cd sonic-mgmt/ansible
+$ sudo ./setup-management-network.sh
+$ sudo ./setup-br1-nat.sh <name of server's external facing port>
+```
+2. Setup virtual switch testbed as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.VsSetup.md).
+3. In [`ansible/k8s_ubuntu_vtb`](../k8s_ubuntu_vtb), replace `use_own_value` with the username for the server, corresponds to the username used while setting up [`ansible/veos_vtb`](../veos_vtb) for the virtual switch testbed.
+4. Specify DNS server IP to be used by Ubuntu KVMs in [`ansible/host_vars/STR-ACS-VSERV-21.yml`](../host_vars/STR-ACS-VSERV-21.yml); this should be the same DNS server IP as used by the testbed server.
+5. From inside the `sonic-mgmt` docker set up in step 2, run:
+```
+$ cd /data/sonic-mgmt/ansible
+$ ./testbed-cli.sh -m k8s_ubuntu_vtb create-master k8s_server_21 password.txt  
+```
+#### To remove a HA Kubernetes master for Virtual DUT: 
+```
+$ cd /data/sonic-mgmt/ansible
+$ ./testbed-cli.sh k8s_ubuntu_vtb destroy-master k8s_server_21 password.txt  
+```
 
 ## Testing Scope
 

ansible/host_vars/STR-ACS-VSERV-21.yml

@@ -0,0 +1,5 @@
+mgmt_bridge_k8s: br1
+mgmt_prefixlen_k8s: 24
+mgmt_gw_k8s: 10.250.0.1
+dns_server_ip: use_own_value
+

ansible/k8s-ubuntu → ansible/k8s_ubuntu

@@ -14,7 +14,20 @@ all:
       children:
         k8s_server_19:
         k8s_server_20:
-
+
+k8s_server_19:
+  vars:
+    host_var_file: host_vars/STR-ACS-SERV-19.yml
+  children:
+    k8s_vm_host19:
+    k8s_vms1_19:
+
+k8s_server_20:
+  vars:
+    host_var_file: host_vars/STR-ACS-SERV-20.yml
+  children:
+    k8s_vm_host20:
+    k8s_vms1_20:
 
 k8s_vm_host19:
   hosts:
@@ -98,18 +111,4 @@ k8s_vms2_20:
       ansible_host: 10.251.0.118
       haproxy: true 
 
-# The groups below are helper to limit running playbooks to specific server(s) only
-k8s_server_19:
-  vars:
-    host_var_file: host_vars/STR-ACS-SERV-19.yml
-  children:
-    k8s_vm_host19:
-    k8s_vms1_19:
-
-k8s_server_20:
-  vars:
-    host_var_file: host_vars/STR-ACS-SERV-20.yml
-  children:
-    k8s_vm_host20:
-    k8s_vms1_20:
 

ansible/k8s_ubuntu_vtb

@@ -0,0 +1,66 @@
+all:
+  children:
+    k8s_vm_host:
+      children:
+        k8s_vm_host21:
+    k8s_ubu:
+      children:
+        k8s_vms1_21:
+        k8s_vms2_21:
+    k8s_servers:
+      children:
+        k8s_server_21:
+
+k8s_server_21:
+  vars:
+    host_var_file: host_vars/STR-ACS-VSERV-21.yml
+  children:
+    k8s_vm_host21:
+    k8s_vms1_21:
+
+k8s_vm_host21:
+  hosts:
+    STR-ACS-VSERV-21:
+      ansible_host: 172.17.0.1
+      ansible_user: use_own_value
+
+k8s_vms1_21:
+  hosts:
+    kvm21-1m1:
+      ansible_host: 10.250.0.106
+      master: true
+      master_leader: true
+    kvm21-1m2:
+      ansible_host: 10.250.0.107
+      master: true
+      master_member: true
+    kvm21-1m3:
+      ansible_host: 10.250.0.108
+      master_member: true
+      master: true
+    kvm21-1ha:
+      ansible_host: 10.250.0.109
+      haproxy: true 
+
+k8s_vms2_21:
+  hosts:
+    kvm21-2m1:
+      ansible_host: 10.250.0.110
+      master: true
+      master_leader: true
+    kvm21-2m2:
+      ansible_host: 10.250.0.111
+      master: true
+      master_member: true
+    kvm21-2m3:
+      ansible_host: 10.250.0.112
+      master_member: true
+      master: true
+    kvm21-2ha:
+      ansible_host: 10.250.0.113
+      haproxy: true 
+
+
+
+
+

ansible/roles/k8s_master/tasks/main.yml

@@ -51,6 +51,7 @@
   template:
     src: 'docker-proxy.j2'
     dest: '/etc/systemd/system/docker.service.d/http-proxy.conf'
+  when: proxy_env['https_proxy'] is defined
 
 - name: Enable docker systemd service
   service:

ansible/roles/k8s_master/vars/main.yml

@@ -1,4 +1,3 @@
----
 KUBERNETES_VERSION: 1.19.2-00
 app_user: "ubuntu"
 k8s_port: 6443

ansible/roles/vm_set/tasks/start_k8s.yml

@@ -6,6 +6,7 @@
 
 - name: Ensure {{ k8s_root_path }} exists
   file: path={{ k8s_root_path }} state=directory
+  become: yes
 
 - name: Find current server group
   set_fact: current_server={{ group_names | extract_by_prefix('k8s_server_') }}
@@ -15,6 +16,7 @@
 
 - name: Create directory for Ubuntu VM image
   file: path={{ k8s_root_path }}/images state=directory mode=0755
+  become: yes
 
 - name: Check local cloud image 
   stat: 

ansible/roles/vm_set/tasks/start_k8s_vm.yml

@@ -1,5 +1,6 @@
 - name: Create directory for Ubuntu VM disks
-  file: path={{ k8s_root_path }}/{{ vm_name }} state=directory mode=0755
+  file: path={{ k8s_root_path }}/{{ vm_name }} state=directory mode=0777
+  become: yes
 
 - name: Check destination file existence
   stat: path={{ disk_image }} get_checksum="no"
@@ -41,22 +42,23 @@
         --noautoconsole
         --name {{ vm_name }}
         --memory 4098
-        --vcpus  2
+        --vcpus 2
         --disk {{ disk_image }},device=disk,bus=virtio 
         --disk {{ k8s_root_path }}/{{ vm_name }}/{{ vm_name }}_cloud-config.img,device=cdrom 
         --os-type linux 
         --os-variant ubuntu18.04
         --virt-type kvm 
         --graphics none 
-        --network bridge={{ mgmt_bridge_k8s }},model=virtio 
+        --network bridge={{ mgmt_bridge_k8s }},model=virtio
         --import
   become: yes
   when: vm_name not in vm_list_defined.list_vms
 
 - name: "{{ vm_name }} start with host startup"
   virt: 
     name: "{{ vm_name }}"
-    autostart: true 
+    autostart: true
+  become: yes 
 
 - name: "Wait for K8s Ubuntu VM to be accessible via SSH"
   wait_for:

ansible/roles/vm_set/tasks/stop_k8s_vm.yml

@@ -14,4 +14,5 @@
 
 - name: Remove VM disk images and config files for {{ vm_name }}
   file: path={{ vm_folder }} state=absent
+  become: yes
 

ansible/roles/vm_set/templates/network-config-v2.yml.j2

@@ -4,4 +4,7 @@ ethernets:
     dhcp4: false
     addresses: [{{ mgmt_ip_address }}/{{ mgmt_prefixlen_k8s }}, ]
     gateway4: {{ mgmt_gw }}
-
+{% if 'vtb' in inventory_file %}
+    nameservers:
+      addresses: [{{ dns_server_ip }}]
+{% endif %}

ansible/setup-br1-nat.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+if [[ $(id -u) -ne 0 ]]; then
+    echo "Root privelege required"
+    exit
+fi
+
+if [ $# -eq 0 ]; then
+    echo "Please specify server's external facing port name"
+    exit
+fi
+
+echo "Setting up NAT..."
+iptables -t nat -A POSTROUTING -s 10.250.0.0/24 -o $1 -j MASQUERADE
+iptables -A FORWARD -i $1 -j ACCEPT
+iptables -A FORWARD -i br1 -j ACCEPT
+echo "Updated IP tables rules: "
+iptables -v -L

ansible/testbed-cli.sh

@@ -27,7 +27,7 @@ function usage
   echo "    <vault-password-file> : Path to file containing Ansible Vault password"
   echo "    <topo-name>           : Name of the target topology"
   echo "    <inventory>           : Name of the Ansible inventory containing the DUT"
-  echo "    <k8s-server-name>     : Server identifier in form k8s_server_{id}, corresponds to k8s-ubuntu inventory group name"
+  echo "    <k8s-server-name>     : Server identifier in form k8s_server_{id}, corresponds to k8s_ubuntu inventory group name"
   echo
   echo "To start all VMs on a server: $0 start-vms 'server-name' ~/.password"
   echo "To restart a subset of VMs:"
@@ -55,8 +55,8 @@ function usage
   echo "        -e enable_data_plane_acl=true"
   echo "        -e enable_data_plane_acl=false"
   echo "        by default, data acl is enabled"
-  echo "To create Kubernetes master on a server: $0 -m k8s-ubuntu create-master 'k8s-server-name'  ~/.password"
-  echo "To destroy Kubernetes master on a server: $0 -m k8s-ubuntu destroy-master 'k8s-server-name' ~/.password"
+  echo "To create Kubernetes master on a server: $0 -m k8s_ubuntu create-master 'k8s-server-name'  ~/.password"
+  echo "To destroy Kubernetes master on a server: $0 -m k8s_ubuntu destroy-master 'k8s-server-name' ~/.password"
   echo
   echo "You should define your topology in testbed CSV file"
   echo