[Kubernetes] Update HA Master Setup README

ansible/doc/README.testbed.k8s.Setup.md

@@ -1,16 +1,16 @@
 # SONiC Kubernetes Design
 
-This document describes the design to test Kubernetes features in SONiC. 
+This document describes the testbed design for Kubernetes features in SONiC and provides instructions to set up the high availability Kubernetes master.
 
 ## Background
 
 Each SONiC DUT is a worker node managed by a High Availability Kubernetes master. The High Availability Kubernetes master is composed of three master node machines and one load balancer machine.
 
-By connecting each SONiC DUT to HA Kubernetes master, containers running in SONiC can be managed by the Kubernetes master. SONiC containers managed by the Kubernetes master are termed to be running in "Kubernetes mode" as opposed to the original "Local mode." 
+By connecting each SONiC DUT to HA Kubernetes master, containers running in SONiC can be managed by the Kubernetes master. SONiC containers managed by the Kubernetes master are termed to be running in "kube mode" as opposed to the original "local mode." 
 
-In Kubernetes mode, SONiC container properties are based on specifications defined in the associated Kubernetes manifest. A Kubernetes manifest is a file in the Kubernetes master that defines the Kubernetes object and container configurations. In our case, we use Kubernetes Daemonset objects. The Kubernetes Daemonset object ensures that each worker node is running exactly one container of the image specified in the Daemonset manifest file.  
+In local mode, the SONiC feature container runs based on the image tagged `feature_name:latest` in the local image store; this feature runs independent of the Kubernetes master. In kube mode, SONiC container properties are based on specifications defined in the associated Kubernetes manifest. A Kubernetes manifest is a file in the Kubernetes master that defines the Kubernetes object and container configurations, including a URL from which to source the feature image. In our case, we use Kubernetes Daemonset objects. The Kubernetes Daemonset object ensures that each worker node is running exactly one container of the image specified in the Daemonset manifest file. 
 
-For example, in order to run SNMP and Telemetry containers in Kubernetes mode, we must have two manifests that define two Kubernetes Daemonset objects- one for each container running in "Kubernetes mode." 
+For example, in order to run SNMP and Telemetry containers in kube mode, we must have two manifests that define two Kubernetes Daemonset objects- one for each container/feature running in kube mode.
 
 The following is a snippet of the Telemetry Daemonset manifest file that specifies the Kubernetes object type and container image:
 
@@ -39,15 +39,17 @@ spec:
 
 ## Topology Overview
 
-In order to connect each SONiC DUT to a High Availability Kubernetes master, we need to set up the following topology: 
+In order to connect each physical SONiC DUT to a High Availability Kubernetes master, we need to set up the following topology: 
 ![alt text](https://github.com/isabelmsft/k8s-ha-master-starlab/blob/master/k8s-testbed-linux.png)
 - Each high availability master setup requires 4 new Linux KVMs running on a Testbed Server via bridged networking.
     - 3 Linux KVMs to serve as 3-node high availability Kubernetes master
-    - 1 Linux KVM to serve as HAProxy Load Balancer node    
+    - 1 Linux KVM to serve as HAProxy Load Balancer node
 - Each KVM has one management interface assigned an IP address reachable from SONiC DUT.
 - HAProxy Load Balancer proxies requests to 3 backend Kubernetes master nodes. 
 
-Our setup meets Kubernetes Minimum Requirements to setup a High Available cluster. The Minimum Requirements are as follows:
+In the case of a virtual SONiC DUT, the SONiC KVM and 4 new Linux KVMs for the Kubernetes master are all running on the Testbed Server (or host VM). Each KVM is connected to an internal management network, Linux bridge br1. Internal management network setup for the virtual DUT is described in [How to Setup High Availability Kubernetes Master for Virtual DUT](#how-to-setup-high-availability-kubernetes-master-for-virtual-dut) below.
+
+Our setup meets Kubernetes Minimum Requirements to setup a High Availability cluster. The Minimum Requirements are as follows:
 - 2 GB or more of RAM per machine
 - 2 CPUs or more per machine
 - Full network connectivity between all machines in the cluster (public or private network)
@@ -56,7 +58,7 @@ Our setup meets Kubernetes Minimum Requirements to setup a High Available cluste
 
 ## How to Setup High Availability Kubernetes Master for Physical DUT
 
-#### To create a HA Kubernetes master for Virtual DUT: 
+#### To create a HA Kubernetes master for Physical DUT: 
 1. Prepare Testbed Server and build and run `docker-sonic-mgmt` container as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.Setup.md) 
 2. Allocate 4 available IPs reachable from SONiC DUT.
 3. Update [`ansible/k8s_ubuntu`](../k8s_ubuntu) to include your 4 newly allocated IP addresses for the HA Kubernetes master and IP address of testbed server.
@@ -118,10 +120,10 @@ k8s_server_19:
 `./testbed-cli.sh -m k8s_ubuntu create-master k8s_server_19 ~/.password` 
   - OPTIONAL: We offer the functionality to run multiple master sets on one server. 
     - Each master set is one HA Kubernetes master composed of 4 Linux KVMs. 
-    - Should an additional HA master set be necessary on an occupied server, add the option `-s <msetnumber>`, where `msetnumber` would be 2 if this is the 2nd master set running on `<k8s-server-name>`. Make sure that [`ansible/k8s_ubuntu`](../k8s_ubuntu) is updated accordingly. Specifically, make sure that the IPS are set in the correct group `k8s_vms{msetnumber}_{servernumber}` and the `children` are properly updated for `k8s_server_{servernumber}` at the bottom of the inventory file. `msetnumber` is 1 by default. 
-
-
-9. Join Kubernetes-enabled SONiC DUT to cluster (kube_join function to be written).
+    - Should an additional HA master set be necessary on an occupied server, add the option `-s <msetnumber>`, where `msetnumber` would be 2 if this is the 2nd master set running on `<k8s-server-name>`. Make sure that [`ansible/k8s-ubuntu`](../k8s-ubuntu) is updated accordingly. Specifically, make sure that the IPS are set in the correct group `k8s_vms{msetnumber}_{servernumber}` and the `children` are properly updated for `k8s_server_{servernumber}` at the bottom of the inventory file. `msetnumber` is 1 by default. 
+9. Join Kubernetes-enabled SONiC DUT to master by configuring VIP and enabling the Kubernetes server/master connection. Kubernetes server is enabled by default
+    - `sudo config kube server ip <VIP>`
+    - `sudo config kube server disable off` (default configuration)
 
 
 #### To remove a HA Kubernetes master for Physical DUT:
@@ -139,9 +141,9 @@ $ cd sonic-mgmt/ansible
 $ sudo ./setup-management-network.sh
 $ sudo ./setup-br1-nat.sh <name of server's external facing port>
 ```
-2. Setup virtual switch testbed as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.VsSetup.md).
+2. Setup virtual switch testbed as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.VsSetup.md). **Note:** if the host machine is a VM, nested virtualization must be enabled. 
 3. In [`ansible/k8s_ubuntu_vtb`](../k8s_ubuntu_vtb), replace `use_own_value` with the username for the server, corresponds to the username used while setting up [`ansible/veos_vtb`](../veos_vtb) for the virtual switch testbed.
-4. Specify DNS server IP to be used by Ubuntu KVMs in [`ansible/host_vars/STR-ACS-VSERV-21.yml`](../host_vars/STR-ACS-VSERV-21.yml); this should be the same DNS server IP as used by the testbed server.
+4. Specify DNS server IP to be used by Ubuntu KVMs in [`ansible/host_vars/STR-ACS-VSERV-21.yml`](../host_vars/STR-ACS-VSERV-21.yml); this should be the same DNS server IP as used by the host machine.
 5. From inside the `sonic-mgmt` docker set up in step 2, run:
 ```
 $ cd /data/sonic-mgmt/ansible
@@ -150,33 +152,21 @@ $ ./testbed-cli.sh -m k8s_ubuntu_vtb create-master k8s_server_21 password.txt
 #### To remove a HA Kubernetes master for Virtual DUT: 
 ```
 $ cd /data/sonic-mgmt/ansible
-$ ./testbed-cli.sh k8s_ubuntu_vtb destroy-master k8s_server_21 password.txt  
+$ ./testbed-cli.sh -m k8s_ubuntu_vtb destroy-master k8s_server_21 password.txt  
 ```
 
 ## Testing Scope
 
 This setup allows us to test the following: 
-- Successful deployment of SONiC containers via manifests defined in master
-- Expected container behavior after the container is intentionally or unintentionally stopped
-- Switching between Local and Kubernetes management mode for a given container
-  - Addition and removal of SONiC DUT labels
-- Changing image version in middle of Daemonset deployment
+- Joining and removing/resetting of SONiC DUT from Kubernetes master
+- Upgrades of kube mode feature images via Kubernetes manifests
+- SONiC feature transition between kube mode and local mode
+- Proper management of kube mode features before and after SONiC reboots
 
 During each of the following states:
-- When all master servers are up and running
-- When one master server is down
-- When two master servers are down
-- When all master servers are down
-
-Down: shut off, disconnected, or in the middle of reboot
-
+- When master VIP is reachable
+- When master VIP is unrechable
+- When Kubernetes API server is available
+- When Kubernetes API server unavailable
 
 In this setup, we do not consider load balancer performance. For Kubernetes feature testing purposes, HAProxy is configured to perform vanilla round-robin load balancing on available master servers.
-
-
-## How to Create Tests
-Each manifest is a yaml file
-
-CLI to make changes to manifest files
-
-pytests to apply manifest changes and check status