diff --git a/ansible/doc/README.testbed.k8s.Setup.md b/ansible/doc/README.testbed.k8s.Setup.md index 33dbd6564b6..be6390ad4d3 100644 --- a/ansible/doc/README.testbed.k8s.Setup.md +++ b/ansible/doc/README.testbed.k8s.Setup.md @@ -54,13 +54,14 @@ Our setup meets Kubernetes Minimum Requirements to setup a High Available cluste - sudo privileges on all machines - SSH access from one device to all nodes in the system -## How to Setup High Availability Kubernetes Master +## How to Setup High Availability Kubernetes Master for Physical DUT +#### To create a HA Kubernetes master for Virtual DUT: 1. Prepare Testbed Server and build and run `docker-sonic-mgmt` container as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.Setup.md) 2. Allocate 4 available IPs reachable from SONiC DUT. -3. Update [`ansible/k8s-ubuntu`](../k8s-ubuntu) to include your 4 newly allocated IP addresses for the HA Kubernetes master and IP address of testbed server. +3. Update [`ansible/k8s_ubuntu`](../k8s_ubuntu) to include your 4 newly allocated IP addresses for the HA Kubernetes master and IP address of testbed server. - - We will walk through an example of setting up HA Kubernetes master set 1 on server 19 (STR-ACS-SERV-19). The following snippets are the relevant portions from [`ansible/k8s-ubuntu`](../k8s-ubuntu). + - We will walk through an example of setting up HA Kubernetes master set 1 on server 19 (STR-ACS-SERV-19). The following snippets are the relevant portions from [`ansible/k8s_ubuntu`](../k8s_ubuntu). ``` k8s_vm_host19: @@ -92,7 +93,7 @@ Our setup meets Kubernetes Minimum Requirements to setup a High Available cluste - Replace each `ansible_host` value with an IP address allocated in step 2. - - Take note of the group name `k8s_vms1_19`. At the bottom of [`ansible/k8s-ubuntu`](../k8s-ubuntu), make sure that `k8s_server_19` has its `host_var_file` and two `children` properly set: + - Take note of the group name `k8s_vms1_19`. At the top of [`ansible/k8s_ubuntu`](../k8s_ubuntu), make sure that `k8s_server_19` has its `host_var_file` and two `children` properly set: ``` k8s_server_19: @@ -110,23 +111,47 @@ k8s_server_19: 6. Update the testbed server credentials in [`ansible/group_vars/k8s_vm_host/creds.yml`](../group_vars/k8s_vm_host/creds.yml). 7. If using Azure Storage to source Ubuntu 18.04 KVM image, set `k8s_vmimage_saskey` in [`ansible/vars/azure_storage.yml`](../vars/azure_storage.yml). - To source image from public URL: download from [here](https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img). Then, convert img to qcow2 by running `qemu-img convert -f qcow2 bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.qcow2`. Store qcow2 image at the path `/home/azure/ubuntu-vm/images/bionic-server-cloudimg-amd64.qcow2` on your testbed server. -8. From `docker-sonic-mgmt` container, `cd` into `sonic-mgmt/ansible` directory and run `./testbed-cli.sh -m k8s-ubuntu [additional OPTIONS] create-master ~/.password` - - `k8s-server-name` corresponds to the group name used to describe the testbed server in the [`ansible/k8s-ubuntu`](../k8s-ubuntu) inventory file, of the form `k8s_server_{unit}`. +8. From `docker-sonic-mgmt` container, `cd` into `sonic-mgmt/ansible` directory and run `./testbed-cli.sh -m k8s_ubuntu [additional OPTIONS] create-master ~/.password` + - `k8s-server-name` corresponds to the group name used to describe the testbed server in the [`ansible/k8s_ubuntu`](../k8s_ubuntu) inventory file, of the form `k8s_server_{unit}`. - Please note: `~/.password` is the ansible vault password file name/path. Ansible allows users to use ansible-vault to encrypt password files. By default, this shell script requires a password file. If you are not using ansible-vault, just create an empty file and pass the file name to the command line. The file name and location are created and maintained by the user. - For HA Kubernetes master set 1 running on server 19 shown above, the proper command would be: -`./testbed-cli.sh -m k8s-ubuntu create-master k8s_server_19 ~/.password` +`./testbed-cli.sh -m k8s_ubuntu create-master k8s_server_19 ~/.password` - OPTIONAL: We offer the functionality to run multiple master sets on one server. - Each master set is one HA Kubernetes master composed of 4 Linux KVMs. - - Should an additional HA master set be necessary on an occupied server, add the option `-s `, where `msetnumber` would be 2 if this is the 2nd master set running on ``. Make sure that [`ansible/k8s-ubuntu`](../k8s-ubuntu) is updated accordingly. Specifically, make sure that the IPS are set in the correct group `k8s_vms{msetnumber}_{servernumber}` and the `children` are properly updated for `k8s_server_{servernumber}` at the bottom of the inventory file. `msetnumber` is 1 by default. + - Should an additional HA master set be necessary on an occupied server, add the option `-s `, where `msetnumber` would be 2 if this is the 2nd master set running on ``. Make sure that [`ansible/k8s_ubuntu`](../k8s_ubuntu) is updated accordingly. Specifically, make sure that the IPS are set in the correct group `k8s_vms{msetnumber}_{servernumber}` and the `children` are properly updated for `k8s_server_{servernumber}` at the bottom of the inventory file. `msetnumber` is 1 by default. 9. Join Kubernetes-enabled SONiC DUT to cluster (kube_join function to be written). -#### To remove a HA Kubernetes master: -- Run `./testbed-cli.sh -m k8s-ubuntu [additional OPTIONS] destroy-master ~/.password` +#### To remove a HA Kubernetes master for Physical DUT: +- Run `./testbed-cli.sh -m k8s_ubuntu [additional OPTIONS] destroy-master ~/.password` - For HA Kubernetes master set 1 running on server 19 shown above, the proper command would be: -`./testbed-cli.sh -m k8s-ubuntu destroy-master k8s_server_19 ~/.password` +`./testbed-cli.sh -m k8s_ubuntu destroy-master k8s_server_19 ~/.password` + +## How to Setup High Availability Kubernetes Master for Virtual DUT + +#### To create a HA Kubernetes master for Virtual DUT: +1. Setup internal management network: +``` +$ git clone https://github.com/Azure/sonic-mgmt +$ cd sonic-mgmt/ansible +$ sudo ./setup-management-network.sh +$ sudo ./setup-br1-nat.sh +``` +2. Setup virtual switch testbed as described [here](https://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.VsSetup.md). +3. In [`ansible/k8s_ubuntu_vtb`](../k8s_ubuntu_vtb), replace `use_own_value` with the username for the server, corresponds to the username used while setting up [`ansible/veos_vtb`](../veos_vtb) for the virtual switch testbed. +4. Specify DNS server IP to be used by Ubuntu KVMs in [`ansible/host_vars/STR-ACS-VSERV-21.yml`](../host_vars/STR-ACS-VSERV-21.yml); this should be the same DNS server IP as used by the testbed server. +5. From inside the `sonic-mgmt` docker set up in step 2, run: +``` +$ cd /data/sonic-mgmt/ansible +$ ./testbed-cli.sh -m k8s_ubuntu_vtb create-master k8s_server_21 password.txt +``` +#### To remove a HA Kubernetes master for Virtual DUT: +``` +$ cd /data/sonic-mgmt/ansible +$ ./testbed-cli.sh k8s_ubuntu_vtb destroy-master k8s_server_21 password.txt +``` ## Testing Scope diff --git a/ansible/host_vars/STR-ACS-VSERV-21.yml b/ansible/host_vars/STR-ACS-VSERV-21.yml new file mode 100644 index 00000000000..13a190ea7d5 --- /dev/null +++ b/ansible/host_vars/STR-ACS-VSERV-21.yml @@ -0,0 +1,5 @@ +mgmt_bridge_k8s: br1 +mgmt_prefixlen_k8s: 24 +mgmt_gw_k8s: 10.250.0.1 +dns_server_ip: use_own_value + diff --git a/ansible/k8s-ubuntu b/ansible/k8s_ubuntu similarity index 96% rename from ansible/k8s-ubuntu rename to ansible/k8s_ubuntu index afdbb48aaba..d221b27d234 100644 --- a/ansible/k8s-ubuntu +++ b/ansible/k8s_ubuntu @@ -14,7 +14,20 @@ all: children: k8s_server_19: k8s_server_20: - + +k8s_server_19: + vars: + host_var_file: host_vars/STR-ACS-SERV-19.yml + children: + k8s_vm_host19: + k8s_vms1_19: + +k8s_server_20: + vars: + host_var_file: host_vars/STR-ACS-SERV-20.yml + children: + k8s_vm_host20: + k8s_vms1_20: k8s_vm_host19: hosts: @@ -98,18 +111,4 @@ k8s_vms2_20: ansible_host: 10.251.0.118 haproxy: true -# The groups below are helper to limit running playbooks to specific server(s) only -k8s_server_19: - vars: - host_var_file: host_vars/STR-ACS-SERV-19.yml - children: - k8s_vm_host19: - k8s_vms1_19: - -k8s_server_20: - vars: - host_var_file: host_vars/STR-ACS-SERV-20.yml - children: - k8s_vm_host20: - k8s_vms1_20: diff --git a/ansible/k8s_ubuntu_vtb b/ansible/k8s_ubuntu_vtb new file mode 100644 index 00000000000..6cd8bcf3eca --- /dev/null +++ b/ansible/k8s_ubuntu_vtb @@ -0,0 +1,66 @@ +all: + children: + k8s_vm_host: + children: + k8s_vm_host21: + k8s_ubu: + children: + k8s_vms1_21: + k8s_vms2_21: + k8s_servers: + children: + k8s_server_21: + +k8s_server_21: + vars: + host_var_file: host_vars/STR-ACS-VSERV-21.yml + children: + k8s_vm_host21: + k8s_vms1_21: + +k8s_vm_host21: + hosts: + STR-ACS-VSERV-21: + ansible_host: 172.17.0.1 + ansible_user: use_own_value + +k8s_vms1_21: + hosts: + kvm21-1m1: + ansible_host: 10.250.0.106 + master: true + master_leader: true + kvm21-1m2: + ansible_host: 10.250.0.107 + master: true + master_member: true + kvm21-1m3: + ansible_host: 10.250.0.108 + master_member: true + master: true + kvm21-1ha: + ansible_host: 10.250.0.109 + haproxy: true + +k8s_vms2_21: + hosts: + kvm21-2m1: + ansible_host: 10.250.0.110 + master: true + master_leader: true + kvm21-2m2: + ansible_host: 10.250.0.111 + master: true + master_member: true + kvm21-2m3: + ansible_host: 10.250.0.112 + master_member: true + master: true + kvm21-2ha: + ansible_host: 10.250.0.113 + haproxy: true + + + + + diff --git a/ansible/roles/k8s_master/tasks/main.yml b/ansible/roles/k8s_master/tasks/main.yml index 5c880785839..dbf9684e905 100644 --- a/ansible/roles/k8s_master/tasks/main.yml +++ b/ansible/roles/k8s_master/tasks/main.yml @@ -51,6 +51,7 @@ template: src: 'docker-proxy.j2' dest: '/etc/systemd/system/docker.service.d/http-proxy.conf' + when: proxy_env['https_proxy'] is defined - name: Enable docker systemd service service: diff --git a/ansible/roles/k8s_master/vars/main.yml b/ansible/roles/k8s_master/vars/main.yml index b89b157a703..0a4d8415cfc 100644 --- a/ansible/roles/k8s_master/vars/main.yml +++ b/ansible/roles/k8s_master/vars/main.yml @@ -1,4 +1,3 @@ ---- KUBERNETES_VERSION: 1.19.2-00 app_user: "ubuntu" k8s_port: 6443 diff --git a/ansible/roles/vm_set/tasks/start_k8s.yml b/ansible/roles/vm_set/tasks/start_k8s.yml index fd87bd5b1b3..32164d7bd51 100644 --- a/ansible/roles/vm_set/tasks/start_k8s.yml +++ b/ansible/roles/vm_set/tasks/start_k8s.yml @@ -6,6 +6,7 @@ - name: Ensure {{ k8s_root_path }} exists file: path={{ k8s_root_path }} state=directory + become: yes - name: Find current server group set_fact: current_server={{ group_names | extract_by_prefix('k8s_server_') }} @@ -15,6 +16,7 @@ - name: Create directory for Ubuntu VM image file: path={{ k8s_root_path }}/images state=directory mode=0755 + become: yes - name: Check local cloud image stat: diff --git a/ansible/roles/vm_set/tasks/start_k8s_vm.yml b/ansible/roles/vm_set/tasks/start_k8s_vm.yml index 1594f1c5dd0..97c96010af1 100644 --- a/ansible/roles/vm_set/tasks/start_k8s_vm.yml +++ b/ansible/roles/vm_set/tasks/start_k8s_vm.yml @@ -1,5 +1,6 @@ - name: Create directory for Ubuntu VM disks - file: path={{ k8s_root_path }}/{{ vm_name }} state=directory mode=0755 + file: path={{ k8s_root_path }}/{{ vm_name }} state=directory mode=0777 + become: yes - name: Check destination file existence stat: path={{ disk_image }} get_checksum="no" @@ -41,14 +42,14 @@ --noautoconsole --name {{ vm_name }} --memory 4098 - --vcpus 2 + --vcpus 2 --disk {{ disk_image }},device=disk,bus=virtio --disk {{ k8s_root_path }}/{{ vm_name }}/{{ vm_name }}_cloud-config.img,device=cdrom --os-type linux --os-variant ubuntu18.04 --virt-type kvm --graphics none - --network bridge={{ mgmt_bridge_k8s }},model=virtio + --network bridge={{ mgmt_bridge_k8s }},model=virtio --import become: yes when: vm_name not in vm_list_defined.list_vms @@ -56,7 +57,8 @@ - name: "{{ vm_name }} start with host startup" virt: name: "{{ vm_name }}" - autostart: true + autostart: true + become: yes - name: "Wait for K8s Ubuntu VM to be accessible via SSH" wait_for: diff --git a/ansible/roles/vm_set/tasks/stop_k8s_vm.yml b/ansible/roles/vm_set/tasks/stop_k8s_vm.yml index 2fd4820f660..8d05aa4eded 100644 --- a/ansible/roles/vm_set/tasks/stop_k8s_vm.yml +++ b/ansible/roles/vm_set/tasks/stop_k8s_vm.yml @@ -14,4 +14,5 @@ - name: Remove VM disk images and config files for {{ vm_name }} file: path={{ vm_folder }} state=absent + become: yes diff --git a/ansible/roles/vm_set/templates/network-config-v2.yml.j2 b/ansible/roles/vm_set/templates/network-config-v2.yml.j2 index e0f3f6017b3..4c3d6c73f95 100644 --- a/ansible/roles/vm_set/templates/network-config-v2.yml.j2 +++ b/ansible/roles/vm_set/templates/network-config-v2.yml.j2 @@ -4,4 +4,7 @@ ethernets: dhcp4: false addresses: [{{ mgmt_ip_address }}/{{ mgmt_prefixlen_k8s }}, ] gateway4: {{ mgmt_gw }} - +{% if 'vtb' in inventory_file %} + nameservers: + addresses: [{{ dns_server_ip }}] +{% endif %} diff --git a/ansible/setup-br1-nat.sh b/ansible/setup-br1-nat.sh new file mode 100755 index 00000000000..9c52240934b --- /dev/null +++ b/ansible/setup-br1-nat.sh @@ -0,0 +1,18 @@ +#!/bin/bash + +if [[ $(id -u) -ne 0 ]]; then + echo "Root privelege required" + exit +fi + +if [ $# -eq 0 ]; then + echo "Please specify server's external facing port name" + exit +fi + +echo "Setting up NAT..." +iptables -t nat -A POSTROUTING -s 10.250.0.0/24 -o $1 -j MASQUERADE +iptables -A FORWARD -i $1 -j ACCEPT +iptables -A FORWARD -i br1 -j ACCEPT +echo "Updated IP tables rules: " +iptables -v -L diff --git a/ansible/testbed-cli.sh b/ansible/testbed-cli.sh index 2271b3d8cde..aaaa486ff14 100755 --- a/ansible/testbed-cli.sh +++ b/ansible/testbed-cli.sh @@ -27,7 +27,7 @@ function usage echo " : Path to file containing Ansible Vault password" echo " : Name of the target topology" echo " : Name of the Ansible inventory containing the DUT" - echo " : Server identifier in form k8s_server_{id}, corresponds to k8s-ubuntu inventory group name" + echo " : Server identifier in form k8s_server_{id}, corresponds to k8s_ubuntu inventory group name" echo echo "To start all VMs on a server: $0 start-vms 'server-name' ~/.password" echo "To restart a subset of VMs:" @@ -55,8 +55,8 @@ function usage echo " -e enable_data_plane_acl=true" echo " -e enable_data_plane_acl=false" echo " by default, data acl is enabled" - echo "To create Kubernetes master on a server: $0 -m k8s-ubuntu create-master 'k8s-server-name' ~/.password" - echo "To destroy Kubernetes master on a server: $0 -m k8s-ubuntu destroy-master 'k8s-server-name' ~/.password" + echo "To create Kubernetes master on a server: $0 -m k8s_ubuntu create-master 'k8s-server-name' ~/.password" + echo "To destroy Kubernetes master on a server: $0 -m k8s_ubuntu destroy-master 'k8s-server-name' ~/.password" echo echo "You should define your topology in testbed CSV file" echo