Skip to content

[Kubernetes] Add ability to setup high availability Kubernetes master on testbed server#2240

Merged
lguohan merged 15 commits intosonic-net:masterfrom
isabelmsft:k8s-master-integrate
Sep 26, 2020
Merged

[Kubernetes] Add ability to setup high availability Kubernetes master on testbed server#2240
lguohan merged 15 commits intosonic-net:masterfrom
isabelmsft:k8s-master-integrate

Conversation

@isabelmsft
Copy link
Contributor

@isabelmsft isabelmsft commented Sep 18, 2020 •
edited
Loading

Description of PR

Summary:
Add Ansible jobs to setup and tear down high availability Kubernetes master on testbed server. Manageable through testbed-cli.sh script

Type of change

[ ] Bug fix
[ x ] Testbed and Framework(new/improvement)
[ ] Test case(new/improvement)

Approach

What is the motivation for this PR?

Automate setup and teardown of high availability Kubernetes master on testbed server

How did you do it?

  • Define Kubernetes specific inventory file
  • Modify preexisting Ansible role vm_set to start and stop Ubuntu KVMs for HA Kubernetes master, set up bridged networking configuration
  • Create new Ansible roles: k8s_haproxy and k8s_master to configure HA Kubernetes master KVMs
  • Configure HAProxy for round robin load balancing between 3 HA Kubernetes master backends
  • Initialize Kubernetes master nodes via HAProxy load balancer endpoint
  • Modify testbed-cli.sh script to add extra options to manage Kubernetes master

How did you verify/test it?

Run script to set up master on testbed server
sudo virsh list --all on testbed server shows all 4 newly generated KVMs
KVM directory (contains 2 disk images and user, networking config files) is properly generated for each KVM
kubectl get nodes on any of the three master node KVMs shows 3 master nodes status as "Ready"
systemctl status haproxy on HAProxy KVM shows service active and running, listening on port 80

Run script to tear down master on testbed server
sudo virsh list --all no longer shows the 4 KVMs that were destroyed
KVM directory (contains 2 disk images and user, networking config files) is properly removed for each KVM

From token generated join, SONiC DUT to HA Kubernetes cluster
kubectl get nodes shows SONiC DUT and each master node

Documentation

ansible/doc/README-k8s.testbed.design.md

lguohan
lguohan reviewed Sep 18, 2020
View reviewed changes
ansible/roles/vm_set/tasks/main.yml
dest: "{{ root_path }}"
mode: 0755

- name: Retrieve a list of the defined VMs
Copy link
Contributor

@lguohan lguohan Sep 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why move up?

Copy link
Contributor Author

@isabelmsft isabelmsft Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other code in the block is required for veos VM, but not for k8s Ubuntu VM. In k8s VM set up, I reference these registers/lists

@isabelmsft isabelmsft marked this pull request as ready for review September 21, 2020 23:23
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
qiluo-msft
qiluo-msft reviewed Sep 23, 2020
View reviewed changes
Copy link
Contributor

@qiluo-msft qiluo-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comments

qiluo-msft
qiluo-msft reviewed Sep 24, 2020
View reviewed changes
@isabelmsft
Copy link
Contributor Author

isabelmsft commented Sep 24, 2020

retest vsimage please

qiluo-msft
qiluo-msft approved these changes Sep 24, 2020
View reviewed changes
Copy link
Contributor

@qiluo-msft qiluo-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please also check with other reviewers.

@isabelmsft
Copy link
Contributor Author

isabelmsft commented Sep 24, 2020

retest vsimage please

1 similar comment
@isabelmsft
Copy link
Contributor Author

isabelmsft commented Sep 25, 2020

retest vsimage please

@lguohan lguohan merged commit d190bb2 into sonic-net:master Sep 26, 2020
kazinator-arista pushed a commit to kazinator-arista/sonic-mgmt that referenced this pull request Mar 4, 2026
@yxieca
[202205][linkmgrd][utilities][swss][sairedis][platform-daemon][platfo…
b1456ee 
…rm-common] advance submodule head (sonic-net#11578)

linkmgrd:
* e0fe1d4 2022-07-27 | TSA enhancement (sonic-net#98) (HEAD -> 202205) [Jing Zhang]

utilities:
* 7d7e15e 2022-07-18 | [vnet_route_check] Align DB data parse logic with format used by swsscommon API (sonic-net#2268) (HEAD -> 202205) [Volodymyr Samotiy]
* b3d5d18 2022-07-20 | [MultiAsic] sudo reboot command doesn't gracefully stop Asic syncd# on multiasic platform (sonic-net#2258) [Marty Y. Lok]
* 504ebe6 2022-07-08 | Add 'traffic_shift_away' option to config load_minigraph (sonic-net#2240) [tjchadaga]
* 4079e4a 2022-06-20 | Gives cisco-8000 more flexibility to easily add subcommnads under show platform (sonic-net#2213) [Nathan Cohen]
* 46443c6 2022-07-27 | Update db_migrator to support `PORT_QOS_MAP|global` (sonic-net#2205) [bingwang-ms]
* d7fbdd6 2022-05-26 | fix for non-coherent cmis modules (sonic-net#2163) [qinchuanares]
* 79b4439 2022-06-24 | [sfpshow/sfputil] Enhance sfpshow and sfputil to behavior correctly on RJ45 ports (sonic-net#2111) [Kebo Liu]

swss:
* 275f311 2022-07-26 | [DualToR] Handle race condition between tunnel_decap and mux orchestrator (sonic-net#2397) (HEAD -> 202205) [Devesh Pathak]
* 47586e8 2022-07-22 | [EVPN]Fix missing Vlan member update notification in P2MP scenario (sonic-net#2388) [Sudharsan Dhamal Gopalarathnam]
* 7d5c73f 2022-07-19 | [macsecmgr]: Fix cleanup macsec objs if container stop (sonic-net#2376) [Ze Gan]
* c03996f 2022-07-17 | [orchagent]: Enhance initSaiPhyApi (sonic-net#2367) [andywongarista]
* 57890d7 2022-07-27 | Fix for remote system interface not getting created (sonic-net#2364) [skeesara-nokia]
* 1a93ec9 2022-07-13 | Orchagent changes for synchronizing npu/phy device Tx in the data path before enabling transceiver<CMIS compliant> Tx. (sonic-net#2277) [jaganbal-a]

sairedis:
* a4903be 2022-07-20 | Update PN with XPN support (sonic-net#1081) (HEAD -> 202205, github/202205) [Ze Gan]
* 2cb5671 2022-07-27 | Add SAI_OBJECT_TYPE_TUNNEL object to break-before-make list (sonic-net#1075) [Vaibhav Hemant Dixit]

platform-daemon:
* 901c6a1 2022-06-28 | [CMIS]Improved 400G link bring up sequence (sonic-net#254) (HEAD -> 202205) [Prince George]

platform-common:
* f223b3f 2022-07-09 | Support get_port_or_cage_type (sonic-net#288) (HEAD -> 202205) [Stephen Sun]

Signed-off-by: Ying Xie <ying.xie@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lguohan lguohan lguohan left review comments

@qiluo-msft qiluo-msft qiluo-msft approved these changes

@renukamanavalan renukamanavalan Awaiting requested review from renukamanavalan

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@isabelmsft @lguohan @qiluo-msft