Add k8s join and disjoin test cases by lixiaoyuner · Pull Request #16141 · sonic-net/sonic-mgmt

lixiaoyuner · 2024-12-18T15:31:13Z

Description of PR

Summary: This PR is adding a test case to test if the sonic is able to join and disjoin k8s cluster and test whether k8s daemonset pods are able to run on sonic.
Fixes # (issue): It's a new test case to test k8s feature

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

Test the sonic device join k8s cluster and disjoin k8s cluster.

How did you do it?

Setup a single master node cluster on the server of testbed, make sonic device join the cluster and disjoin the cluster to check if it works, deploy a daemonset to check if the pods are able to run on sonic.

How did you verify/test it?

Run it in the KVM testbed and physical testbed, it works

Any platform specific information?

No, should be good for all platform

Supported testbed topology if it's a new test case?

Any

Documentation

In the PR md file.

mssonicbld · 2024-12-18T15:31:17Z

/azp run

azure-pipelines · 2024-12-18T15:31:30Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-19T08:16:09Z

/azp run

azure-pipelines · 2024-12-19T08:16:22Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-20T02:43:14Z

/azp run

azure-pipelines · 2024-12-20T02:43:25Z

Azure Pipelines successfully started running 1 pipeline(s).

tests/kubesonic/test_k8s_join_disjion.py

mssonicbld · 2024-12-25T09:11:18Z

/azp run

azure-pipelines · 2024-12-25T09:11:24Z

Pull request contains merge conflicts.

mssonicbld · 2024-12-25T09:21:59Z

/azp run

azure-pipelines · 2024-12-25T09:22:09Z

Azure Pipelines successfully started running 1 pipeline(s).

tests/kubesonic/kubesonic_test.md

mssonicbld · 2024-12-25T11:27:52Z

/azp run

azure-pipelines · 2024-12-25T11:28:04Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-25T12:39:49Z

/azp run

azure-pipelines · 2024-12-25T12:40:00Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-26T08:29:10Z

/azp run

azure-pipelines · 2024-12-26T08:29:22Z

Azure Pipelines successfully started running 1 pipeline(s).

make1980 · 2024-12-26T17:28:11Z

tests/common/plugins/conditional_mark/tests_mark_conditions.yaml

+  skip:
+    reason: "kubesonic feature is not supported in slim image"
+    conditions:
+      - "hwsku in ['Arista-7050-QX-32S', 'Arista-7050-Q16S64', 'Arista-7060CX-32S-C32', 'Arista-7060CX-32S-C32-T1', 'Arista-7060CX-32S-Q32', 'Arista-7060CX-32S-D48C8', 'Arista-7050QX-32S-S4Q31', 'Arista-7050QX32S-Q32', 'Celestica-E1031-T48S4']"


Arista-7050-QX32/Arista-720DT-G48S4/Nexus-3132-GX-Q32 should be using slim image as well, right? is there any more generic way to figure out if the test is invoked against a slim images?

I got this sku list from the firmware file, the sku are configured slim image certainly. For these 3 you mentioned, Arista-720DT-G48S4 is not using slim image. Arista-7050-QX32/Nexus-3132-GX-Q32 run old sonic versions, and I don't see any nightly test for this sku.

tests/kubesonic/kubesonic_test.md

tests/kubesonic/test_k8s_join_disjoin.py

make1980 · 2024-12-26T18:15:40Z

tests/kubesonic/test_k8s_join_disjoin.py

+
+
+def restore_vmhost_param(vmhost):
+    logger.info("Start to restore vmhost param")


this is not guaranteed to be executed, right? the test can fail in the middle so that this function won't be executed.

It's in teardown function, should be executed, it's done by pytest.

If the test plan is cancelled, the kernel parameter can't be restored. Although it can't be restored, but there should no impact.

tests/kubesonic/test_k8s_join_disjoin.py

tests/kubesonic/kubesonic_test.md

tests/kubesonic/test_k8s_join_disjoin.py

isabelmsft · 2024-12-26T22:03:21Z

tests/kubesonic/test_k8s_join_disjoin.py

+
+def check_dut_k8s_version_supported(duthost):
+    logger.info("Check if the k8s version is supported")
+    k8s_version = duthost.shell("kubeadm version -o short", module_ignore_errors=True)["stdout"]


What if kubeadm package is not installed on SONiC image? I think the code still works, but the log would be misleading

Very helpful, this is so critical. I removed the module_ignore_errors=True, if no kubeadm installed, the test will fail directly.

isabelmsft · 2024-12-26T23:00:48Z

tests/kubesonic/test_k8s_join_disjoin.py

+
+
+@pytest.fixture()
+def setup_and_teardown(duthost, vmhost, creds):


conftest.py can be used to share fixture across multiple files. This could be helpful as we add more test cases to this directory. This can be done here or in a future PR

ref: https://docs.pytest.org/en/stable/reference/fixtures.html#conftest-py-sharing-fixtures-across-multiple-files
https://github.com/sonic-net/sonic-mgmt/blob/master/tests/generic_config_updater/conftest.py

Okay, got it. Very useful info, let's do it in the future if we have more test cases.

tests/kubesonic/test_k8s_join_disjoin.py

isabelmsft · 2024-12-27T00:07:11Z

tests/kubesonic/test_k8s_join_disjoin.py

+                logger.info(f"Minikube setup has started for {time_diff_seconds} seconds")
+                if time_diff_seconds > MINIKUBE_SETUP_MAX_SECOND:
+                    logger.info("Minikube setup timeout, need to re-setup")
+                    clean_up_and_setup_minikube(vmhost, creds)


IIUC, this while loop is for the file lock described in the readme. Here, if one testbed which first requested minikube master has taken more time than expected, then we restart the minikube master setup triggered by a second testbed. Is that right?

Depending on the reason for the delay, isn't it possible to still run into the minikube master setup conflict here if the first operation was not cancelled? I think we should fail for testbed1 if testbed1 requests and minikube takes longer than expected to start. Without this failure, we may also run into an infinite loop

Yes, you are right. The download step and setup step both have timeout(6 mins) now, it will fail if it's too long.

mssonicbld · 2024-12-27T07:00:00Z

/azp run

azure-pipelines · 2024-12-27T07:01:12Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-27T08:32:24Z

/azp run

azure-pipelines · 2024-12-27T08:32:35Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-27T09:00:31Z

/azp run

azure-pipelines · 2024-12-27T09:00:49Z

Azure Pipelines successfully started running 1 pipeline(s).

What is the motivation for this PR? Test the sonic device join k8s cluster and disjoin k8s cluster. How did you do it? Setup a single master node cluster on the server of testbed, make sonic device join the cluster and disjoin the cluster to check if it works, deploy a daemonset to check if the pods are able to run on sonic. How did you verify/test it? Run it in the KVM testbed and physical testbed, it works Any platform specific information? No, should be good for all platform Supported testbed topology if it's a new test case? Any Documentation In the PR md file.

yutongzhang-microsoft · 2025-01-03T05:54:20Z

@lixiaoyuner Can this new added test script support kvm testbed?

lixiaoyuner · 2025-01-03T08:44:29Z

@lixiaoyuner Can this new added test script support kvm testbed?

Yes, it supports

* Add k8s join and disjoin test cases (#16141) What is the motivation for this PR? Test the sonic device join k8s cluster and disjoin k8s cluster. How did you do it? Setup a single master node cluster on the server of testbed, make sonic device join the cluster and disjoin the cluster to check if it works, deploy a daemonset to check if the pods are able to run on sonic. How did you verify/test it? Run it in the KVM testbed and physical testbed, it works Any platform specific information? No, should be good for all platform Supported testbed topology if it's a new test case? Any Documentation In the PR md file. * Update the conditional mark for kubesonic test * Filter the version number less than and equal 30 for broadcom

mssonicbld · 2025-01-06T19:44:43Z

@lixiaoyuner PR conflicts with 202411 branch

[202411] Add k8s join and disjoin test cases

What is the motivation for this PR? Test the sonic device join k8s cluster and disjoin k8s cluster. How did you do it? Setup a single master node cluster on the server of testbed, make sonic device join the cluster and disjoin the cluster to check if it works, deploy a daemonset to check if the pods are able to run on sonic. How did you verify/test it? Run it in the KVM testbed and physical testbed, it works Any platform specific information? No, should be good for all platform Supported testbed topology if it's a new test case? Any Documentation In the PR md file.

* Add k8s join and disjoin test cases (#16141) What is the motivation for this PR? Test the sonic device join k8s cluster and disjoin k8s cluster. How did you do it? Setup a single master node cluster on the server of testbed, make sonic device join the cluster and disjoin the cluster to check if it works, deploy a daemonset to check if the pods are able to run on sonic. How did you verify/test it? Run it in the KVM testbed and physical testbed, it works Any platform specific information? No, should be good for all platform Supported testbed topology if it's a new test case? Any Documentation In the PR md file. * Update the condition mark for 202405 * Add the conditions logical operator * Remove the asic filter

yutongzhang-microsoft · 2025-01-10T07:13:48Z

Hi, @lixiaoyuner , can you help me confirm if this script can run on kvm testbed?

What is the motivation for this PR? Test the sonic device join k8s cluster and disjoin k8s cluster. How did you do it? Setup a single master node cluster on the server of testbed, make sonic device join the cluster and disjoin the cluster to check if it works, deploy a daemonset to check if the pods are able to run on sonic. How did you verify/test it? Run it in the KVM testbed and physical testbed, it works Any platform specific information? No, should be good for all platform Supported testbed topology if it's a new test case? Any Documentation In the PR md file.

Add k8s join and disjoin test cases

429483a

Fix the code check comment

aca9431

lixiaoyuner marked this pull request as ready for review December 20, 2024 02:20

Fix the code check reported issue

ff1d0e5

losha-ms approved these changes Dec 20, 2024

View reviewed changes

tests/kubesonic/test_k8s_join_disjion.py Outdated Show resolved Hide resolved

Improve the test case and add doc

91f4124

Merge branch 'master' into dev/yunli1/add-k8s-join-disjoin-testcase

a4962d6

losha-ms reviewed Dec 25, 2024

View reviewed changes

Fix pre-commit checker comments

9afe27b

Refine the doc

9fb97f8

Fix typo

17d6a43

make1980 reviewed Dec 26, 2024

View reviewed changes

isabelmsft reviewed Dec 26, 2024

View reviewed changes

tests/kubesonic/kubesonic_test.md Outdated Show resolved Hide resolved

isabelmsft reviewed Dec 26, 2024

View reviewed changes

isabelmsft reviewed Dec 27, 2024

View reviewed changes

Fix comments

aff1ef9

Fix the timeout parameter

320c14c

Ignore error for ctrmgrd status

d0b2c0d

wangxin approved these changes Dec 27, 2024

View reviewed changes

isabelmsft self-requested a review December 27, 2024 22:22

isabelmsft approved these changes Dec 27, 2024

View reviewed changes

wangxin merged commit a1f63f9 into sonic-net:master Dec 30, 2024

lixiaoyuner added Request for 202311 branch Request for 202405 branch Request for 202411 branch labels Dec 30, 2024

kperumalbfn added the Approved for 202411 branch label Jan 6, 2025

mssonicbld added the Cherry Pick Conflict_202411 label Jan 6, 2025

kperumalbfn pushed a commit that referenced this pull request Jan 6, 2025

[202411] Add k8s join and disjoin test cases (#16141) (#16260)

fa43330

[202411] Add k8s join and disjoin test cases



		def restore_vmhost_param(vmhost):
		logger.info("Start to restore vmhost param")



		@pytest.fixture()
		def setup_and_teardown(duthost, vmhost, creds):

Conversation

lixiaoyuner commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Dec 18, 2024

Uh oh!

azure-pipelines bot commented Dec 18, 2024

Uh oh!

mssonicbld commented Dec 19, 2024

Uh oh!

azure-pipelines bot commented Dec 19, 2024

Uh oh!

mssonicbld commented Dec 20, 2024

Uh oh!

azure-pipelines bot commented Dec 20, 2024

Uh oh!

Uh oh!

mssonicbld commented Dec 25, 2024

Uh oh!

azure-pipelines bot commented Dec 25, 2024

Uh oh!

mssonicbld commented Dec 25, 2024

Uh oh!

azure-pipelines bot commented Dec 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Dec 25, 2024

Uh oh!

azure-pipelines bot commented Dec 25, 2024

Uh oh!

mssonicbld commented Dec 25, 2024

Uh oh!

azure-pipelines bot commented Dec 25, 2024

Uh oh!

mssonicbld commented Dec 26, 2024

Uh oh!

azure-pipelines bot commented Dec 26, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lixiaoyuner commented Dec 18, 2024 •

edited

Loading

yutongzhang-microsoft commented Jan 3, 2025 •

edited

Loading