NO-ISSUE: OTA-1605 Create first e2e test OCP-42543 #1249
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
JianLi-RH
force-pushed
the
implement_client_oc
branch
4 times, most recently
from
a6a8ada to
37caff4
Compare
JianLi-RH
changed the title
JianLi-RH
force-pushed
the
implement_client_oc
branch
4 times, most recently
from
a79646a to
d944000
Compare
|
Please note that
|
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
d944000 to
067cd4b
Compare
JianLi-RH
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@JianLi-RH: This pull request explicitly references no jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Tested on my local machine: Run the case: |
JianLi-RH
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
067cd4b to
bac7793
Compare
|
/cc |
|
/cc |
.openshift-tests-extension/openshift_payload_cluster-version-operator.json
Show resolved
Hide resolved
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
bac7793 to
51facb3
Compare
|
After modification, the case is still valid: |
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
51facb3 to
8e9da36
Compare
.openshift-tests-extension/openshift_payload_cluster-version-operator.json
Outdated
Show resolved
Hide resolved
JianLi-RH
force-pushed
the
implement_client_oc
branch
2 times, most recently
from
72354d1 to
3a8631b
Compare
|
@hongkailiu @DavidHurta I just removed the case, right now the PR only finished one thing: MustGetKubeClient(). |
JianLi-RH
changed the title
|
/retest |
|
/hold I want to investigate more about client-go. |
@JianLi-RH, in case the investigation is not done by the end of the day, you want simply more time to explore or are simply busy, I would like to ask you whether it would be fine for me to rename the sanity test in a separate PR. That specific change is blocking my openshift/origin#30316 PR. I am happy to rename the test personally in a separate PR to unblock my PR while we are working on this one. |
|
@DavidHurta of course, go ahead, please. I am not sure when I can finish the investigation. |
JianLi-RH
force-pushed
the
implement_client_oc
branch
6 times, most recently
from
d2300d9 to
f98e03c
Compare
|
/unhold |
openshift-ci
bot
removed
the
do-not-merge/hold
Clientset instance can be used to connect to cluster, manage resources. With clientset instance we can evaluate the posibility of moving other tests to OTE. The reference to OTE framework: https://docs.google.com/document/d/1cFZj9QdzW8hbHc3H0Nce-2xrJMtpDJrwAse9H7hLiWk/edit?tab=t.0#heading=h.8cf3f4eii1q8
JianLi-RH
force-pushed
the
implement_client_oc
branch
4 times, most recently
from
878cbc8 to
1102f9c
Compare
|
@hongkailiu @DavidHurta Just added back the case, please help review it. Here is the output: $ ginkgo ./test/...
Running Suite: CVO Suite - /home/jianl/1_code/cluster-version-operator/test/cvo
===============================================================================
Random Seed: 1763353592
Will run 2 of 2 specs
•Test Authentication...
namespace: default
- Test services...
- Test RoleBinding...
- Test CronJob...
success
namespace: kube-node-lease
- Test services...
- Test RoleBinding...
- Test CronJob...
success
namespace: kube-public
- Test services...
- Test RoleBinding...
- Test CronJob...
.....................
namespace: openshift-vsphere-infra
- Test services...
- Test RoleBinding...
- Test CronJob...
success
•
Ran 2 of 2 Specs in 82.840 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
Ginkgo ran 1 suite in 1m24.112444203s
Test Suite PassedIf there is resource installed unexpected, it will fail as below: $ ginkgo ./test/...
Running Suite: CVO Suite - /home/jianl/1_code/cluster-version-operator/test/cvo
===============================================================================
Random Seed: 1763353871
Will run 2 of 2 specs
•Test Authentication...
------------------------------
• [FAILED] [1.522 seconds]
[Jira:"Cluster Version Operator"] cluster-version-operator-tests [It] should not install resources annotated with release.openshift.io/delete=true [cvo, Conformance, High, 42543]
/home/jianl/1_code/cluster-version-operator/test/cvo/cvo.go:20
[FAILED] Unexpectedly installed authentication cluster which has release.openshift.io/delete annotation
Expected
<bool>: true
not to be true
In [It] at: /home/jianl/1_code/cluster-version-operator/test/cvo/cvo.go:30 @ 11/17/25 12:31:14.175
------------------------------
Summarizing 1 Failure:
[FAIL] [Jira:"Cluster Version Operator"] cluster-version-operator-tests [It] should not install resources annotated with release.openshift.io/delete=true [cvo, Conformance, High, 42543]
/home/jianl/1_code/cluster-version-operator/test/cvo/cvo.go:30
Ran 2 of 2 Specs in 1.522 seconds
FAIL! -- 1 Passed | 1 Failed | 0 Pending | 0 Skipped
--- FAIL: TestGinkgo (1.52s)
FAIL
Ginkgo ran 1 suite in 2.725034617s
Test Suite Failed |
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
1102f9c to
cc98b16
Compare
JianLi-RH
changed the title
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
cc98b16 to
c168996
Compare
…ient-go openshift/client-go gives us more efficient functions to access cluster
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
c168996 to
e35f937
Compare
After adding new OTE test cases, we need to update test metadata
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
4dbdb2c to
946876f
Compare
WalkthroughThis PR adds a test case verifying that cluster version operator does not install resources annotated with Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20–30 minutes
✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting
📒 Files selected for processing (3)
.openshift-tests-extension/openshift_payload_cluster-version-operator.json(1 hunks)test/cvo/cvo.go(1 hunks)test/utilities/connection.go(1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**
⚙️ CodeRabbit configuration file
-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.
Files:
test/utilities/connection.gotest/cvo/cvo.go
🧬 Code graph analysis (1)
test/cvo/cvo.go (1)
test/utilities/connection.go (2)
MustGetV1Client(65-71)MustGetKubeClient(56-62)
🔇 Additional comments (3)
.openshift-tests-extension/openshift_payload_cluster-version-operator.json (1)
13-25: New test metadata entry looks consistent and well-formedThe added metadata object matches the existing structure (name, labels, resources, source, lifecycle, environmentSelector) and aligns with the new CVO test and label conventions (42543, Conformance, High, cvo). No issues from a structure/maintainability perspective.
test/utilities/connection.go (1)
1-71: Client helper utilities are correct and suitable for test usageKUBECONFIG handling, config construction, clientset creation, and error wrapping all look correct. The
MustGet*helpers are appropriate for tests where a missing/invalid config should hard-fail immediately. No blocking issues here.test/cvo/cvo.go (1)
17-21: Sanity test Describe block is fineThe lightweight
[Jira:"Cluster Version Operator"] cluster-version-operator-testssanity spec is simple and effective as an integration signal. No issues.
| var _ = g.Describe("[Jira:Cluster Version Operator] The cluster version operator", g.Ordered, g.Label("cvo"), func() { | ||
| defer g.GinkgoRecover() | ||
| var client *v1.ConfigV1Client | ||
| var kubeclient *kubernetes.Clientset | ||
|
|
||
| g.BeforeAll(func() { | ||
| client = utilities.MustGetV1Client() | ||
| kubeclient = utilities.MustGetKubeClient() | ||
| }) | ||
|
|
||
| g.It(`should not install resources annotated with release.openshift.io/delete=true`, g.Label("Conformance", "High", "42543"), func() { | ||
| annotation := "release.openshift.io/delete" | ||
|
|
||
| auths, err := client.Authentications().List(context.TODO(), metav1.ListOptions{}) | ||
| o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing authentications") | ||
|
|
||
| g.By(fmt.Sprintf("checking if authentication with %s annotation exists", annotation)) | ||
| for _, auth := range auths.Items { | ||
| if _, ok := auth.Annotations[annotation]; ok { | ||
| o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed authentication %s which has '%s' annotation", auth.Name, annotation)) | ||
| } | ||
| } | ||
|
|
||
| namespaces, err := kubeclient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{}) | ||
| o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing namespaces") | ||
|
|
||
| g.By(fmt.Sprintf("checking if special resources with %s annotation exist in all namespaces", annotation)) | ||
| for _, ns := range namespaces.Items { | ||
| namespace := ns.Name | ||
| fmt.Printf("namespace: %s\n", namespace) | ||
|
|
||
| fmt.Println(" - Test services...") | ||
| services, err := kubeclient.CoreV1().Services(namespace).List(context.TODO(), metav1.ListOptions{}) | ||
| o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing services") | ||
| for _, service := range services.Items { | ||
| if _, ok := service.Annotations[annotation]; ok { | ||
| o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed service %s which has '%s' annotation", service.Name, annotation)) | ||
| } | ||
| } | ||
|
|
||
| fmt.Println(" - Test RoleBinding...") | ||
| rolebindings, err := kubeclient.RbacV1().RoleBindings(namespace).List(context.TODO(), metav1.ListOptions{}) | ||
| o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing rolebindings") | ||
| for _, rb := range rolebindings.Items { | ||
| if _, ok := rb.Annotations[annotation]; ok { | ||
| o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed RoleBinding %s which has '%s' annotation", rb.Name, annotation)) | ||
| } | ||
| } | ||
|
|
||
| fmt.Println(" - Test CronJob...") | ||
| cronjobs, err := kubeclient.BatchV1().CronJobs(namespace).List(context.TODO(), metav1.ListOptions{}) | ||
| o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing cronjobs") | ||
| for _, cj := range cronjobs.Items { | ||
| if _, ok := cj.Annotations[annotation]; ok { | ||
| o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed CronJob %s which has %s annotation", cj.Name, annotation)) | ||
| } | ||
| } | ||
|
|
||
| fmt.Println("success") | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
Fix List error handling to fail on real API errors, not just NotFound
All the List calls currently do:
auths, err := client.Authentications().List(...)
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing ...")This has two problems:
-
False positives on failures
- Any non-
NotFounderror (e.g., auth, RBAC, networking, API server issues) makeskerrors.IsNotFound(err)returnfalse, so the expectation passes. - The test then proceeds, typically iterating over an empty/zero-valued list, and can still pass even though the underlying List failed.
- That means the test can report success while not actually verifying the cluster state.
- Any non-
-
Misleading messages
- The description says “The NotFound error should occur when listing …” while the matcher asserts
BeFalse(), which is contradictory.
- The description says “The NotFound error should occur when listing …” while the matcher asserts
You want the test to fail on any error from those List calls. A more robust pattern is to assert on err directly, for each List:
- auths, err := client.Authentications().List(context.TODO(), metav1.ListOptions{})
- o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing authentications")
+ auths, err := client.Authentications().List(context.TODO(), metav1.ListOptions{})
+ o.Expect(err).NotTo(o.HaveOccurred(), "failed to list authentications")
- namespaces, err := kubeclient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
- o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing namespaces")
+ namespaces, err := kubeclient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
+ o.Expect(err).NotTo(o.HaveOccurred(), "failed to list namespaces")
- services, err := kubeclient.CoreV1().Services(namespace).List(context.TODO(), metav1.ListOptions{})
- o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing services")
+ services, err := kubeclient.CoreV1().Services(namespace).List(context.TODO(), metav1.ListOptions{})
+ o.Expect(err).NotTo(o.HaveOccurred(), "failed to list services")
- rolebindings, err := kubeclient.RbacV1().RoleBindings(namespace).List(context.TODO(), metav1.ListOptions{})
- o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing rolebindings")
+ rolebindings, err := kubeclient.RbacV1().RoleBindings(namespace).List(context.TODO(), metav1.ListOptions{})
+ o.Expect(err).NotTo(o.HaveOccurred(), "failed to list rolebindings")
- cronjobs, err := kubeclient.BatchV1().CronJobs(namespace).List(context.TODO(), metav1.ListOptions{})
- o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing cronjobs")
+ cronjobs, err := kubeclient.BatchV1().CronJobs(namespace).List(context.TODO(), metav1.ListOptions{})
+ o.Expect(err).NotTo(o.HaveOccurred(), "failed to list cronjobs")This way, any failure to talk to the API (including NotFound on the resource type) will correctly fail the spec instead of silently passing.
If you want to double‑check semantics, please confirm in the Kubernetes/client-go docs (or local experiments) that List should normally return an empty list with err == nil when no items exist, and that kerrors.IsNotFound is intended for handling Get/Delete style operations rather than normal successful Lists.
🌐 Web query:
How does the Kubernetes client-go List API signal errors vs. an empty result, and when is k8s.io/apimachinery/pkg/api/errors.IsNotFound typically used?
💡 Result:
Short answer: List returns an empty list with err == nil (the returned List object's Items slice is just empty). IsNotFound is not used for a successful List — it's used when an operation asking for a specific named resource (e.g., Get or Delete by name) returns HTTP 404; k8s.io/apimachinery/pkg/api/errors.IsNotFound(err) detects that 404 StatusError. Use IsNotFound when you expect a single resource might not exist and you want to handle "resource missing" specially.
References: k8s error helpers (IsNotFound). [1] Client List/Get semantics (List returns list object; Reader/List returns error only on failure). [2] REST practice: collection endpoints return 200 + empty array when no items. [3]
Sources:
[1] k8s.io/apimachinery/pkg/api/errors (IsNotFound description).
[2] sigs.k8s.io/controller-runtime/pkg/client (Client.List / Reader semantics).
[3] RESTful API design discussion (return 200 with empty array for empty collections).
Replace error assertions with proper nil checks for all List calls
The web search confirms your analysis: Kubernetes List operations return an empty list with err == nil on success. The IsNotFound check is not intended for List operations—it detects HTTP 404 on named-resource operations (Get/Delete), not on collection endpoints.
This means the current code has a critical flaw:
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing ...")—will pass whenever IsNotFound(err) == false, which includes:
- Normal success with no items ✓
- Network/auth/RBAC failures ✗ (silent pass when should fail)
Your proposed fix is correct. Replace all five List error checks in test/cvo/cvo.go (lines 26–27, 41–42, 51–52, 60–61, 68–69) with:
-o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing ...")
+o.Expect(err).NotTo(o.HaveOccurred(), "failed to list ...")This ensures any API failure is caught, not silently suppressed.
🤖 Prompt for AI Agents
In test/cvo/cvo.go around lines 23 to 82, the List() error checks use
kerrors.IsNotFound(err) which is inappropriate for collection endpoints and can
silently ignore real API errors; replace each usage at lines ~26–27, ~41–42,
~51–52, ~60–61, and ~68–69 with a direct nil-check assertion (i.e., assert err
is nil) so any List failure causes the test to fail; keep the existing
descriptive messages but change the assertion to ensure err == nil.
|
@JianLi-RH: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
5 participants
In this PR, I finished 2 things:
/cc @wking @DavidHurta @PratikMahajan @hongkailiu @jhou1 @dis016 @jiajliu