OpenShift over VPN: await_condition fails with `peer closed connection without sending TLS close_notify`

[starpit]

Current and expected behavior

When i use oc get deploy -w I see no issues. But when i use kube-rs on the same machine, at the same time (i.e. i do this concurrently) with this rust code:

await_condition(deployments, name, is_deployment_completed()).await?;

I get the following error:

Error: failed to probe for whether the condition is fulfilled yet: watch stream failed: Error reading events stream: ServiceError: error reading a body from connection

Caused by:
    0: watch stream failed: Error reading events stream: ServiceError: error reading a body from connection
    1: Error reading events stream: ServiceError: error reading a body from connection
    2: ServiceError: error reading a body from connection
    3: error reading a body from connection
    4: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

I have tried both ring and aws_lc_rs, both result in the above error.

Possible solution

No response

Additional context

This is using an OpenShift server over a "tunnel all" Cisco Secure Connect VPN.

Environment

Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.31.11
Warning: version difference between client (1.34) and server (1.31) exceeds the supported minor version skew of +/-1

This is running OpenShift.

Configuration and features

kube = { version = "3.0.0", default-features = false, features = ["client", "runtime", "rustls-tls", "ring"], optional = true }
k8s-openapi = { version = "0.27.0", features = ["latest", "schemars"], optional = true }

YAML

No response

Affected crates

No response

Would you like to work on fixing this bug?

None

[clux]

thanks for the report. althought it's hard to tell what's the issue could be from this alone.

maybe it's a way the openshift cli dealing with auth that's different from us (maybe there's a way to collect information from this somehow). like if oc injects things into the kubeconfig in a way that kubectl does not.. does it work with regular kubectl?

maybe it's some proxy configuration that it passes by default (there's a proxy_url field that can be set, but i doubt a vpn would use that)

is there anything in the logs if you run your cargo project with e.g. RUST_LOG=trace cargo run ...?

what type of kubeconfig auth is this? e.g. what users keys used in .kube/config. is this an exec based auth?

[starpit]

Hey there, thanks for the help!

The user appears to be sha256 token based.

The tail of the log shows the following. Not knowing rustls, it's hard for me to say anything deep here. The linked manual page states that rustls purposefully considers an EOF without a close_notify as an error, to be handled by the client (in this case, kube-rs?). Perhaps kubectl and the openshift cli treat EOF without close_notify as a reason to retry rather than give up?

2026-01-22T21:49:50Z DEBUG rustls::common_state] Sending warning alert CloseNotify
[2026-01-22T21:49:50Z TRACE tracing::span::active] -> HTTP;
[2026-01-22T21:49:50Z TRACE tracing::span::active] <- HTTP;
[2026-01-22T21:49:50Z DEBUG kube_runtime::watcher] watcher error: ReadEvents(Custom { kind: Other, error: Service(hyper::Error(Body, Custom { kind: UnexpectedEof, error: "peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof" })) })
[2026-01-22T21:49:50Z TRACE tracing::span] -- HTTP;
[2026-01-22T21:49:50Z TRACE tower::buffer::worker] worker polling for next message
Error: failed to probe for whether the condition is fulfilled yet: watch stream failed: Error reading events stream: ServiceError: error reading a body from connection

Caused by:
    0: watch stream failed: Error reading events stream: ServiceError: error reading a body from connection
    1: Error reading events stream: ServiceError: error reading a body from connection
    2: ServiceError: error reading a body from connection
    3: error reading a body from connection
    4: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

eof.log

[doxxx93]

await_condition() → watch_object() → watcher()

• When a TLS connection drops without close_notify, watcher() emits Error::WatchFailed but internally keeps state to reconnect.
• However, watch_object() passes this error through, and await_condition() fails immediately on try_next() — before the watcher has a chance to reconnect.

is the cause I think so..?

[nightkr]

How long are you waiting for before the connection is dropped? Is the API server behind some kind of reverse proxy? Are you seeing any BOOKMARK events in the TRACE logs while waiting?

I remember that we've seen at least AKS kill connections that are inactive for too long. Normally, bookmarks are supposed to work around that, but maybe we could do some kind of keepalive poking from our side too.

Perhaps kubectl and the openshift cli treat EOF without close_notify as a reason to retry rather than give up?

For what it's worth, you can do this today: await_condition is a pretty thin wrapper around watch_object, which is fine to keep polling after it has returned an error.

So this:

let obj = await_condition(api, name, cond).await?;

Could become something like this:

let obj = while {
    match watch_object(api, name) {
        Some(Ok(obj)) => if cond.matches_object(obj.as_ref()) {
            break obj;
        },
        Some(Err(err)) => {
            tracing::warn!(error = &err as &dyn std::error::Error, "error awaiting condition, retrying");
            // Probably do some kind of backoff here before continuing
        }
        None => return Err(AwaitConditionCancelled),
    }
}

is the cause I think so..?

Yeah, that's about right.

[starpit]

Thanks for the advice. Yes, the plan for now on our side was to do the retry. I'll let you know how it goes.

Regarding timing, the error occurs after only about 20-30 seconds.