TLS handshake error from: EOF

[ritazh]

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

Getting the following intermittent errors in the gatekeeper-system logs:

http: TLS handshake error from 172.16.0.3:42672: EOF

kube-apiserver logs during the same time range do not have equivalent errors.
Everything is functioning. No impact on functionality.

NOTE:
There isn't any actual functional issues related to these error messages and the policies are working as expected. Lots of other webhook projects have reported the same issue, the error is coming from the kube-apiserver when it drops the connection prematurely and retries afterwards.

Please provide feedback in the following issues:

The EOF errors seems be related to a Go bug golang/go#50984 and appear on Kubernetes 1.23 and 1.24 and later. see kubernetes/kubernetes#109022

What did you expect to happen:
No TLS error in pod logs

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Gatekeeper version: v3.8.1 and v3.7.1
• Kubernetes version: (use kubectl version): 1.23.5

[ritazh]

The EOF errors seems be related to a Go bug golang/go#50984 and appear on Kubernetes 1.23 and 1.24 see kubernetes/kubernetes#109022

From the issue description, it does not seem like there are any actual functional issues related to these error messages (as the policies are working as expected). At the moment, there is nothing we can do to fix this, as the error is coming from Kubernetes core. We can continue to monitor this after the linked issue has been fixed and released as part of a future Kubernetes patch release.

[ritazh]

xref: #866 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[punnarpulusu]

This is not just related to on Kubernetes 1.23 and 1.24 this is happening on all kuberenetes ( AWS EKS ) version 1.21

[ritazh]

@punnarpulusu Can you share the exact error in the log and kubernetes and gatekeeper version?

[punnarpulusu]

@ritazh Here is the error log ... redacted some information for security purpose.

gatekeeper version is 3.8.1

  k logs -n gatekeeper deploy/gatekeeper-controller-manager -f
  Found 3 pods, using pod/gatekeeper-controller-manager-xxxxxxx-ldsc7
2022/09/08 01:14:32 http: TLS handshake error from x.x.x.x:49070: EOF
2022/09/08 01:46:37 http: TLS handshake error from x.x.x.x:35184: EOF
2022/09/08 02:47:46 http: TLS handshake error from x.x.x.x:39938: EOF
2022/09/08 06:47:20 http: TLS handshake error from x.x.x.x:38652: EOF
2022/09/08 12:37:59 http: TLS handshake error from x.x.x.x:49956: EOF
2022/09/08 13:16:45 http: TLS handshake error from x.x.x.x:56032: EOF
2022/09/08 13:41:48 http: TLS handshake error from x.x.x.x:56232: EOF
2022/09/08 16:38:13 http: TLS handshake error from x.x.x.x:60828: EOF
2022/09/08 19:02:34 http: TLS handshake error from x.x.x.x:36744: EOF

sorry about the delayed response.

[punnarpulusu]

@ritazh I am getting the same error on gatekeeper 3.9.0 as well

image: artifactory.dev.earnin.net/docker-remote/openpolicyagent/gatekeeper:v3.9.0

Here is the log

k logs -n gatekeeper deploy/gatekeeper-controller-manager -f
Found 3 pods, using pod/gatekeeper-controller-manager-69b88d77ff-v6fn8
2022/09/28 16:39:44 maxprocs: Updating GOMAXPROCS=5: determined from CPU quota
2022/09/28 16:40:01 http: TLS handshake error from x.x.x.x:48490: EOF

any idea on whats causing this issue and how I can get it fixed.

[meons]

Same here on GKE 1.22 + Gatekeeper 3.9.0:

kubectl -n gatekeeper logs deployment/gatekeeper-controller-manager | grep error
Found 3 pods, using pod/gatekeeper-controller-manager-888b9f574-h4vjz
2022/10/09 10:27:38 http: TLS handshake error from x.x.x.x:50782: EOF
2022/10/09 10:36:56 http: TLS handshake error from x.x.x.x:52720: EOF
2022/10/09 10:56:14 http: TLS handshake error from x.x.x.x:55364: EOF
2022/10/09 11:05:39 http: TLS handshake error from x.x.x.x:58102: EOF
2022/10/09 11:34:01 http: TLS handshake error from x.x.x.x:49868: EOF
2022/10/09 13:30:59 http: TLS handshake error from x.x.x.x:46064: EOF
2022/10/09 14:45:18 http: TLS handshake error from x.x.x.x:55056: EOF
2022/10/09 15:06:19 http: TLS handshake error from x.x.x.x:54452: EOF
2022/10/09 16:07:31 http: TLS handshake error from x.x.x.x:54824: EOF
2022/10/09 16:16:03 http: TLS handshake error from x.x.x.x:37644: EOF
2022/10/09 16:43:38 http: TLS handshake error from x.x.x.x:46590: EOF
2022/10/09 21:27:03 http: TLS handshake error from x.x.x.x:54706: EOF
2022/10/09 21:40:59 http: TLS handshake error from x.x.x.x:47458: EOF
2022/10/09 22:47:10 http: TLS handshake error from x.x.x.x:50688: EOF
2022/10/10 00:18:11 http: TLS handshake error from x.x.x.x:44814: EOF
2022/10/10 01:13:12 http: TLS handshake error from x.x.x.x:42378: EOF
2022/10/10 02:45:59 http: TLS handshake error from x.x.x.x:47150: EOF
2022/10/10 02:56:03 http: TLS handshake error from x.x.x.x:32800: EOF
2022/10/10 03:33:21 http: TLS handshake error from x.x.x.x:52332: EOF
2022/10/10 03:58:46 http: TLS handshake error from x.x.x.x:45042: EOF
2022/10/10 04:59:54 http: TLS handshake error from x.x.x.x:48482: EOF
2022/10/10 05:49:48 http: TLS handshake error from x.x.x.x:42376: EOF
2022/10/10 05:57:33 http: TLS handshake error from x.x.x.x:41712: EOF
2022/10/10 07:11:39 http: TLS handshake error from x.x.x.x:60302: EOF

Actually x.x.x.x are GKE control planes IPs.

[ZiaUrRehman-GBI]

Kubernetes version : 1.24.3-gke.2100

textPayload: "2022/10/31 11:20:06 http: TLS handshake error from x.x.x.x:41398: EOF"

[kfox1111]

seen on k8s 1.21.11
2022/11/03 19:17:10 http: TLS handshake error from 10.17.0.0:52110: EOF

not sure its affecting anything.

[tspearconquest]

Hello, I've noticed these before but not had time to do some proper investigation until now.

I found that these messages are coming from an IP belonging to the konnectivity pods in my kube-system namespace in Azure.

This pod is facilitating the control plane to cluster communications as per https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

Digging into the kube-system namespace labels, I see that there is control-plane: true on that namespace.

I believe what's going on which is causing this, is that konnectivity-agent is looking for all namespaces where the label control-plane exists (regardless of the value) and trying to make a connection to the gatekeeper pods.

I found #1061 which covers the removal of the control-plane label, however it has only been partially implemented by removing the check from the validating webhook configuration in Gatekeeper (#758)

Is it safe to remove the control-plane: controller-manager label from the gatekeeper-system namespace currently, if we have already applied the admission.gatekeeper.sh/ignore: no-self-managing label?

In case it is safe, then we should push to have the control-plane label removed from the namespace as soon as possible, as this is really causing problems for teams with log monitoring agents like fluentd.