From c50c151dc71327602a9e60542db9bf0f707160e9 Mon Sep 17 00:00:00 2001 From: Markus Lehtonen Date: Wed, 19 Jul 2023 13:12:15 +0300 Subject: [PATCH 1/4] docs: document kubelet cgroup driver detection from the runtime Document the KubeletCgroupDriverFromCRI feature gate. Also, add notes of this feature in parts of the documentation that describe cgroup driver configuration. --- .../feature-gates.md | 9 +++++++++ .../container-runtimes.md | 19 +++++++++++++++++++ .../kubeadm/configure-cgroup-driver.md | 7 +++++++ 3 files changed, 35 insertions(+) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 88e2cae5d5ec8..9d94802ae787e 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -125,6 +125,7 @@ For a reference to old feature gates that are removed, please refer to | `JobReadyPods` | `true` | Beta | 1.24 | | | `KMSv2` | `false` | Alpha | 1.25 | 1.26 | | `KMSv2` | `true` | Beta | 1.27 | | +| `KubeletCgroupDriverFromCRI` | `false` | Alpha | 1.28 | | | `KubeletInUserNamespace` | `false` | Alpha | 1.22 | | | `KubeletPodResourcesDynamicResources` | `false` | Alpha | 1.27 | | | `KubeletPodResourcesGet` | `false` | Alpha | 1.27 | | @@ -559,6 +560,14 @@ Each feature gate is designed for enabling/disabling a specific feature: The Job controller uses Pod finalizers and a field in the Job status to keep track of the finished Pods to count towards completion. - `KMSv2`: Enables KMS v2 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details. +- `KubeletCgroupDriverFromCRI`: Enable detection of the kubelet cgroup driver + configuration option from the CRI. This feature gate requires the user run a + container runtime that supports the `RuntimeConfig` CRI call. If both CRI and + Kubelet support this feature, the kubelet will ignore the cgroupDriver + (--cgroup-driver) configuration option. If the container runtime doesn't + support it, the kubelet will fallback to using its cgroupDriver option. + See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver) + for more details. - `KubeletCredentialProviders`: Enable kubelet exec credential providers for image pull credentials. - `KubeletInUserNamespace`: Enables support for running kubelet in a diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index 46ffce3bb9d54..8b50d1836505d 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -157,6 +157,13 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. {{< /note >}} +{{< note >}} +Starting with v1.28 and later, with KubeletCgroupDriverFromCRI feature gate +enabled and a container runtime that supports the RuntimeConfig CRI rpc, +kubelet will automatically detect the cgroup driver from the runtime and the +cgroupDriver setting in kubelet configuration is ignored. +{{< /note >}} + If you configure `systemd` as the cgroup driver for the kubelet, you must also configure `systemd` as the cgroup driver for the container runtime. Refer to the documentation for your container runtime for instructions. For example: @@ -251,6 +258,12 @@ sudo systemctl restart containerd When using kubeadm, manually configure the [cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver). +{{< note >}} +Starting with v1.28 and later, kubelet cgroup driver does not need to be +configured if the KubeletCgroupDriverFromCRI feature gate is enabled and a +version of containerd that supports the RuntimeConfig CRI rpc is being used. +{{< /note >}} + #### Overriding the sandbox (pause) image {#override-pause-image-containerd} In your [containerd config](https://github.com/containerd/containerd/blob/main/docs/cri/config.md) you can overwrite the @@ -292,6 +305,12 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O in sync. +{{< note >}} +Starting with v1.28 and later, kubelet cgroup driver does not need to be +configured if the KubeletCgroupDriverFromCRI feature gate is enabled and a +version of CRI-O that supports the RuntimeConfig CRI rpc is being used. +{{< /note >}} + For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default. #### Overriding the sandbox (pause) image {#override-pause-image-cri-o} diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md index f4308c2b0eb35..9c65eb67a70c8 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -38,6 +38,13 @@ In v1.22 and later, if the user does not set the `cgroupDriver` field under `Kub kubeadm defaults it to `systemd`. {{< /note >}} +{{< note >}} +In v1.28 and later, with KubeletCgroupDriverFromCRI feature gate +enabled and a container runtime that supports the RuntimeConfig CRI rpc, +kubelet will automatically detect the cgroup driver from the runtime and the +kubelet cgroup driver setting does not need to be manually configured. +{{< /note >}} + A minimal example of configuring the field explicitly: ```yaml From 7e208b05f1adb7873f3d6fe77dd08ff959e44095 Mon Sep 17 00:00:00 2001 From: Markus Lehtonen Date: Mon, 24 Jul 2023 21:42:56 +0300 Subject: [PATCH 2/4] docs: incorporate review comments from sftim Co-authored-by: Tim Bannister --- .../feature-gates.md | 13 +++++---- .../container-runtimes.md | 27 ++++++++----------- .../kubeadm/configure-cgroup-driver.md | 10 +++---- 3 files changed, 23 insertions(+), 27 deletions(-) diff --git a/content/en/docs/reference/command-line-tools-reference/feature-gates.md b/content/en/docs/reference/command-line-tools-reference/feature-gates.md index 9d94802ae787e..bd47a9cf0d6d8 100644 --- a/content/en/docs/reference/command-line-tools-reference/feature-gates.md +++ b/content/en/docs/reference/command-line-tools-reference/feature-gates.md @@ -561,11 +561,14 @@ Each feature gate is designed for enabling/disabling a specific feature: track of the finished Pods to count towards completion. - `KMSv2`: Enables KMS v2 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details. - `KubeletCgroupDriverFromCRI`: Enable detection of the kubelet cgroup driver - configuration option from the CRI. This feature gate requires the user run a - container runtime that supports the `RuntimeConfig` CRI call. If both CRI and - Kubelet support this feature, the kubelet will ignore the cgroupDriver - (--cgroup-driver) configuration option. If the container runtime doesn't - support it, the kubelet will fallback to using its cgroupDriver option. + configuration option from the {{}}. + You can use this feature gate on nodes with a kubelet that supports the feature gate + and where there is a CRI container runtime that supports the `RuntimeConfig` + CRI call. If both CRI and kubelet support this feature, the kubelet ignores the + `cgroupDriver` configuration setting (or deprecated `--cgroup-driver` command + line argument). If you enable this feature gate and the container runtime + doesn't support it, the kubelet falls back to using the driver configured using + the `cgroupDriver` configuration setting. See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver) for more details. - `KubeletCredentialProviders`: Enable kubelet exec credential providers for diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index 8b50d1836505d..ce49a7347db06 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -157,12 +157,11 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. {{< /note >}} -{{< note >}} -Starting with v1.28 and later, with KubeletCgroupDriverFromCRI feature gate -enabled and a container runtime that supports the RuntimeConfig CRI rpc, -kubelet will automatically detect the cgroup driver from the runtime and the -cgroupDriver setting in kubelet configuration is ignored. -{{< /note >}} +For Kubernetes v1.28 and later, with the `KubeletCgroupDriverFromCRI` +[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) +enabled and a container runtime that supports the `RuntimeConfig` CRI RPC, +the kubelet automatically detects the appropriate cgroup driver from the runtime, +and ignores the `cgroupDriver` setting within the kubelet configuration. If you configure `systemd` as the cgroup driver for the kubelet, you must also configure `systemd` as the cgroup driver for the container runtime. Refer to @@ -258,11 +257,9 @@ sudo systemctl restart containerd When using kubeadm, manually configure the [cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver). -{{< note >}} -Starting with v1.28 and later, kubelet cgroup driver does not need to be -configured if the KubeletCgroupDriverFromCRI feature gate is enabled and a -version of containerd that supports the RuntimeConfig CRI rpc is being used. -{{< /note >}} +Starting with v1.28 and later, you can enable automatic detection of the +cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver) +for more details. #### Overriding the sandbox (pause) image {#override-pause-image-containerd} @@ -305,11 +302,9 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O in sync. -{{< note >}} -Starting with v1.28 and later, kubelet cgroup driver does not need to be -configured if the KubeletCgroupDriverFromCRI feature gate is enabled and a -version of CRI-O that supports the RuntimeConfig CRI rpc is being used. -{{< /note >}} +Starting with v1.28 and later, you can enable automatic detection of the +cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver) +for more details. For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md index 9c65eb67a70c8..e3133054f5ac0 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -36,13 +36,11 @@ driver of the kubelet. {{< note >}} In v1.22 and later, if the user does not set the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. -{{< /note >}} -{{< note >}} -In v1.28 and later, with KubeletCgroupDriverFromCRI feature gate -enabled and a container runtime that supports the RuntimeConfig CRI rpc, -kubelet will automatically detect the cgroup driver from the runtime and the -kubelet cgroup driver setting does not need to be manually configured. +Starting with v1.28 and later, you can enable automatic detection of the +cgroup driver as an alpha feature. +See [systemd cgroup driver](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver) +for more details. {{< /note >}} A minimal example of configuring the field explicitly: From 0d2962b0f80ab626209788d1c00cfabc91c09dae Mon Sep 17 00:00:00 2001 From: Markus Lehtonen Date: Wed, 26 Jul 2023 11:18:14 +0300 Subject: [PATCH 3/4] docs: change wording of k8s versioning on KubeletCgroupDriverFromCRI --- .../docs/setup/production-environment/container-runtimes.md | 6 +++--- .../administer-cluster/kubeadm/configure-cgroup-driver.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index ce49a7347db06..36bf4ef299448 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -157,7 +157,7 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. {{< /note >}} -For Kubernetes v1.28 and later, with the `KubeletCgroupDriverFromCRI` +In Kubernetes {{< skew currentVersion >}}, with the `KubeletCgroupDriverFromCRI` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled and a container runtime that supports the `RuntimeConfig` CRI RPC, the kubelet automatically detects the appropriate cgroup driver from the runtime, @@ -257,7 +257,7 @@ sudo systemctl restart containerd When using kubeadm, manually configure the [cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver). -Starting with v1.28 and later, you can enable automatic detection of the +In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver) for more details. @@ -302,7 +302,7 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O in sync. -Starting with v1.28 and later, you can enable automatic detection of the +In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver) for more details. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md index e3133054f5ac0..afaa370011b82 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -37,7 +37,7 @@ driver of the kubelet. In v1.22 and later, if the user does not set the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. -Starting with v1.28 and later, you can enable automatic detection of the +In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the cgroup driver as an alpha feature. See [systemd cgroup driver](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver) for more details. From fa73830e0b4159f2a31fd7a144f1c05bbec4e61b Mon Sep 17 00:00:00 2001 From: Markus Lehtonen Date: Fri, 4 Aug 2023 14:06:35 +0300 Subject: [PATCH 4/4] docs: another change of wording of k8s versioning --- .../docs/setup/production-environment/container-runtimes.md | 6 +++--- .../administer-cluster/kubeadm/configure-cgroup-driver.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index 36bf4ef299448..957b8d3c0fe13 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -157,7 +157,7 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. {{< /note >}} -In Kubernetes {{< skew currentVersion >}}, with the `KubeletCgroupDriverFromCRI` +In Kubernetes v1.28, with the `KubeletCgroupDriverFromCRI` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/) enabled and a container runtime that supports the `RuntimeConfig` CRI RPC, the kubelet automatically detects the appropriate cgroup driver from the runtime, @@ -257,7 +257,7 @@ sudo systemctl restart containerd When using kubeadm, manually configure the [cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver). -In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the +In Kubernetes v1.28, you can enable automatic detection of the cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver) for more details. @@ -302,7 +302,7 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O in sync. -In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the +In Kubernetes v1.28, you can enable automatic detection of the cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver) for more details. diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md index afaa370011b82..b811985819c00 100644 --- a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -37,7 +37,7 @@ driver of the kubelet. In v1.22 and later, if the user does not set the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`. -In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the +In Kubernetes v1.28, you can enable automatic detection of the cgroup driver as an alpha feature. See [systemd cgroup driver](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver) for more details.