Skip to content

docs: document kubelet cgroup driver detection from the runtime #42160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
k8s-ci-robot merged 4 commits into from
Aug 5, 2023
Merged

docs: document kubelet cgroup driver detection from the runtime #42160

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c50c151
docs: document kubelet cgroup driver detection from the runtime
marquiz Jul 19, 2023
7e208b0
docs: incorporate review comments from sftim
marquiz Jul 24, 2023
0d2962b
docs: change wording of k8s versioning on KubeletCgroupDriverFromCRI
marquiz Jul 26, 2023
fa73830
docs: another change of wording of k8s versioning
marquiz Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions content/en/docs/reference/command-line-tools-reference/feature-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ For a reference to old feature gates that are removed, please refer to
| `JobReadyPods` | `true` | Beta | 1.24 | |
| `KMSv2` | `false` | Alpha | 1.25 | 1.26 |
| `KMSv2` | `true` | Beta | 1.27 | |
| `KubeletCgroupDriverFromCRI` | `false` | Alpha | 1.28 | |
| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | |
| `KubeletPodResourcesDynamicResources` | `false` | Alpha | 1.27 | |
| `KubeletPodResourcesGet` | `false` | Alpha | 1.27 | |
Expand Down Expand Up @@ -559,6 +560,17 @@ Each feature gate is designed for enabling/disabling a specific feature:
The Job controller uses Pod finalizers and a field in the Job status to keep
track of the finished Pods to count towards completion.
- `KMSv2`: Enables KMS v2 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details.
- `KubeletCgroupDriverFromCRI`: Enable detection of the kubelet cgroup driver
configuration option from the {{<glossary_tooltip term_id="cri" text="CRI">}}.
You can use this feature gate on nodes with a kubelet that supports the feature gate
and where there is a CRI container runtime that supports the `RuntimeConfig`
CRI call. If both CRI and kubelet support this feature, the kubelet ignores the
`cgroupDriver` configuration setting (or deprecated `--cgroup-driver` command
line argument). If you enable this feature gate and the container runtime
doesn't support it, the kubelet falls back to using the driver configured using
the `cgroupDriver` configuration setting.
See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver)
for more details.
- `KubeletCredentialProviders`: Enable kubelet exec credential providers for
image pull credentials.
- `KubeletInUserNamespace`: Enables support for running kubelet in a
Expand Down
14 changes: 14 additions & 0 deletions content/en/docs/setup/production-environment/container-runtimes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,12 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user
the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`.
{{< /note >}}

For Kubernetes v1.28 and later, with the `KubeletCgroupDriverFromCRI`
Copy link
Member

@mikebrow mikebrow Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bit of a tldr issue I believe.. giving the instructions above for kubelet config driven and default if not configured.. then ending with 1.28 alpha feature gate info.

Suggest mentioning the feature gate above in direct association with the kubelet config steps / options. Something like unless the alpha level feature gate is set to automatically retrieve the appropriate cgroup driver from the runtime, additional configuration may be required...

Copy link
Contributor Author

@marquiz marquiz Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit cautious on this ("advertising" too much) as this is an alpha feature and there are no released container runtime supporting this, yet. When we get to beta, sure. WDYT @sftim

Copy link
Member

@mikebrow mikebrow Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point...

Copy link
Contributor

@sftim sftim Aug 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should document how Kubernetes behaves, even if the compatible runtimes don't exist on release day.

[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
enabled and a container runtime that supports the `RuntimeConfig` CRI RPC,
the kubelet automatically detects the appropriate cgroup driver from the runtime,
and ignores the `cgroupDriver` setting within the kubelet configuration.

If you configure `systemd` as the cgroup driver for the kubelet, you must also
configure `systemd` as the cgroup driver for the container runtime. Refer to
the documentation for your container runtime for instructions. For example:
Expand Down Expand Up @@ -251,6 +257,10 @@ sudo systemctl restart containerd
When using kubeadm, manually configure the
[cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver).

Starting with v1.28 and later, you can enable automatic detection of the
Copy link
Member

@mikebrow mikebrow Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Starting with v1.28 and later, you can enable automatic detection of the
Starting with Kubernetes v1.28 and later, you can enable automatic detection of the

Copy link
Contributor

@sftim sftim Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aside, but also a bit relevant:
I'm slightly wary of saying “and later” about alpha features that we can't prove will graduate to stable.

Copy link
Contributor Author

@marquiz marquiz Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, have to agree on that. I now changed all of these to In Kubernetes {{< skew currentVersion >}}...

ok @sftim @mikebrow ?

Copy link
Member

@mikebrow mikebrow Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given the concern over skew "and later" comments perhaps.. for Kubernetes v1.2.8, you can..

cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.

#### Overriding the sandbox (pause) image {#override-pause-image-containerd}

In your [containerd config](https://github.com/containerd/containerd/blob/main/docs/cri/config.md) you can overwrite the
Expand Down Expand Up @@ -292,6 +302,10 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val
cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O
in sync.

Starting with v1.28 and later, you can enable automatic detection of the
Copy link
Member

@mikebrow mikebrow Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Starting with v1.28 and later, you can enable automatic detection of the
Starting with Kubernetes v1.28 and later, you can enable automatic detection of the

Copy link
Contributor Author

@marquiz marquiz Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed with In Kubernetes {{< skew currentVersion >}}

cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.

For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default.

#### Overriding the sandbox (pause) image {#override-pause-image-cri-o}
Expand Down
5 changes: 5 additions & 0 deletions content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ driver of the kubelet.
{{< note >}}
In v1.22 and later, if the user does not set the `cgroupDriver` field under `KubeletConfiguration`,
kubeadm defaults it to `systemd`.

Starting with v1.28 and later, you can enable automatic detection of the
Copy link
Member

@mikebrow mikebrow Jul 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see tldr comment

cgroup driver as an alpha feature.
See [systemd cgroup driver](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver)
for more details.
{{< /note >}}

A minimal example of configuring the field explicitly:
Expand Down