Skip to content

docs: document kubelet cgroup driver detection from the runtime #42160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
k8s-ci-robot merged 4 commits into from
Aug 5, 2023
Merged

docs: document kubelet cgroup driver detection from the runtime #42160

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c50c151
docs: document kubelet cgroup driver detection from the runtime
marquiz Jul 19, 2023
7e208b0
docs: incorporate review comments from sftim
marquiz Jul 24, 2023
0d2962b
docs: change wording of k8s versioning on KubeletCgroupDriverFromCRI
marquiz Jul 26, 2023
fa73830
docs: another change of wording of k8s versioning
marquiz Aug 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions content/en/docs/reference/command-line-tools-reference/feature-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ For a reference to old feature gates that are removed, please refer to
| `JobReadyPods` | `true` | Beta | 1.24 | |
| `KMSv2` | `false` | Alpha | 1.25 | 1.26 |
| `KMSv2` | `true` | Beta | 1.27 | |
| `KubeletCgroupDriverFromCRI` | `false` | Alpha | 1.28 | |
| `KubeletInUserNamespace` | `false` | Alpha | 1.22 | |
| `KubeletPodResourcesDynamicResources` | `false` | Alpha | 1.27 | |
| `KubeletPodResourcesGet` | `false` | Alpha | 1.27 | |
Expand Down Expand Up @@ -559,6 +560,17 @@ Each feature gate is designed for enabling/disabling a specific feature:
The Job controller uses Pod finalizers and a field in the Job status to keep
track of the finished Pods to count towards completion.
- `KMSv2`: Enables KMS v2 API for encryption at rest. See [Using a KMS Provider for data encryption](/docs/tasks/administer-cluster/kms-provider) for more details.
- `KubeletCgroupDriverFromCRI`: Enable detection of the kubelet cgroup driver
configuration option from the {{<glossary_tooltip term_id="cri" text="CRI">}}.
You can use this feature gate on nodes with a kubelet that supports the feature gate
and where there is a CRI container runtime that supports the `RuntimeConfig`
CRI call. If both CRI and kubelet support this feature, the kubelet ignores the
`cgroupDriver` configuration setting (or deprecated `--cgroup-driver` command
line argument). If you enable this feature gate and the container runtime
doesn't support it, the kubelet falls back to using the driver configured using
the `cgroupDriver` configuration setting.
See [Configuring a cgroup driver](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver)
for more details.
- `KubeletCredentialProviders`: Enable kubelet exec credential providers for
image pull credentials.
- `KubeletInUserNamespace`: Enables support for running kubelet in a
Expand Down
14 changes: 14 additions & 0 deletions content/en/docs/setup/production-environment/container-runtimes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,12 @@ Starting with v1.22 and later, when creating a cluster with kubeadm, if the user
the `cgroupDriver` field under `KubeletConfiguration`, kubeadm defaults it to `systemd`.
{{< /note >}}

In Kubernetes {{< skew currentVersion >}}, with the `KubeletCgroupDriverFromCRI`
Copy link
Contributor

@tengqm tengqm Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suppose we are in v1.29 now, the above line will be rendered into something like
"In Kubernetes 1.29, with the KubeletCgroupDriverFromCRI".
Is this statement still appropriate?
Suppose we are in v1.34 now and the feature gate has been GA'ed and removed,
the above line will be rendered into something like
"In Kubernetes 1.34, with the KubeletCgroupDriverFromCRI".
Is this statement still appropriate?

Clearly stating that this feature is in Alpha starting 1.28 is good enough.
Why do we use the skew shortcode the make this release number a moving target?

Copy link
Contributor Author

@marquiz marquiz Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm indifferent on this, can change it to v1.29. This (In Kubernetes {{< skew currentVersion >}}) was something that @sftim suggested in one of his earlier comments so I took it from there.

Copy link
Contributor

@sftim sftim Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suppose we are in v1.34 now and the feature gate has been GA'ed and removed,

When we change Kubernetes, we should remove the docs. If we don't trust ourselves to do that, I would focus on fixing that lack of trust.

If we could trust the release docs process, I think this change would be fine.

Copy link
Contributor Author

@marquiz marquiz Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a contributor I'm inclined to agree with @sftim about the docs process here. Especially with a KEP involved, the process should ensure that the documentation is changed when the feature is moved to beta and later GA (or removed before graduating further). WDYT @tengqm?

Copy link
Contributor

@sftim sftim Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per #42160 (comment), let's follow the path of least risk to the release, at least for now.

Copy link
Contributor Author

@marquiz marquiz Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K, I changed this now to In Kubernetes v1.28, ... (not to Starting with v1.28 ... as per #42160 (comment))

PTAL @sftim @tengqm

[feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
enabled and a container runtime that supports the `RuntimeConfig` CRI RPC,
the kubelet automatically detects the appropriate cgroup driver from the runtime,
and ignores the `cgroupDriver` setting within the kubelet configuration.

If you configure `systemd` as the cgroup driver for the kubelet, you must also
configure `systemd` as the cgroup driver for the container runtime. Refer to
the documentation for your container runtime for instructions. For example:
Expand Down Expand Up @@ -251,6 +257,10 @@ sudo systemctl restart containerd
When using kubeadm, manually configure the
[cgroup driver for kubelet](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#configuring-the-kubelet-cgroup-driver).

In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the
Copy link
Contributor

@tengqm tengqm Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

In addition to the above confusion, using this skew shortcode makes it difficult
to remove contents that are no longer relevant in the future.

cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.

#### Overriding the sandbox (pause) image {#override-pause-image-containerd}

In your [containerd config](https://github.com/containerd/containerd/blob/main/docs/cri/config.md) you can overwrite the
Expand Down Expand Up @@ -292,6 +302,10 @@ You should also note the changed `conmon_cgroup`, which has to be set to the val
cgroup driver configuration of the kubelet (usually done via kubeadm) and CRI-O
in sync.

In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the
Copy link
Contributor

@tengqm tengqm Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

cgroup driver as an alpha feature. See [systemd cgroup driver](#systemd-cgroup-driver)
for more details.

For CRI-O, the CRI socket is `/var/run/crio/crio.sock` by default.

#### Overriding the sandbox (pause) image {#override-pause-image-cri-o}
Expand Down
5 changes: 5 additions & 0 deletions content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ driver of the kubelet.
{{< note >}}
In v1.22 and later, if the user does not set the `cgroupDriver` field under `KubeletConfiguration`,
kubeadm defaults it to `systemd`.

In Kubernetes {{< skew currentVersion >}}, you can enable automatic detection of the
Copy link
Contributor

@tengqm tengqm Jul 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

cgroup driver as an alpha feature.
See [systemd cgroup driver](/docs/setup/production-environment/container-runtimes/#systemd-cgroup-driver)
for more details.
{{< /note >}}

A minimal example of configuring the field explicitly:
Expand Down