kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

[t0rr3sp3dr0]

What were you trying to accomplish?

Create a new nodegroup of m5zn.xlarge instances using the Ubuntu2004 AMI Family.

What happened?

kubelet fails to start with the following error:

2021-01-14T23:41:55Z kubelet-eks.daemon[5432]: F0114 23:41:55.418802    5432 server.go:274] failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"

How to reproduce it?

apiVersion: eksctl.io/v1alpha5
cloudWatch:
  clusterLogging:
    enableTypes:
      - '*'
kind: ClusterConfig
metadata:
  name: ***
  region: us-east-1
  tags:
    Owner: ***
  version: '1.18'
nodeGroups:
  - asgMetricsCollection:
      - granularity: 1Minute
        metrics:
          - GroupMinSize
          - GroupMaxSize
          - GroupDesiredCapacity
          - GroupInServiceInstances
          - GroupPendingInstances
          - GroupStandbyInstances
          - GroupTerminatingInstances
          - GroupTotalInstances
    availabilityZones:
      - us-east-1f
    desiredCapacity: 1
    ebsOptimized: true
    iam:
      instanceProfileARN: ***
      instanceRoleARN: ***
    instanceType: m5zn.xlarge
    kubeletExtraConfig:
      cpuCFSQuota: false
      evictionHard:
        memory.available: 300Mi
        nodefs.available: 10%
      kubeReserved:
        cpu: 300m
        ephemeral-storage: 1Gi
        memory: 300Mi
      kubeReservedCgroup: /kube-reserved
      systemReserved:
        cpu: 300m
        ephemeral-storage: 1Gi
        memory: 300Mi
    maxSize: 100
    minSize: 0
    name: ***
    amiFamily: Ubuntu2004
    preBootstrapCommands:
      - ***
      - echo '{"registry-mirrors":["https://mirror.gcr.io"]}' > /etc/docker/daemon.json
      - service docker restart # Must restart daemon in order to use the registry mirror
      - echo '4294967296' > /proc/sys/fs/aio-max-nr
      - echo 'vm.max_map_count = 262144' > /etc/sysctl.d/90-maxmapcount.conf
      - sysctl -p /etc/sysctl.d/90-maxmapcount.conf
    privateNetworking: true
    tags:
      Owner: ***
      k8s.io/cluster-autoscaler/enabled: 'true'
      k8s.io/cluster-autoscaler/***: owned
    volumeEncrypted: true
    volumeSize: 32
vpc:
  clusterEndpoints:
    privateAccess: true
    publicAccess: false
  subnets:
    private:
      us-east-1c:
        id: ***
      us-east-1f:
        id: ***

Versions

$ eksctl version
0.35.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-14T05:15:04Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[Callisto13]

Hi @t0rr3sp3dr0 thanks for reporting this 👍 .

This should work. The submitter of the PR which brought the cgroup driver change verified that it worked with the Ubuntu 2004 AMI family #2962.

It is possible that your preBootstrapCommand echo '{"registry-mirrors":["https://mirror.gcr.io"]}' > /etc/docker/daemon.json is overwriting the daemon.json which is where the driver is set on the docker daemon. Could you try adding that mirror config using a sed instead?

[t0rr3sp3dr0]

Hi @Callisto13,

I was only able to confirm that what you said was indeed the problem now.

Replacing

echo '{"registry-mirrors":["https://mirror.gcr.io"]}' > /etc/docker/daemon.json

with

cat <<< $(jq '."registry-mirrors" |= . + ["https://mirror.gcr.io"]' /etc/docker/daemon.json) > /etc/docker/daemon.json

solve the problem.

Thanks for the help!