Linkerd 19.2.5 not auto-injecting into all containers

[christianhuening]

Bug Report

What is the issue?

When using Linkerd 19.2.5 with mTLS and auto-inject it does work for some Deployments in an annotated namespace and for some not.

The deployments are almost the same except for some command args of the starting container. Also both do not contain disabled annotations. Here are some specs:

Working:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 9588b3336733648253c899ec818098b639b2cb7d0fdb9ebad66be1b0aa072b7d
    kubernetes.io/psp: restricted
    linkerd.io/created-by: linkerd/proxy-injector edge-19.2.5
    linkerd.io/identity-mode: optional
    linkerd.io/proxy-version: edge-19.2.5
  creationTimestamp: "2019-02-28T13:32:02Z"
(...)

vs. not working

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 9588b3336733648253c899ec818098b639b2cb7d0fdb9ebad66be1b0aa072b7d
    kubernetes.io/psp: restricted
  creationTimestamp: "2019-02-28T13:32:00Z"
(...)

And here are excerpts from the proxy-injector pod:
working:

time="2019-02-28T13:32:00Z" level=info msg="working on v1/deployment fu-figo-api-task-sync-payment-parameter.."
time="2019-02-28T13:32:00Z" level=info msg="patch generated:

vs not working:

time="2019-02-28T13:31:59Z" level=info msg="working on v1/deployment fu-figo-api-task-register.."
time="2019-02-28T13:31:59Z" level=info msg="skipping deployment fu-figo-api-task-register"

so it seems the injector simply skips this deployment for some reason.

How can it be reproduced?

Logs, error output, etc

(If the output is long, please create a gist and
paste the link here.)

linkerd check output

your output here ...

Environment

• Kubernetes Version: 1.13.3
• Cluster Environment: (GKE, AKS, kops, ...): bare-metal
• Host OS: ContainerLinux 1967.6.0
• Linkerd version: edge-19.2.5

Possible solution

Additional context

[ihcsim]

Following our slack convo, please let us know if this is reproducible in another namespace on your cluster. For the record, the proxy injector will skip the spec mutation if either:

1. There is another proxy sidecar or proxy init container (including istio and envoy) in the same workload
2. There is a linkerd.io/inject: disabled explicitly defined either at the namespace or pod level. (This doesn't seem to be the case here.)

[christianhuening]

It's reproducible in another namespace as well. Interestingly enough it's always the same deployments. I double checked they don't have linkerd.io/inject: disabled defined and we certainly do not have istio in the cluster.

[ihcsim]

Interestingly enough it's always the same deployments

Strange 🤔 Are you able to send me a stripped down version of the deployment YAML?

[christianhuening]

Working:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fu-redacted-api-task-register
  labels:
    app.kubernetes.io/name: redacted-api-task-register
    app.kubernetes.io/instance: fu
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: redacted-api-task-register
      app.kubernetes.io/instance: fu
  template:
    metadata:
      annotations:
        checksum/config: 9588b3336733648253c899ec818098b639b2cb7d0fdb9ebad66be1b0aa072b7d
      labels:
        app.kubernetes.io/name: redacted-api-task-register
        app.kubernetes.io/instance: fu
    spec:
      containers:
        - name: fu-redacted-api-task
          image: <redacted>
          imagePullPolicy: IfNotPresent
          args: ["register"]
          volumeMounts:
            - name: config
              mountPath: /etc/redacted/redacted.ini
              subPath: redacted.ini
              readOnly: true
            - name: certificates
              mountPath: /etc/pki/tls/
              readOnly: true
          resources:
            {}

      imagePullSecrets:
        - name: redacted-api-pull
      volumes:
        - name: config
          configMap:
            name: fu-redacted-api
        - name: certificates
          secret:
            secretName: redacted-api-certificates

Not Working:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fu-redacted-api-task-sync
  labels:
    app.kubernetes.io/name: redacted-api-task-sync
    app.kubernetes.io/instance: fu
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: redacted-api-task-sync
      app.kubernetes.io/instance: fu
  template:
    metadata:
      annotations:
        checksum/config: 9588b3336733648253c899ec818098b639b2cb7d0fdb9ebad66be1b0aa072b7d
      labels:
        app.kubernetes.io/name: redacted-api-task-sync
        app.kubernetes.io/instance: fu
    spec:
      containers:
        - name: fu-redacted-api-task
          image: <redacted>
          imagePullPolicy: IfNotPresent
          args: ["sync"]
          volumeMounts:
            - name: config
              mountPath: /etc/redacted/redacted.ini
              subPath: redacted.ini
              readOnly: true
            - name: certificates
              mountPath: /etc/pki/tls/
              readOnly: true
          resources:
            {}
            
      imagePullSecrets:
        - name: redacted-api-pull
      volumes:
        - name: config
          configMap:
            name: fu-redacted-api
        - name: certificates
          secret:
            secretName: redacted-api-certificates

[christianhuening]

So i found that I am still running on an older version of the MutatingWebhookConfiguration (

linkerd2/controller/proxy-injector/tmpl/mutating_webhook_configuration.go

Lines 19 to 22 in d805cb9

	- operations: [ "CREATE" , "UPDATE" ]
	apiGroups: ["apps", "extensions"]
	apiVersions: ["v1", "v1beta1", "v1beta2"]
	resources: ["deployments"]`

) .

Mine still looks like so:

apiVersion: v1
items:
- apiVersion: admissionregistration.k8s.io/v1beta1
  kind: MutatingWebhookConfiguration
  metadata:
    creationTimestamp: "2019-02-19T07:16:26Z"
    generation: 5
    name: linkerd-proxy-injector-webhook-config
  webhooks:
  - clientConfig:
      caBundle: <redacted>
      service:
        name: linkerd-proxy-injector
        namespace: linkerd
        path: /
    failurePolicy: Ignore
    name: linkerd-proxy-injector.linkerd.io
    namespaceSelector: {}
    rules:
    - apiGroups:
      - apps
      - extensions
      apiVersions:
      - v1
      - v1beta1
      - v1beta2
      operations:
      - CREATE
      resources:
      - deployments
    sideEffects: Unknown
kind: List

[ihcsim]

So sounds like it's the same issue as #2425, where the MWC isn't upgraded during a version upgrade. I have submitted a fix in PR #2431. Meanwhile, you can use kubectl edit to update the webhooks.rules[0].operations list to include UPDATE.

[christianhuening]

@ihcsim Ok, I recreated the MutatingWebhookConfiguration by deleting it and restarting the injector proxy. It got created just fine with the UPDATE verb in it. I then redeployed everything and all pods were injected automatically.
This is great, but I find it a bit weird that the UPDATE verb's presence or absence alters the general behavior of the injector-proxy. Or what kind of issue could this be?

[ihcsim]

It can also be that the CA bundle in the MWC got out-of-sync during the upgrade to edge-19.2.5. If that's the case, you should see the proxy injector TLS errors in the k8s API server logs. PR #2431 will fix that too.

[christianhuening]

would a restart of the injector fix the CA bundle desync?