agent-inject-command break command after any bash operators @ vault 1.70

[mattgialelis]

Description
Based off this response im creating an issue here for this as the vault agent uses this consul template to do its work :)
hashicorp/vault-k8s#246 (comment)

As of vault 1.7.0 there is an issue with the agent-inject-command not fully reading in bash commands after any operators. e.g ; , && , ||, '>'

Multiple different methods to escape the operators was tried none seemed to make any difference the command on its own is:

"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'"

Which is used to restart the deployment on a secret change.

This causes the commands either to not run at all or just error out like the logs provided below, ive tested with vault 1.5.4 and 1.6.3 which both work well and breaks on 1.7.0

Vault 1.7.0 (Broken)

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.7.0
2021-04-14T08:48:04.821Z [INFO]  sink.file: creating file sink
2021-04-14T08:48:04.821Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
             Version Sha: 4e222b85c40a810b74400ee3c54449479e32bb9f
             
 (Removed extra lines due to spam)
 
[DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":""},"Templates":[{"Backup":false,"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","CommandTimeout":30000000000,"Contents":"{{- with secret \"secret/myapp/config\" -}}\n  vars:\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n{{- with secret \"secret/generic/config\" -}}\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n","CreateDestDirs":true,"Destination":"/vault/secrets/config.yml","ErrMissingKey":false,"Exec":{"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","Enabled":true,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://10.200.0.150:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}
             
[DEBUG] (runner) vault.read(secret/myapp/config) is still needed
[DEBUG] (runner) vault.read(secret/generic/config) is still needed
[INFO] (runner) executing command "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'" from "(dynamic)" => "/vault/secrets/config.yml"
[INFO] (child) spawning: sh -c { if [ -f /vault/secrets/config.check ]
if: line 1: syntax error: unexpected end of file (expecting "}")
[INFO] (runner) stopping
[DEBUG] (runner) stopping watcher
[DEBUG] (watcher) stopping all views

Vault 1.6.3 (working)

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.6.3
             Version Sha: b540be4b7ec48d0dd7512c8d8df9399d6bf84d76

 (Removed extra lines due to spam)

2021/04/14 08:27:57.112387 [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":""},"Templates":[{"Backup":false,"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","CommandTimeout":30000000000,"Contents":"{{- with secret \"secret/myapp/config\" -}}\n  vars:\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n{{- with secret \"secret/generic/config\" -}}\n  {{ range $k, $v := .Data }}\n    {{ $k }}: {{ $v }}\n  {{ end }}\n{{- end }}\n","CreateDestDirs":true,"Destination":"/vault/secrets/config.yml","ErrMissingKey":false,"Exec":{"Command":"sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'","Enabled":true,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://10.200.0.150:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":3,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}

2021/04/14 08:27:57.281910 [DEBUG] (runner) vault.read(secret/myapp/config) is still needed
2021/04/14 08:27:57.281916 [DEBUG] (runner) vault.read(secret/generic/config) is still needed
2021/04/14 08:27:57.281930 [INFO] (runner) executing command "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'" from "(dynamic)" => "/vault/secrets/config.yml"
2021/04/14 08:27:57.282119 [INFO] (child) spawning: sh -c { if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }
2021/04/14 08:27:57.283618 [DEBUG] (runner) watching 2 dependencies
2021/04/14 08:27:57.283735 [DEBUG] (runner) all templates rendered
2021/04/14 08:27:57.283633 [INFO] (runner) stopping

To Reproduce
Steps to reproduce the behavior:

1. Setup Vault-Kube auth
2. Deploy application using provided manifest
3. See error (vault-agent-init container) kubectl logs POD-NAME vault-agent-init

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-agent-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-example
  template:
    metadata:
      labels:
        app: app-example
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-config.yml: "secret/myapp/config"
        vault.hashicorp.com/log-level: debug
        vault.hashicorp.com/agent-inject-template-config.yml: |
          {{- with secret "secret/myapp/config" -}}
            vars:
            {{ range $k, $v := .Data }}
              {{ $k }}: {{ $v }}
            {{ end }}
          {{- end }}
        vault.hashicorp.com/role: 'example'
        vault.hashicorp.com/agent-inject-command-config.yml: "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'"
    spec:
      containers:
        - image: alpine:latest
          name: alpine
          command:
            - /bin/sh
            - "-c"
            - "sleep 60m"
          imagePullPolicy: IfNotPresent
      serviceAccountName: vault-auth

kubectl logs POD-NAME vault-agent-init ( this is where all logs provided are from )

Expected behavior
Command to be fully executed as it was in previous version of vaults-agent

Environment

• Kubernetes version: 1.18.15
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): KOPS in AWS
  • Other configuration options or runtime services (istio, etc.):
  • vault-k8s version: 0.9.0

==> Vault agent configuration:
                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.7.0
             Version Sha: 4e222b85c40a810b74400ee3c54449479e32bb9f

Additional context
We are using an external vault server and the agent injector only from the Vault Helm Chart

    image: hashicorp/vault-k8s:0.9.0
    imageID: docker-pullable://hashicorp/vault-k8s@sha256:65731b0513c95f683ee52528e6ccf24f6de0092700e869cdc5ff5d8354b5d86e

Currently the only workaround is to make our own vault docker image with a bash script which we can call as a single line command store in /usr/local/bin/reload-pods
reload-pods deployment-name

Which does work as it does not require any extra operators in the annotations provided to vault

Have also just tested using the configmap method with the same results

   "auto_auth" = {
      "method" = {
        "config" = {
          "role" = "example"
        }
        "type" = "kubernetes"
      }

      "sink" = {
        "config" = {
          path = "/home/vault/.vault-token"
        }

        "type" = "file"
      }
    }

    "exit_after_auth" = true
    "pid_file" = "/home/vault/.pid"

    "template" = {
      "contents" = "{{- with secret \"secret/myapp/config\" -}}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}{{ end }}{{- end }}"
      "destination" = "/vault/secrets/config"
      "command" =  "sh -c '{ if [ -f /vault/secrets/config.check ]; then kubectl rollout restart deployment vault-agent-example; else touch /vault/secrets/config.check; fi }'"
    }
    "vault" = {
      "address" = "http://10.200.0.150:8200"
    }

[eikenb]

I think this is the same issue as was reported in #1456.

TLDR; It is a known bug reported upstream to the shell parsing library. They haven't fixed it yet and I'm looking into my options.