vault agent - command execution over 30s disables template server completely

[avoidik]

Describe the bug

an issue was found when the consul rendering server inside the vault agent service had failed to render templates and was stopped, but the vault-agent service itself left in a half-working state running (i.e. auto-auth works, rendering templates doesn't), and the only solution was to restart it

vault[16133]: 2020-05-23T04:41:44.742Z [INFO]  auth.handler: renewed auth token
vault[16133]: 2020/05/23 04:41:44.842069 [INFO] (runner) rendered "/etc/blah.conf" => "/etc/blah.conf"
vault[16133]: 2020/05/23 04:41:44.842826 [INFO] (runner) executing command "systemctl restart unrelated.service" from "/etc/blah.conf" => "/etc/blah.conf"
vault[16133]: 2020/05/23 04:41:44.843651 [INFO] (child) spawning: systemctl restart unrelated.service
vault[16133]: 2020/05/23 04:41:45.080307 [INFO] (runner) rendered "(dynamic)" => "/etc/ca.pem"
vault[16133]: 2020/05/23 04:41:45.080730 [INFO] (runner) executing command "/usr/bin/blah restart" from "(dynamic)" => "/etc/ca.pem"
vault[16133]: 2020/05/23 04:41:45.080955 [INFO] (child) spawning: /usr/bin/blah restart
vault[16133]: Stopping blah...
vault[16133]: Shutting down.  Please wait, as this may take a few minutes.
vault[16133]: .........2020-05-23T04:42:15.084Z [ERROR] template.server: template server error: error="1 error occurred:
vault[16133]:         * failed to execute command "/usr/bin/blah restart" from "(dynamic)" => "/etc/ca.pem": child: command did not exit within "30s":
vault[16133]:     /usr/bin/blah restart
vault[16133]: Commands must exit in a timely manner in order for processing to
vault[16133]: continue. Consider using a process supervisor or utilizing the
vault[16133]: built-in exec mode instead.
vault[16133]: "
vault[16133]: 2020-05-23T04:42:15.084Z [INFO]  template.server: template server stopped
vault[16133]: 2020-05-28T03:44:24.936Z [INFO]  auth.handler: renewed auth token
vault[16133]: 2020-05-28T03:44:24.937Z [INFO]  auth.handler: lifetime watcher done channel triggered
vault[16133]: 2020-05-28T03:44:24.938Z [INFO]  auth.handler: authenticating
vault[16133]: 2020-05-28T03:44:25.284Z [INFO]  auth.handler: authentication successful, sending token to sinks
vault[16133]: 2020-05-28T03:44:25.285Z [INFO]  auth.handler: starting renewal process
vault[16133]: 2020-05-28T03:44:25.289Z [INFO]  sink.file: token written: path=/etc/token
vault[16133]: 2020-05-28T03:44:25.304Z [INFO]  auth.handler: renewed auth token

it is clearly visible that the template server had stopped with template.server: template server stopped error, and auto-auth left as is

To Reproduce

agent-config.hcl

vault {
  address = "http://127.0.0.1:8200"
}

auto_auth {
  method {
    type      = "approle"
    config = {
      role_id_file_path = "/etc/role_id"
      secret_id_file_path = "/etc/secret_id"
    }
  }

  sink {
    type = "file"
    config = {
      path = "/etc/token"
    }
  }
}

template {
  source = "template.yml.tpl"
  destination = "template.yml"
  command = "script.sh"
}

script.sh

#!/usr/bin/env bash
echo "Sleep for 60s"
sleep 60s

I understand Vault Agent is not a replacement for a process monitoring, but proper error handling is expected

command (object: optional) - This is the optional command to run when the template is rendered. The command will only run if the resulting template changes. The command must return within 30s (configurable), and it must have a successful exit code. Vault Agent is not a replacement for a process monitor or init system.

Expected behavior

Exit with failure

Environment:

• Vault Server Version (retrieve with vault status): unsealed, initialized
• Vault CLI Version (retrieve with vault version): 1.4.2
• Server Operating System/Architecture: Ubuntu 18.04 amd64

Vault server configuration file(s):

docker run --cap-add=IPC_LOCK -p 8200:8200 -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' vault

Additional context

vault-agent being executing as systemd service