Vault no longer respects AWS_ROLE_ARN or AWS_WEB_IDENTITY_TOKEN_FILE for AWS KMS

[dqsully]

Describe the bug
AWS KMS seals no longer respect the AWS_ROLE_ARN or AWS_WEB_IDENTITY_TOKEN_FILE environment variables, which are required for assuming IAM roles via Kubernetes ServiceAccount tokens. Instead, Vault attempts to use the EC2 instance's IAM role (if available) to access the KMS key instead of using the Kubernetes ServiceAccount.

To Reproduce
Steps to reproduce the behavior:

1. Create an AWS EKS cluster
2. Connect the EKS cluster as an OIDC provider for AWS IAM
   
3. Create an AWS KMS key
4. Create an AWS IAM role with an assume-role policy authorizing that OIDC provider, and an inline policy authorizing that IAM role for actions on the KMS key you created
5. Deploy Vault with an "awskms" seal, setting only kms_key_id, and adding a ServiceAccount annotation eks.amazonaws.com/role-arn: <IAM role ARN>

Expected behavior
Vault should assume the IAM role configured in the Kubernetes ServiceAccount annotation and referenced by AWS_ROLE_ARN (injected by EKS because of the annotation), using the Kubernetes ServiceAccount token file referenced by AWS_WEB_IDENTITY_TOKEN_FILE (also injected by EKS) for authentication with AWS.

Environment:

• Vault Server Version: 1.14.0
• Server Operating System/Architecture: running on EKS 1.24, deployed using Vault's official Helm chart v0.25.0

Vault server configuration file(s):

ui = true
listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  x_forwarded_for_authorized_addrs = ["####", "####"]
  x_forwarded_for_reject_not_present = false
}
storage "consul" {
  path = "vault"
  address = "HOST_IP:8500"
}

service_registration "kubernetes" {}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "######"
}

Additional context
There is an easy workaround for this bug, which is to set role_arn and web_identity_token_file in the seal settings like so:

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "######"

  # Fixes for Vault 1.14+
  role_arn = "arn:aws:iam::<account ID>:role/<role name>"
  web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
}

Also, as far as I could trace it, the issue seems to come from this list of approved(?) environment variables for the AWS KMS wrapper?: 254d8f8#diff-8669cb5f3518deb7d1841c405e7e8b222348751cf85f81e6077a1184e9ed767dR15-R23

Hopefully the fix is as easy as adding AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN to that list.

[ubajze]

We are running into a similar issue, but unfortunatelly the workaround is not working for us.

I still get:

vault Error parsing Seal configuration: error fetching AWS KMS wrapping key information: NoCredentialProviders: no valid providers in chain. Deprecated.
vault     For verbose messaging see aws.Config.CredentialsChainVerboseErrors

This is how our seal configuration looks like:

    # Auto unseal is enabled using the AWS KMS service
    seal "awskms" {
      region = "us-east-2"
      kms_key_id = "KMS_KEY_ID"
      web_identity_token_file = "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
    }

[dqsully]

@ubajze I apologize, I was in a bit of a rush to solve this issue on our own side, and I missed that apparently the AWS_ROLE_ARN environment variable is also not being recognized, so you have to set the role_arn setting in the seal too. I did that on my own end just in case, but I assumed that it wasn't important. I'll update my original comment with that additional info.

[ubajze]

@dqsully it is working after adding the role_arn. Thank you for the update.

[dqsully]

@hsimon-hashicorp I don't know the conventions here but would this be better labeled under core/seal since that's the component that's affected? It's unrelated to the AWS secrets backend or the OIDC auth method, and while my example runs on Kubernetes with the Helm chart, this issue isn't unique to Kubernetes.

[igor-nikiforov]

We have the same issue after upgrade to 1.14.0. This config no longer works:

seal "awskms" {
    region     = "${data.aws_region.current.name}"
    kms_key_id = "${module.aws_kms_key.key_id}"
}

[tolleiv]

This seems to be also relevant for the s3 storage backend which doesn't support the web_identity_token_file and role_arn configuration.