Terraform AWS provider unable to assume role using profile that assumes a role itself

[rekahsoft]

When using a a chain of aws cli profiles, one of which assumes a role, the aws provider fails to assume roles, as there are no credentials in ~/.aws/credentials for the corresponding profile. That is, given 2 profiles, A and R where:

• A is an IAM user and thus credentials for this profile exist within~/.aws/credentials
• R is a role assumed using the profile A. Note this means there are no credentials available for this profile in ~/.aws/credentials

Finally, there exists a role T which can be assumed by R.

When using terraform with the profile R, the aws provider is unable to assume role T. However, when using the awscli, this works with the following configuration:

[profile A]
region=<region>

[profile R]
source_profile=A
role_arn=arn:aws:iam::xxxxxxxxxxxx:role/Role-A

[profile T]
source_profile=R
role_arn=arn:aws:iam::xxxxxxxxxxxx:role/Role-T

All of the following calls succeed and use the correct role/identity, implying that the A profile can assume the role arn:aws:iam::xxxxxxxxxxxx:role/Role-A via the profile R which can then assume the role arn:aws:iam::xxxxxxxxxxxx:role/Role-T via the profile T.

aws --profile A sts get-caller-identity
aws --profile R sts get-caller-identity
aws --profile T sts get-caller-identity

This issue can be worked around by using the profile A after allowing it to assume the role T, however this greatly increases our maintenance overhead and is not acceptable.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.11
+ provider.aws v2.3.0

Your version of Terraform is out of date! The latest version
is 0.11.13. You can update by downloading from www.terraform.io/downloads.html

Affected Resource(s)

Unable to provision resources as role cannot be assumed by the aws provider.

Terraform Configuration Files

variable "region" {
  default = "us-west-2"
}

variable "cluster_master_role" {
  default = "arn:aws:iam::xxxxxxxxxxxx:role/Role-T"
}

provider "aws" {
  version = "~> 2.0"
  region = "${var.region}"
}

provider "aws" {
  version = "~> 2.0"
  alias   = "eks_master"
  region  = "${var.region}"

  assume_role = [{
    role_arn = "${var.cluster_master_role}"
  }]
}

Debug Output

I'm not providing debug output as it contains private information, however here are a few small snippets that seem relevant:

2019/03/22 23:12:04 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <Error>
    <Type>Sender</Type>
    <Code>ValidationError</Code>
    <Message>Must specify userName when calling with non-User credentials</Message>
  </Error>
  <RequestId>xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</RequestId>
</ErrorResponse>

2019-03-22T23:12:16.824-0400 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 2019/03/22 23:12:16 [INFO] assume_role configuration set: (ARN: "arn:aws:iam::xxxxxxxxxxxx:role/Role-T", SessionID: "", ExternalID: "", Policy: "")
2019-03-22T23:12:16.824-0400 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 2019/03/22 23:12:16 [INFO] Building AWS auth structure
2019-03-22T23:12:16.824-0400 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 2019/03/22 23:12:16 [INFO] Setting AWS metadata API timeout to 100ms
2019-03-22T23:12:17.494-0400 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 2019/03/22 23:12:17 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2019-03-22T23:12:17.630-0400 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 2019/03/22 23:12:17 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2019-03-22T23:12:17.631-0400 [DEBUG] plugin.terraform-provider-aws_v2.3.0_x4: 2019/03/22 23:12:17 [INFO] Attempting to AssumeRole arn:aws:iam::xxxxxxxxxxxx:role/Role-T (SessionName: "", ExternalId: "", Policy: "")
2019/03/22 23:12:17 [ERROR] root: eval: *terraform.EvalConfigProvider, err: No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Panic Output

N/A

Expected Behavior

Terraform aws provider assumes the role arn:aws:iam::xxxxxxxxxxxx:role/Role-T using the profile R.

Actual Behavior

Terraform fails to assume the role, failing with the following error message:

Error: Error running plan: 1 error(s) occurred:

* provider.aws.eks_master: No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Steps to Reproduce

When using terraform, the role with arn arn:aws:iam::xxxxxxxxxxxx:role/Role-T cannot be assumed by the provider:

export AWS_SDK_LOAD_CONFIG="true"
export AWS_PROFILE=R
terraform init

terraform plan

Important Factoids

N/A

References

• [bug] AWS provider is not loading role_arn from CLI config file under each profile #758

[andywirv]

Similar behaviour with latest version of terraform and the roles defined in ~/.aws/credentials and aws provider config specifying profile = rather than assume_role . Fine with aws cli but fails with error

provider.aws.dev: Error creating AWS session: SharedConfigAssumeRoleError: failed to load assume role for arn:aws:iam::[******]:role/Operations, source profile has no shared credentials

terraform --version
Terraform v0.11.13
+ provider.aws v2.3.0

[rekahsoft]

So I have determined why this is occurring. terraform-provider-aws uses the library aws-sdk-go-base which takes care of retrieving credentials for the provider. Within aws-sdk-go-base, the aws-go-sdk credentials package is used to obtain credentials for the provider via a ChainProvider. Now you would think that the EnvProvider used in the ChainProvider would behave the same as the aws-go-sdk session package, in that it would respect the environment variable AWS_SDK_LOAD_CONFIG, however it does not, and because of this, any profile that doesn't have credentials in the shared credentials file (by default ~/.aws/credentials) will not work with the terraform aws provider assume_role or profile options.

I'm happy to submit a PR to fix this, however feel that the PR would be better suited for the aws-go-sdk instead of the terraform-provider-aws or aws-sdk-go-base, as this issue will occur for any user of the aws-go-sdk credential package.

See:

• https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
• https://godoc.org/github.com/aws/aws-sdk-go/aws/credentials

[YakDriver]

I believe this is fixed with hashicorp/aws-sdk-go-base#5 PR. It sounds very similar.

[ianwsperber]

I'm encountering what I believe to be the same issue, using an AWS profile with a source_profile, eg

[profile me]
role_arn = ROLE_ARN
source_profile = foobar

I first noticed this when trying to add a provider which used an assume_role to access a resource in another AWS account, but have noticed this happens even when I do not provide the assume_role part - all I need to do is provide a second AWS provider to encounter the error

provider "aws" {
  region = "${var.region}"
}

provider "aws" {
  alias   = "my_alias"
  region = "${var.region}"
}

The error I see:

No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

[seddarj]

Same thing happening to me with a configuration similar to @ianwsperber's except instead of using 2 providers it happens with one provider and an S3 bucket as the backend. Works fine without the backend.

[bflad]

Please note that #8987, which was just merged and will release in version 2.16.0 of the Terraform AWS Provider later today, included this upstream fix aws/aws-sdk-go#2579, which is listed in the AWS Go SDK CHANGELOG as:

Adds support chaining assume role credentials from the shared config/credentials files. This change allows you to create an assume role chain of multiple levels of assumed IAM roles. The config profile the deepest in the chain must use static credentials, or credential_source. If the deepest profile doesn't have either of these the session will fail to load.

Hopefully this will help here. I also submitted this in Terraform Core to ensure the S3 Backend gets this update as well: hashicorp/terraform#21815

[bflad]

Hi folks 👋

This should be resolved in the S3 Backend as of Terraform version 0.12.3 and in the Terraform AWS Provider as of version 2.16.0.

I verified this locally via this configuration:

provider "aws" {
  region  = "us-east-2"
  version = "2.17.0"
}

data "aws_caller_identity" "current" {}

output "caller_arn" {
  value = "${data.aws_caller_identity.current.arn}"
}

This setup of AWS credentials and configuration files locally:

$ cat ~/.aws/config

[profile tf-acc-assume-role]
role_arn = arn:aws:iam::--OMITTED--:role/tf-acc-assume-role
source_profile = tf-acc

[profile tf-acc-assume-role-2]
role_arn = arn:aws:iam::--OMITTED--:role/tf-acc-assume-role-2
source_profile = tf-acc-assume-role

$ cat ~/.aws/credentials

[tf-acc]
aws_access_key_id = --OMITTED--
aws_secret_access_key = --OMITTED--

And running:

$ export AWS_SDK_LOAD_CONFIG=1
$ export AWS_PROFILE=tf-acc-assume-role-2
$ terraform apply
data.aws_caller_identity.current: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

caller_arn = arn:aws:sts::--OMITTED--:assumed-role/tf-acc-assume-role-2/1562206728701794000

---

For future bug reports or feature requests relating to provider authentication, even if they look similar to the error messages reported here, please submit new GitHub issues following the bug report and feature request issue templates for further triage. These types of issues tend to be very environment specific. 👍