aws secret backend (STS) + terraform is not working for AWS China

[jvanan24]

When trying to use datasource "vault_aws_access_credentials" for an AWS China (north-1) account, always the same error :

[ERROR] <root>: eval: *terraform.EvalSequence, err: InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: XXXXXX

Terraform code:

provider vault {}

data "vault_aws_access_credentials" "aws_creds" {
	backend  = CHINA_ACCOUNT
        role    = "MyRole"
        type    = "sts"
}

Vault config

vault read aws/CHINA_ACCOUNT/config/root

Key             Value
---             -----
access_key      XXXXXX
iam_endpoint    https://iam.cn-north-1.amazonaws.com.cn
max_retries     -1
region          cn-north-1
sts_endpoint    https://sts.cn-north-1.amazonaws.com.cn

vault read aws/CHINA_ACCOUNT/roles/MyRole

Key                         Value
---                         -----
credential_type             assumed_role
default_sts_ttl             1h
max_sts_ttl                 3h
permissions_boundary_arn    n/a
policy_arns                 <nil>
policy_document             n/a
role_arns                   [arn:aws-cn:iam::XXXXXX:role/MyRole]
user_path                   n/a

To reproduce just configure aws secret backend for a China account with STS role,

vault secrets enable -path aws/CHINA_ACCOUNT aws
vault write aws/CHINA_ACCOUNT/config/root access_key=XXX secret_key="XXX" region=cn-north-1 sts_endpoint=https://sts.cn-north-1.amazonaws.com.cn iam_endpoint=https://iam.cn-north-1.amazonaws.com.cn
vault write aws/CHINA_ACCOUNT/roles/MyRole role_arns=arn:aws-cn:iam::XXXXXXX:role/MyRolee credential_type=assumed_role

then "terraform init" & "terraform plan"

Vault should be able to generate credentials and pass them to terraform

Letting default parameter for endpoints leads to same error

Problem only occurs with a China account, EU, US ... accounts are OK

By setting TF_LOG=1 I can see the credentials are correctly created and they are usable (through aws cli for example).
Generating credentials from vault command line or from vault UI is working well too.
The only error is with "token validation".

2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: HTTP/1.1 200 OK
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: Content-Length: 745
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: Cache-Control: no-store
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: Connection: keep-alive
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: Content-Type: application/json
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: Date: Tue, 12 May 2020 13:21:26 GMT
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: {
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "request_id": "XXXXX",
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "lease_id": "aws/CHINA_ACCOUNT/sts/MyRole/XXXX",
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "renewable": false,
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "lease_duration": 3600,
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "data": {
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:   "access_key": "XXXXX",
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:   "secret_key": "XXXXXX",
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:   "security_token": "XXXXXX"
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  },
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "wrap_info": null,
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "warnings": null,
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:  "auth": null
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: }
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4:
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: -----------------------------------------------------
2020-05-12T15:21:26.308+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: 2020/05/12 15:21:26 [DEBUG] Read "aws/XXXX" from Vault
2020-05-12T15:21:26.309+0200 [DEBUG] plugin.terraform-provider-vault_v2.10.0_x4: 2020/05/12 15:21:26 [DEBUG] Checking if AWS sts token "aws/XXXXX/sts/MyRole/XXXXXXXXXXXXX" is valid
2020/05/12 15:21:26 [ERROR] <root>: eval: *terraform.EvalReadData, err: InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: XXXXXXX

Environment:
Vault Server v1.4.1
Vault Client v1.3.0
Terraform v0.12.24

[sidewinder12s]

This probably belongs on the terraform provider repo for Vault.

[jvanan24]

Fixed with : hashicorp/terraform-provider-vault#841