OpenSearch: ValidationException on the domain `access_policies`

[samuel-phan]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.27.0

Affected Resource(s)

• aws_opensearch_domain

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  region = var.aws_region
}

data "aws_caller_identity" "current" {}

resource "aws_opensearch_domain" "example" {
  domain_name    = "example"
  engine_version = "OpenSearch_1.3"

  cluster_config {
    instance_type = "r4.large.search"
  }

  ebs_options {
    ebs_enabled = true
    volume_size = 50
  }

  access_policies = data.aws_iam_policy_document.opensearch.json

  tags = {
    Domain = var.domain
  }
}

data "aws_iam_policy_document" "opensearch" {
  statement {
    effect = "Allow"
    principals {
      identifiers = [
        "arn:aws:iam::<AWS_ACCOUNT_ID>:role/foo",
      ]
      type = "AWS"
    }
    actions   = ["es:*"]
    resources = ["arn:aws:es:${var.aws_region}:${data.aws_caller_identity.current.account_id}:domain/${var.domain}/*"]
  }
}

variable "aws_region" {
  default = "us-east-1"
}

variable "domain" {
  default = "TestDomain"
}

Debug Output

https://gist.github.com/samuel-phan/3a0e3df647bb8421445441d0e470f878

Panic Output

None.

Expected Behavior

The second terraform apply should change the OpenSearch domain's policy.

Actual Behavior

The second terraform apply fails.

Steps to Reproduce

1. terraform apply
2. Add a principal in the IAM policy:

   principals {
     identifiers = [
       "arn:aws:iam::<AWS_ACCOUNT_ID>:role/foo",
       "arn:aws:iam::<AWS_ACCOUNT_ID>:role/bar",
     ]
     type = "AWS"
   }

3. terraform apply fails with this error:

   aws_opensearch_domain.example: Modifying... [id=arn:aws:es:us-east-1:859573184227:domain/example]
   ╷
   │ Error: ValidationException: 1 validation error detected: Value '{
   │   "Version": "2012-10-17",
   │   "Statement": [
   │     {
   │       "Sid": "",
   │       "Effect": "Allow",
   │       "Action": "es:*",
   │       "Resource": "arn:aws:es:us-east-1:859573184227:domain/TestDomain/*",
   │       "Principal": {
   │         "AWS": [
   │           "arn:aws:iam::864830204113:role/sso/fed.power.user",
   │           "arn:aws:iam::864830204113:role/sso/fed.admin.user"
   │         ]
   │       }
   │     }
   │   ]
   │ }' at 'accessPolicies' failed to satisfy constraint: Member must satisfy regular expression pattern: .*
   │
   │   with aws_opensearch_domain.example,
   │   on main.tf line 7, in resource "aws_opensearch_domain" "example":
   │    7: resource "aws_opensearch_domain" "example" {
   

Important Factoids

None.

References

None.

[bsch150]

Our team is encountering this issue as well. We found a couple more pieces of info that may be helpful for triage:

• We can copy the failed access policy from our error and save it in the console with no errors
• We suspect that the validation issue is only happening on modify, not on create

[peter-lang-dealogic]

+1
Issue is happening only on modify (create is successful). Same access policy that is printed to console can be copy-pasted into console without issue.

Issue is only happening when using aws_iam_policy_document

Using jsonencode directly works fine:

access_policies = jsonencode({...})

[assaftir]

Looks like AWS backend is having trouble handling newline characters in the policy document.

[Senthil-Sivanath]

This issue happens on updation and not on creation, AWS Support engineer agreed on the issue has opened a ticket for the dev team.

Looks like JSON lines are concatenated on creation but not on updated.

[hong-jiang]

+1
Got the same issue on updating.

[jfxdev]

Looks like the parameter access_policies on aws_opensearch_domain doesn't validate the policy correctly.

A workaround for that is use the resource aws_opensearch_domain_policy instead of put the policy on domain resource.

resource "aws_opensearch_domain" "example" {
  access_policies = null
}

resource "aws_opensearch_domain_policy" "main" {
  access_policies = data.aws_iam_policy_document.opensearch.json
}

[ojaoferreira]

+1
Got the same issue on updating.

The workaround of @jfxdev it worked for me.

Thanks for share.

[xshot9011]

That's work.
Thanks for sharing @jfxdev