builder package not parameterized for aws-gov-us

[kbrian187]

When attempting to create nodegroups in an AWS GOV environment, nodegroup creation fails due to trying to use invalid aws ARNs rather than aws-gov-us ARNs. The issue seems to be in the builder package. The volume and snapshot ARNs are hardcoded to be aws rather than being parameterized.
https://github.com/weaveworks/eksctl/blob/8d285a825adc9db3c9401ba2eac290f9cfd6254c/pkg/cfn/builder/statement.go#L341-L342

What happened?
During nodegroup creation, CloudFormation stack fails due to incorrect ARNs.

AWS::IAM::Policy/PolicyEBS: CREATE_FAILED – "Partition \"aws\" is not valid for resource \"arn:aws:ec2:*:*:volume/*\". (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: ********-****-****-****-********; Proxy: null)"

How to reproduce it?
From an EC2 inside and aws-gov-* account, create a config file similar to the following:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: clusterName
  region: us-gov-west-1
iam:
  serviceRoleARN: "arn:aws-us-gov:iam::***************:role/ddog-eks-cluster-role"
  withOIDC: true


vpc:
  clusterEndpoints:
    privateAccess: true
    publicAccess: false
  id: vpc-***************
  subnets:
    private:
      us-gov-west-1a: { id: subnet-*************** }
      us-gov-west-1b: { id: subnet-*************** }
      us-gov-west-1c: { id: subnet-*************** }
  securityGroup: "sg-0547a3be767ec181b"

managedNodeGroups:
  - name: nodeGroupName
    labels: { role: agent }
    instanceType: t3.large
    desiredCapacity: 3
    privateNetworking: true
    iam:
      withAddonPolicies:
        cloudWatch: true
        autoScaler: true
        certManager: true
        imageBuilder: true
        ebs: true
    ssh:
      publicKeyName: ************************************
      allow: true
    securityGroups:
      attachIDs: ["sg-******************", "sg-*****************"]

addons:
- name: vpc-cni
  version: 1.10.1
  attachPolicyARNs:
  - "arn:aws-us-gov:iam::aws:policy/AmazonEKS_CNI_Policy"

Use eksctl to create the cluster using the config file:
eksctl create cluster -f /opt/cluster.yaml --cfn-disable-rollback --kubeconfig $HOME/.kube/config.eks --verbose 4

Logs

eksctl create cluster -f /opt/cluster.yaml --cfn-disable-rollback --kubeconfig $HOME/.kube/config.eks --verbose 4
2021-12-16 16:53:24 [▶]  role ARN for the current session is "arn:aws-us-gov:iam::************:user/Administrator"
2021-12-16 16:53:24 [ℹ]  eksctl version 0.75.0
2021-12-16 16:53:24 [ℹ]  using region us-gov-west-1
2021-12-16 16:53:24 [!]  warning, having public access disallowed will subsequently interfere with some features of eksctl. This will require running subsequent eksctl (and Kubernetes) commands/API calls from within the VPC.  Running these in the VPC requires making updates to some AWS resources.  See: https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html#private-access for more details
2021-12-16 16:53:26 [v]  using existing VPC (vpc-*****************) and subnets (private:map[us-gov-west-1a:{subnet-***************** us-gov-west-1a *.*.*.*/27} us-gov-west-1b:{subnet-***************** us-gov-west-1b *.*.*.*/27} us-gov-west-1c:{subnet-***************** us-gov-west-1c *.*.*.*/27}] public:map[])
2021-12-16 16:53:26 [!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
2021-12-16 16:53:26 [ℹ]  nodegroup "nodeGroupName" will use "" [AmazonLinux2/1.21]
2021-12-16 16:53:26 [ℹ]  using EC2 key pair "************"
2021-12-16 16:53:26 [ℹ]  using Kubernetes version 1.21
2021-12-16 16:53:26 [ℹ]  creating EKS cluster "clusterName" in "us-gov-west-1" region with managed nodes
2021-12-16 16:53:26 [▶]  cfg.json = \
{
    "kind": "ClusterConfig",
    "apiVersion": "eksctl.io/v1alpha5",
    "metadata": {
        "name": "clusterName",
        "region": "us-gov-west-1",
        "version": "1.21"
    },
    "iam": {
        "serviceRoleARN": "arn:aws-us-gov:iam::************:role/ddog-eks-cluster-role",
        "withOIDC": true,
        "vpcResourceControllerPolicy": true
    },
    "vpc": {
        "id": "vpc-*****************",
        "cidr": "*.*.*.*/23",
        "securityGroup": "sg-*****************",
        "subnets": {
            "private": {
                "us-gov-west-1a": {
                    "id": "subnet-*****************",
                    "az": "us-gov-west-1a",
                    "cidr": "*.*.*.*/27"
                },
                "us-gov-west-1b": {
                    "id": "subnet-*****************",
                    "az": "us-gov-west-1b",
                    "cidr": "*.*.*.*/27"
                },
                "us-gov-west-1c": {
                    "id": "subnet-*****************",
                    "az": "us-gov-west-1c",
                    "cidr": "*.*.*.*/27"
                }
            }
        },
        "manageSharedNodeSecurityGroupRules": true,
        "nat": {
            "gateway": "Disable"
        },
        "clusterEndpoints": {
            "privateAccess": true,
            "publicAccess": false
        }
    },
    "addons": [
        {
            "name": "vpc-cni",
            "version": "1.10.1",
            "attachPolicyARNs": [
                "arn:aws-us-gov:iam::aws:policy/AmazonEKS_CNI_Policy"
            ],
            "wellKnownPolicies": {
                "imageBuilder": false,
                "autoScaler": false,
                "awsLoadBalancerController": false,
                "externalDNS": false,
                "certManager": false,
                "ebsCSIController": false,
                "efsCSIController": false
            }
        }
    ],
    "privateCluster": {
        "enabled": false,
        "skipEndpointCreation": false
    },
    "managedNodeGroups": [
        {
            "name": "nodeGroupName",
            "amiFamily": "AmazonLinux2",
            "instanceType": "t3.large",
            "desiredCapacity": 3,
            "minSize": 3,
            "maxSize": 3,
            "volumeSize": 80,
            "ssh": {
                "allow": true,
                "publicKeyName": "**************"
            },
            "labels": {
                "alpha.eksctl.io/cluster-name": "clusterName",
                "alpha.eksctl.io/nodegroup-name": "nodeGroupName",
                "role": "agent"
            },
            "privateNetworking": true,
            "tags": {
                "alpha.eksctl.io/nodegroup-name": "nodeGroupName",
                "alpha.eksctl.io/nodegroup-type": "managed"
            },
            "iam": {
                "withAddonPolicies": {
                    "imageBuilder": true,
                    "autoScaler": true,
                    "externalDNS": true,
                    "certManager": true,
                    "appMesh": null,
                    "appMeshPreview": null,
                    "ebs": true,
                    "fsx": false,
                    "efs": false,
                    "albIngress": false,
                    "xRay": false,
                    "cloudWatch": true
                }
            },
            "securityGroups": {
                "attachIDs": [
                    "sg-**********",
                    "sg-**********"
                ],
                "withShared": null,
                "withLocal": null
            },
            "volumeType": "gp3",
            "volumeIOPS": 3000,
            "volumeThroughput": 125,
            "disableIMDSv1": false,
            "disablePodIMDS": false,
            "instanceSelector": {},
            "releaseVersion": ""
        }
    ],
    "availabilityZones": [
        "us-gov-west-1b",
        "us-gov-west-1c",
        "us-gov-west-1a"
    ]
}

2021-12-16 16:53:26 [ℹ]  1 nodegroup (nodeGroupName) was included (based on the include/exclude rules)
2021-12-16 16:53:26 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2021-12-16 16:53:26 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2021-12-16 16:53:26 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-gov-west-1 --cluster=clusterName'
2021-12-16 16:53:26 [ℹ]  CloudWatch logging will not be enabled for cluster "clusterName" in "us-gov-west-1"
2021-12-16 16:53:26 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=us-gov-west-1 --cluster=clusterName'
2021-12-16 16:53:26 [ℹ]  Kubernetes API endpoint access will use provided values {publicAccess=false, privateAccess=true} for cluster "clusterName" in "us-gov-west-1"
2021-12-16 16:53:26 [ℹ]
2 sequential tasks: { create cluster control plane "clusterName",
    2 sequential sub-tasks: {
        6 sequential sub-tasks: {
            wait for control plane to become ready,
            update cluster VPC endpoint access configuration,
            associate IAM OIDC provider,
            no tasks,
            restart daemonset "kube-system/aws-node",
            1 task: { create addons },
        },
        create managed nodegroup "nodeGroupName",
    }
}
2021-12-16 16:53:26 [▶]  started task: create cluster control plane "clusterName"
2021-12-16 16:53:26 [ℹ]  building cluster stack "eksctl-clusterName-cluster"
2021-12-16 16:53:26 [▶]  CreateStackInput = {
  DisableRollback: true,
  StackName: "eksctl-clusterName-cluster",
  Tags: [{
      Key: "alpha.eksctl.io/cluster-name",
      Value: "clusterName"
    },{
      Key: "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
      Value: "clusterName"
    },{
      Key: "alpha.eksctl.io/eksctl-version",
      Value: "0.75.0"
    }],
  TemplateBody: "{\n  \"AWSTemplateFormatVersion\": \"2010-09-09\",\n  \"Description\": \"EKS cluster (dedicated VPC: false, dedicated IAM: false) [created and managed by eksctl]\",\n  \"Mappings\": {\n    \"ServicePrincipalPartitionMap\": {\n      \"aws\": {\n        \"EC2\": \"ec2.amazonaws.com\",\n        \"EKS\": \"eks.amazonaws.com\",\n        \"EKSFargatePods\": \"eks-fargate-pods.amazonaws.com\"\n      },\n      \"aws-cn\": {\n        \"EC2\": \"ec2.amazonaws.com.cn\",\n        \"EKS\": \"eks.amazonaws.com\",\n        \"EKSFargatePods\": \"eks-fargate-pods.amazonaws.com\"\n      },\n      \"aws-us-gov\": {\n        \"EC2\": \"ec2.amazonaws.com\",\n        \"EKS\": \"eks.amazonaws.com\",\n        \"EKSFargatePods\": \"eks-fargate-pods.amazonaws.com\"\n      }\n    }\n  },\n  \"Resources\": {\n    \"ClusterSharedNodeSecurityGroup\": {\n      \"Type\": \"AWS::EC2::SecurityGroup\",\n      \"Properties\": {\n        \"GroupDescription\": \"Communication between all nodes in the cluster\",\n        \"Tags\": [\n          {\n            \"Key\": \"Name\",\n            \"Value\": {\n              \"Fn::Sub\": \"${AWS::StackName}/ClusterSharedNodeSecurityGroup\"\n            }\n          }\n        ],\n        \"VpcId\": \"vpc-*****************\"\n      }\n    },\n    \"ControlPlane\": {\n      \"Type\": \"AWS::EKS::Cluster\",\n      \"Properties\": {\n        \"Name\": \"clusterName\",\n        \"ResourcesVpcConfig\": {\n          \"SecurityGroupIds\": [\n            \"sg-*****************\"\n          ],\n          \"SubnetIds\": [\n            \"subnet-*****************\",\n            \"subnet-*****************\",\n            \"subnet-*****************\"\n          ]\n        },\n        \"RoleArn\": \"arn:aws-us-gov:iam::************:role/ddog-eks-cluster-role\",\n        \"Version\": \"1.21\"\n      }\n    },\n    \"IngressDefaultClusterToNodeSG\": {\n      \"Type\": \"AWS::EC2::SecurityGroupIngress\",\n      \"Properties\": {\n        \"Description\": \"Allow managed and unmanaged nodes to communicate with each other (all ports)\",\n        \"FromPort\": 0,\n        \"GroupId\": {\n          \"Ref\": \"ClusterSharedNodeSecurityGroup\"\n        },\n        \"IpProtocol\": \"-1\",\n        \"SourceSecurityGroupId\": {\n          \"Fn::GetAtt\": [\n            \"ControlPlane\",\n            \"ClusterSecurityGroupId\"\n          ]\n        },\n        \"ToPort\": 65535\n      }\n    },\n    \"IngressInterNodeGroupSG\": {\n      \"Type\": \"AWS::EC2::SecurityGroupIngress\",\n      \"Properties\": {\n        \"Description\": \"Allow nodes to communicate with each other (all ports)\",\n        \"FromPort\": 0,\n        \"GroupId\": {\n          \"Ref\": \"ClusterSharedNodeSecurityGroup\"\n        },\n        \"IpProtocol\": \"-1\",\n        \"SourceSecurityGroupId\": {\n          \"Ref\": \"ClusterSharedNodeSecurityGroup\"\n        },\n        \"ToPort\": 65535\n      }\n    },\n    \"IngressNodeToDefaultClusterSG\": {\n      \"Type\": \"AWS::EC2::SecurityGroupIngress\",\n      \"Properties\": {\n        \"Description\": \"Allow unmanaged nodes to communicate with control plane (all ports)\",\n        \"FromPort\": 0,\n        \"GroupId\": {\n          \"Fn::GetAtt\": [\n            \"ControlPlane\",\n            \"ClusterSecurityGroupId\"\n          ]\n        },\n        \"IpProtocol\": \"-1\",\n        \"SourceSecurityGroupId\": {\n          \"Ref\": \"ClusterSharedNodeSecurityGroup\"\n        },\n        \"ToPort\": 65535\n      }\n    }\n  },\n  \"Outputs\": {\n    \"ARN\": {\n      \"Value\": {\n        \"Fn::GetAtt\": [\n          \"ControlPlane\",\n          \"Arn\"\n        ]\n      },\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::ARN\"\n        }\n      }\n    },\n    \"CertificateAuthorityData\": {\n      \"Value\": {\n        \"Fn::GetAtt\": [\n          \"ControlPlane\",\n          \"CertificateAuthorityData\"\n        ]\n      }\n    },\n    \"ClusterSecurityGroupId\": {\n      \"Value\": {\n        \"Fn::GetAtt\": [\n          \"ControlPlane\",\n          \"ClusterSecurityGroupId\"\n        ]\n      },\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::ClusterSecurityGroupId\"\n        }\n      }\n    },\n    \"ClusterStackName\": {\n      \"Value\": {\n        \"Ref\": \"AWS::StackName\"\n      }\n    },\n    \"Endpoint\": {\n      \"Value\": {\n        \"Fn::GetAtt\": [\n          \"ControlPlane\",\n          \"Endpoint\"\n        ]\n      },\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::Endpoint\"\n        }\n      }\n    },\n    \"FeatureNATMode\": {\n      \"Value\": \"Disable\"\n    },\n    \"SecurityGroup\": {\n      \"Value\": \"sg-*****************\",\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::SecurityGroup\"\n        }\n      }\n    },\n    \"ServiceRoleARN\": {\n      \"Value\": \"arn:aws-us-gov:iam::************:role/ddog-eks-cluster-role\",\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::ServiceRoleARN\"\n        }\n      }\n    },\n    \"SharedNodeSecurityGroup\": {\n      \"Value\": {\n        \"Ref\": \"ClusterSharedNodeSecurityGroup\"\n      },\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::SharedNodeSecurityGroup\"\n        }\n      }\n    },\n    \"SubnetsPrivate\": {\n      \"Value\": {\n        \"Fn::Join\": [\n          \",\",\n          [\n            \"subnet-*****************\",\n            \"subnet-*****************\",\n            \"subnet-*****************\"\n          ]\n        ]\n      },\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::SubnetsPrivate\"\n        }\n      }\n    },\n    \"VPC\": {\n      \"Value\": \"vpc-*****************\",\n      \"Export\": {\n        \"Name\": {\n          \"Fn::Sub\": \"${AWS::StackName}::VPC\"\n        }\n      }\n    }\n  }\n}"
}
2021-12-16 16:53:28 [ℹ]  deploying stack "eksctl-clusterName-cluster"
2021-12-16 16:53:58 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 16:54:28 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 16:55:30 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 16:56:31 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 16:57:32 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 16:58:34 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 16:59:36 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 17:00:37 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 17:01:39 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 17:02:40 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 17:03:42 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 17:04:43 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-cluster"
2021-12-16 17:04:45 [▶]  processing stack outputs
2021-12-16 17:04:46 [▶]  completed task: create cluster control plane "clusterName"
2021-12-16 17:04:46 [▶]  started task:
    2 sequential sub-tasks: {
        6 sequential sub-tasks: {
            wait for control plane to become ready,
            update cluster VPC endpoint access configuration,
            associate IAM OIDC provider,
            no tasks,
            restart daemonset "kube-system/aws-node",
            1 task: { create addons },
        },
        create managed nodegroup "nodeGroupName",
    }

2021-12-16 17:04:46 [▶]  started task:
    6 sequential sub-tasks: {
        wait for control plane to become ready,
        update cluster VPC endpoint access configuration,
        associate IAM OIDC provider,
        no tasks,
        restart daemonset "kube-system/aws-node",
        1 task: { create addons },
    }

2021-12-16 17:04:46 [▶]  started task: wait for control plane to become ready
2021-12-16 17:06:49 [▶]  cluster = {
  Arn: "arn:aws-us-gov:eks:us-gov-west-1:************:cluster/clusterName",
  CertificateAuthority: {
    Data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1USXhOakUyTlRreE5sb1hEVE14TVRJeE5ERTJOVGt4Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSmEvCnBZb1NiL05rVVl5T0R2RUp0SlMzeGU0MEg5cnRmUHJyWmI1SXBoY0Q4ZzhHTE9GYWlKSnZqbEFpVHdUZHUxQUUKVjBVcSszVmZiWjFYY3Zsb3M4U2Q2dWI2a0JCclVDMnA5UTJFaDZIelRLQ2xLVng2QzBaQUtjWVBJZ2ZZbFF0MApWWERIbk5TYVlabm9EeDYzVERadktwcHh2K09rRzdyUzZJMkpDSFZTVDd5WlZXT1FhV0ZuTzd3ckhRY3R4S3p6Ck5JOTZYUzZzUTZidzNsRW10U2lCQndDdm8yZW5ySkR0VCtYV295NFdhc3dVa2tTaUI3bXhFdEpjanhTaDE5UUcKRjloWE40S0s4OWx3blBzMVJLM3p5Q25WRjljY3c5Tnlsb2dBcTk1WkoyVTQ1aDVkeWF6WGlDeHlSYlFkek5HUQpZT3IzL2x5SFJSV1VWa2lzU2NzQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZJYTRZSDI1dnZhbkN3U21EenErdHBxaE90WmNNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFBVk9SLzBUN1pUTmlIK2JDaWdwSFNJbDdYc0p5S2JRWG8wMDZiNUJKb1ZLblRMUWJNOQpnK0NXd1NiT0NxNVNyYmU5ajdhR2lCeTBFVWFTblI2dnQ2YlBUVC9VU0VaQ2VHZDZ0VEx4YUlERHZGUUlVS0pGCmxIdkFMdjhaTVpsc1N2MTN1anVwMU42MWUxNTFsWEVaU2x0ZFFxTG5QZlZ1MEJSWUNWOHJackZuUXBzdVNvOTgKVjg1bEQ5WCtVeFl3UjZFcXBzL05YVUF6N2thTEtFay8xOTJhTjdIdUNVUVlZQ0ZlK1hlRlp3K2Y2QkZQK1B6ZQpiYmRRRjA5RkdUdzE3cG0xQW9XSU5MdEcrckJyb01wNUtvS3doMWV5OUhYZ1dJK2VzRW02ZDFlV2tWaEI0K1hFCjBFVGIvL0RYRVdsUHRHalppR1JnSFZBWEZoMTNkdytjUVNBTQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
  },
  CreatedAt: 2021-12-16 16:53:33.622 +0000 UTC,
  Endpoint: "https://*******************",
  Identity: {
    Oidc: {
      Issuer: "https://*******************"
    }
  },
  KubernetesNetworkConfig: {
    ServiceIpv4Cidr: "*.*.*.*/16"
  },
  Logging: {
    ClusterLogging: [{
        Enabled: false,
        Types: [
          "api",
          "audit",
          "authenticator",
          "controllerManager",
          "scheduler"
        ]
      }]
  },
  Name: "clusterName",
  PlatformVersion: "eks.4",
  ResourcesVpcConfig: {
    ClusterSecurityGroupId: "sg-**************",
    EndpointPrivateAccess: false,
    EndpointPublicAccess: true,
    PublicAccessCidrs: ["0.0.0.0/0"],
    SecurityGroupIds: ["sg-*****************"],
    SubnetIds: ["subnet-*****************","subnet-*****************","subnet-*****************"],
    VpcId: "vpc-*****************"
  },
  RoleArn: "arn:aws-us-gov:iam::************:role/ddog-eks-cluster-role",
  Status: "ACTIVE",
  Tags: {
    aws:cloudformation:stack-name: "eksctl-clusterName-cluster",
    aws:cloudformation:logical-id: "ControlPlane",
    alpha.eksctl.io/cluster-name: "clusterName",
    aws:cloudformation:stack-id: "arn:aws-us-gov:cloudformation:us-gov-west-1:************:stack/eksctl-clusterName-cluster/b0e8a160-5e90-11ec-ba94-0a289af5c77a",
    alpha.eksctl.io/eksctl-version: "0.75.0",
    eksctl.cluster.k8s.io/v1alpha1/cluster-name: "clusterName"
  },
  Version: "1.21"
}
2021-12-16 17:06:49 [▶]  completed task: wait for control plane to become ready
2021-12-16 17:06:49 [▶]  started task: update cluster VPC endpoint access configuration
2021-12-16 17:06:50 [▶]  start waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:06:50 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:07:07 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:07:24 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:07:44 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:08:01 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:08:21 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:08:40 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:08:59 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:09:17 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:09:34 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:09:52 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:10:08 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:10:26 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:10:42 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:10:59 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:11:16 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:11:32 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:11:50 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:12:07 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:12:26 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:12:44 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:13:03 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:13:20 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:13:40 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:13:56 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:14:12 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:14:29 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:14:46 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:15:02 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:15:23 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:15:39 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:15:39 [▶]  done after 8m48.780277434s of waiting for requested "EndpointAccessUpdate" in cluster "clusterName" to succeed
2021-12-16 17:15:39 [▶]  completed task: update cluster VPC endpoint access configuration
2021-12-16 17:15:39 [▶]  started task: associate IAM OIDC provider
2021-12-16 17:15:39 [▶]  cluster = {
  Arn: "arn:aws-us-gov:eks:us-gov-west-1:************:cluster/clusterName",
  CertificateAuthority: {
    Data: "*"
  },
  CreatedAt: 2021-12-16 16:53:33.622 +0000 UTC,
  Endpoint: "https://*******************",
  Identity: {
    Oidc: {
      Issuer: "https://*******************"
    }
  },
  KubernetesNetworkConfig: {
    ServiceIpv4Cidr: "*.*.*.*/16"
  },
  Logging: {
    ClusterLogging: [{
        Enabled: false,
        Types: [
          "api",
          "audit",
          "authenticator",
          "controllerManager",
          "scheduler"
        ]
      }]
  },
  Name: "clusterName",
  PlatformVersion: "eks.4",
  ResourcesVpcConfig: {
    ClusterSecurityGroupId: "sg-**************",
    EndpointPrivateAccess: true,
    EndpointPublicAccess: false,
    PublicAccessCidrs: ["0.0.0.0/0"],
    SecurityGroupIds: ["sg-*****************"],
    SubnetIds: ["subnet-*****************","subnet-*****************","subnet-*****************"],
    VpcId: "vpc-*****************"
  },
  RoleArn: "arn:aws-us-gov:iam::************:role/ddog-eks-cluster-role",
  Status: "ACTIVE",
  Tags: {
    alpha.eksctl.io/eksctl-version: "0.75.0",
    eksctl.cluster.k8s.io/v1alpha1/cluster-name: "clusterName",
    aws:cloudformation:stack-name: "eksctl-clusterName-cluster",
    aws:cloudformation:logical-id: "ControlPlane",
    alpha.eksctl.io/cluster-name: "clusterName",
    aws:cloudformation:stack-id: "arn:aws-us-gov:cloudformation:us-gov-west-1:************:stack/eksctl-clusterName-cluster/b0e8a160-5e90-11ec-ba94-0a289af5c77a"
  },
  Version: "1.21"
}
2021-12-16 17:17:42 [▶]  completed task: associate IAM OIDC provider
2021-12-16 17:17:42 [▶]  started task: no tasks
2021-12-16 17:17:42 [▶]  no actual tasks
2021-12-16 17:17:42 [▶]  completed task: no tasks
2021-12-16 17:17:42 [▶]  started task: restart daemonset "kube-system/aws-node"
2021-12-16 17:17:42 [ℹ]  daemonset "kube-system/aws-node" restarted
2021-12-16 17:17:42 [▶]  completed task: restart daemonset "kube-system/aws-node"
2021-12-16 17:17:42 [▶]  started task: 1 task: { create addons }
2021-12-16 17:17:42 [▶]  started task: create addons
2021-12-16 17:19:44 [▶]  cluster = {
  Arn: "arn:aws-us-gov:eks:us-gov-west-1:************:cluster/clusterName",
  CertificateAuthority: {
    Data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1USXhOakUyTlRreE5sb1hEVE14TVRJeE5ERTJOVGt4Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSmEvCnBZb1NiL05rVVl5T0R2RUp0SlMzeGU0MEg5cnRmUHJyWmI1SXBoY0Q4ZzhHTE9GYWlKSnZqbEFpVHdUZHUxQUUKVjBVcSszVmZiWjFYY3Zsb3M4U2Q2dWI2a0JCclVDMnA5UTJFaDZIelRLQ2xLVng2QzBaQUtjWVBJZ2ZZbFF0MApWWERIbk5TYVlabm9EeDYzVERadktwcHh2K09rRzdyUzZJMkpDSFZTVDd5WlZXT1FhV0ZuTzd3ckhRY3R4S3p6Ck5JOTZYUzZzUTZidzNsRW10U2lCQndDdm8yZW5ySkR0VCtYV295NFdhc3dVa2tTaUI3bXhFdEpjanhTaDE5UUcKRjloWE40S0s4OWx3blBzMVJLM3p5Q25WRjljY3c5Tnlsb2dBcTk1WkoyVTQ1aDVkeWF6WGlDeHlSYlFkek5HUQpZT3IzL2x5SFJSV1VWa2lzU2NzQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZJYTRZSDI1dnZhbkN3U21EenErdHBxaE90WmNNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFBVk9SLzBUN1pUTmlIK2JDaWdwSFNJbDdYc0p5S2JRWG8wMDZiNUJKb1ZLblRMUWJNOQpnK0NXd1NiT0NxNVNyYmU5ajdhR2lCeTBFVWFTblI2dnQ2YlBUVC9VU0VaQ2VHZDZ0VEx4YUlERHZGUUlVS0pGCmxIdkFMdjhaTVpsc1N2MTN1anVwMU42MWUxNTFsWEVaU2x0ZFFxTG5QZlZ1MEJSWUNWOHJackZuUXBzdVNvOTgKVjg1bEQ5WCtVeFl3UjZFcXBzL05YVUF6N2thTEtFay8xOTJhTjdIdUNVUVlZQ0ZlK1hlRlp3K2Y2QkZQK1B6ZQpiYmRRRjA5RkdUdzE3cG0xQW9XSU5MdEcrckJyb01wNUtvS3doMWV5OUhYZ1dJK2VzRW02ZDFlV2tWaEI0K1hFCjBFVGIvL0RYRVdsUHRHalppR1JnSFZBWEZoMTNkdytjUVNBTQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
  },
  CreatedAt: 2021-12-16 16:53:33.622 +0000 UTC,
  Endpoint: "https://*******************",
  Identity: {
    Oidc: {
      Issuer: "https://*******************"
    }
  },
  KubernetesNetworkConfig: {
    ServiceIpv4Cidr: "*.*.*.*/16"
  },
  Logging: {
    ClusterLogging: [{
        Enabled: false,
        Types: [
          "api",
          "audit",
          "authenticator",
          "controllerManager",
          "scheduler"
        ]
      }]
  },
  Name: "clusterName",
  PlatformVersion: "eks.4",
  ResourcesVpcConfig: {
    ClusterSecurityGroupId: "sg-**************",
    EndpointPrivateAccess: true,
    EndpointPublicAccess: false,
    PublicAccessCidrs: ["0.0.0.0/0"],
    SecurityGroupIds: ["sg-*****************"],
    SubnetIds: ["subnet-*****************","subnet-*****************","subnet-*****************"],
    VpcId: "vpc-*****************"
  },
  RoleArn: "arn:aws-us-gov:iam::************:role/ddog-eks-cluster-role",
  Status: "ACTIVE",
  Tags: {
    aws:cloudformation:stack-name: "eksctl-clusterName-cluster",
    aws:cloudformation:logical-id: "ControlPlane",
    alpha.eksctl.io/cluster-name: "clusterName",
    aws:cloudformation:stack-id: "arn:aws-us-gov:cloudformation:us-gov-west-1:************:stack/eksctl-clusterName-cluster/b0e8a160-5e90-11ec-ba94-0a289af5c77a",
    alpha.eksctl.io/eksctl-version: "0.75.0",
    eksctl.cluster.k8s.io/v1alpha1/cluster-name: "clusterName"
  },
  Version: "1.21"
}
2021-12-16 17:19:46 [▶]  setting resolve conflicts to overwrite
2021-12-16 17:19:46 [▶]  addon: &{vpc-cni 1.10.1  [arn:aws-us-gov:iam::aws:policy/AmazonEKS_CNI_Policy] map[]  {false false false false false false false} map[] true}
2021-12-16 17:19:46 [▶]  found known service account location kube-system/aws-node
2021-12-16 17:19:46 [ℹ]  creating role using provided policies ARNs
2021-12-16 17:19:46 [▶]  service account location provided: kube-system/aws-node, adding sub condition
2021-12-16 17:19:46 [▶]  CreateStackInput = {
  Capabilities: ["CAPABILITY_IAM"],
  DisableRollback: true,
  StackName: "eksctl-clusterName-addon-vpc-cni",
  Tags: [
    {
      Key: "alpha.eksctl.io/cluster-name",
      Value: "clusterName"
    },
    {
      Key: "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
      Value: "clusterName"
    },
    {
      Key: "alpha.eksctl.io/eksctl-version",
      Value: "0.75.0"
    },
    {
      Key: "alpha.eksctl.io/addon-name",
      Value: "vpc-cni"
    }
  ],
  TemplateBody: "{\"AWSTemplateFormatVersion\":\"2010-09-09\",\"Description\":\"IAM role for \\\"vpc-cni\\\" [created and managed by eksctl]\",\"Resources\":{\"Role1\":{\"Type\":\"AWS::IAM::Role\",\"Properties\":{\"AssumeRolePolicyDocument\":{\"Statement\":[{\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"oidc.eks.us-gov-west-1.amazonaws.com/id/790F6B15C0620977F62AC37DD1922875:aud\":\"sts.amazonaws.com\",\"oidc.eks.us-gov-west-1.amazonaws.com/id/790F6B15C0620977F62AC37DD1922875:sub\":\"system:serviceaccount:kube-system:aws-node\"}},\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws-us-gov:iam::************:oidc-provider/oidc.eks.us-gov-west-1.amazonaws.com/id/790F6B15C0620977F62AC37DD1922875\"}}],\"Version\":\"2012-10-17\"},\"ManagedPolicyArns\":[\"arn:aws-us-gov:iam::aws:policy/AmazonEKS_CNI_Policy\"]}}},\"Outputs\":{\"Role1\":{\"Value\":{\"Fn::GetAtt\":\"Role1.Arn\"}}}}"
}
2021-12-16 17:19:47 [ℹ]  deploying stack "eksctl-clusterName-addon-vpc-cni"
2021-12-16 17:19:47 [▶]  start waiting for CloudFormation stack "eksctl-clusterName-addon-vpc-cni"
2021-12-16 17:19:47 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-addon-vpc-cni"
2021-12-16 17:20:04 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-addon-vpc-cni"
2021-12-16 17:20:24 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-addon-vpc-cni"
2021-12-16 17:20:25 [▶]  done after 37.113943118s of waiting for CloudFormation stack "eksctl-clusterName-addon-vpc-cni"
2021-12-16 17:20:25 [▶]  patching AWS node
2021-12-16 17:20:25 [▶]  no 'eksctl' managed field found
2021-12-16 17:20:25 [ℹ]  creating addon
2021-12-16 17:20:25 [▶]  EKS Create Addon output: {
  Addon: {
    AddonArn: "arn:aws-us-gov:eks:us-gov-west-1:************:addon/clusterName/vpc-cni/babee221-3281-7990-d86d-bce2792fbeb2",
    AddonName: "vpc-cni",
    AddonVersion: "v1.10.1-eksbuild.1",
    ClusterName: "clusterName",
    CreatedAt: 2021-12-16 17:20:25.51 +0000 UTC,
    Health: {
      Issues: []
    },
    ModifiedAt: 2021-12-16 17:20:25.529 +0000 UTC,
    ServiceAccountRoleArn: "arn:aws-us-gov:iam::************:role/eksctl-clusterName-addon-vpc-cni-Role1-1CNI0IJH6IJO6",
    Status: "CREATING",
    Tags: {

    }
  }
}
2021-12-16 17:20:25 [ℹ]  successfully created addon
2021-12-16 17:20:25 [▶]  completed task: create addons
2021-12-16 17:20:25 [▶]  completed task: 1 task: { create addons }
2021-12-16 17:20:25 [▶]  completed task:
    6 sequential sub-tasks: {
        wait for control plane to become ready,
        update cluster VPC endpoint access configuration,
        associate IAM OIDC provider,
        no tasks,
        restart daemonset "kube-system/aws-node",
        1 task: { create addons },
    }

2021-12-16 17:20:25 [▶]  started task: create managed nodegroup "nodeGroupName"
2021-12-16 17:20:25 [▶]  waiting for 1 parallel tasks to complete
2021-12-16 17:20:25 [▶]  started task: create managed nodegroup "nodeGroupName"
2021-12-16 17:20:25 [ℹ]  building managed nodegroup stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:20:25 [▶]  CreateStackInput = {
  Capabilities: ["CAPABILITY_IAM"],
  DisableRollback: true,
  StackName: "eksctl-clusterName-nodegroup-nodeGroupName",
  Tags: [
    {
      Key: "alpha.eksctl.io/cluster-name",
      Value: "clusterName"
    },
    {
      Key: "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
      Value: "clusterName"
    },
    {
      Key: "alpha.eksctl.io/eksctl-version",
      Value: "0.75.0"
    },
    {
      Key: "alpha.eksctl.io/nodegroup-name",
      Value: "nodeGroupName"
    },
    {
      Key: "alpha.eksctl.io/nodegroup-type",
      Value: "managed"
    }
  ],
  TemplateBody: "{\n  \"AWSTemplateFormatVersion\": \"2010-09-09\",\n  \"Description\": \"EKS Managed Nodes (SSH access: true) [created by eksctl]\",\n  \"Mappings\": {\n    \"ServicePrincipalPartitionMap\": {\n      \"aws\": {\n        \"EC2\": \"ec2.amazonaws.com\",\n        \"EKS\": \"eks.amazonaws.com\",\n        \"EKSFargatePods\": \"eks-fargate-pods.amazonaws.com\"\n      },\n      \"aws-cn\": {\n        \"EC2\": \"ec2.amazonaws.com.cn\",\n        \"EKS\": \"eks.amazonaws.com\",\n        \"EKSFargatePods\": \"eks-fargate-pods.amazonaws.com\"\n      },\n      \"aws-us-gov\": {\n        \"EC2\": \"ec2.amazonaws.com\",\n        \"EKS\": \"eks.amazonaws.com\",\n        \"EKSFargatePods\": \"eks-fargate-pods.amazonaws.com\"\n      }\n    }\n  },\n  \"Resources\": {\n    \"LaunchTemplate\": {\n      \"Type\": \"AWS::EC2::LaunchTemplate\",\n      \"Properties\": {\n        \"LaunchTemplateData\": {\n          \"BlockDeviceMappings\": [\n            {\n              \"DeviceName\": \"/dev/xvda\",\n              \"Ebs\": {\n                \"Iops\": 3000,\n                \"Throughput\": 125,\n                \"VolumeSize\": 80,\n                \"VolumeType\": \"gp3\"\n              }\n            }\n          ],\n          \"KeyName\": \"**************\",\n          \"MetadataOptions\": {\n            \"HttpPutResponseHopLimit\": 2,\n            \"HttpTokens\": \"optional\"\n          },\n          \"SecurityGroupIds\": [\n            {\n              \"Fn::ImportValue\": \"eksctl-clusterName-cluster::ClusterSecurityGroupId\"\n            },\n            \"sg-**********\",\n            \"sg-**********\",\n            {\n              \"Ref\": \"SSH\"\n            }\n          ],\n          \"TagSpecifications\": [\n            {\n              \"ResourceType\": \"instance\",\n              \"Tags\": [\n                {\n                  \"Key\": \"Name\",\n                  \"Value\": \"clusterName-nodeGroupName-Node\"\n                },\n                {\n                  \"Key\": \"alpha.eksctl.io/nodegroup-name\",\n                  \"Value\": \"nodeGroupName\"\n                },\n                {\n                  \"Key\": \"alpha.eksctl.io/nodegroup-type\",\n                  \"Value\": \"managed\"\n                }\n              ]\n            },\n            {\n              \"ResourceType\": \"volume\",\n              \"Tags\": [\n                {\n                  \"Key\": \"Name\",\n                  \"Value\": \"clusterName-nodeGroupName-Node\"\n                },\n                {\n                  \"Key\": \"alpha.eksctl.io/nodegroup-name\",\n                  \"Value\": \"nodeGroupName\"\n                },\n                {\n                  \"Key\": \"alpha.eksctl.io/nodegroup-type\",\n                  \"Value\": \"managed\"\n                }\n              ]\n            },\n            {\n              \"ResourceType\": \"network-interface\",\n              \"Tags\": [\n                {\n                  \"Key\": \"Name\",\n                  \"Value\": \"clusterName-nodeGroupName-Node\"\n                },\n                {\n                  \"Key\": \"alpha.eksctl.io/nodegroup-name\",\n                  \"Value\": \"nodeGroupName\"\n                },\n                {\n                  \"Key\": \"alpha.eksctl.io/nodegroup-type\",\n                  \"Value\": \"managed\"\n                }\n              ]\n            }\n          ]\n        },\n        \"LaunchTemplateName\": {\n          \"Fn::Sub\": \"${AWS::StackName}\"\n        }\n      }\n    },\n    \"ManagedNodeGroup\": {\n      \"Type\": \"AWS::EKS::Nodegroup\",\n      \"Properties\": {\n        \"AmiType\": \"AL2_x86_64\",\n        \"ClusterName\": \"clusterName\",\n        \"InstanceTypes\": [\n          \"t3.large\"\n        ],\n        \"Labels\": {\n          \"alpha.eksctl.io/cluster-name\": \"clusterName\",\n          \"alpha.eksctl.io/nodegroup-name\": \"nodeGroupName\",\n          \"role\": \"agent\"\n        },\n        \"LaunchTemplate\": {\n          \"Id\": {\n            \"Ref\": \"LaunchTemplate\"\n          }\n        },\n        \"NodeRole\": {\n          \"Fn::GetAtt\": [\n            \"NodeInstanceRole\",\n            \"Arn\"\n          ]\n        },\n        \"NodegroupName\": \"nodeGroupName\",\n        \"ScalingConfig\": {\n          \"DesiredSize\": 3,\n          \"MaxSize\": 3,\n          \"MinSize\": 3\n        },\n        \"Subnets\": {\n          \"Fn::Split\": [\n            \",\",\n            {\n              \"Fn::ImportValue\": \"eksctl-clusterName-cluster::SubnetsPrivate\"\n            }\n          ]\n        },\n        \"Tags\": {\n          \"alpha.eksctl.io/nodegroup-name\": \"nodeGroupName\",\n          \"alpha.eksctl.io/nodegroup-type\": \"managed\"\n        }\n      }\n    },\n    \"NodeInstanceRole\": {\n      \"Type\": \"AWS::IAM::Role\",\n      \"Properties\": {\n        \"AssumeRolePolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"sts:AssumeRole\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Principal\": {\n                \"Service\": [\n                  {\n                    \"Fn::FindInMap\": [\n                      \"ServicePrincipalPartitionMap\",\n                      {\n                        \"Ref\": \"AWS::Partition\"\n                      },\n                      \"EC2\"\n                    ]\n                  }\n                ]\n              }\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"ManagedPolicyArns\": [\n          {\n            \"Fn::Sub\": \"arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser\"\n          },\n          {\n            \"Fn::Sub\": \"arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly\"\n          },\n          {\n            \"Fn::Sub\": \"arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy\"\n          },\n          {\n            \"Fn::Sub\": \"arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore\"\n          },\n          {\n            \"Fn::Sub\": \"arn:${AWS::Partition}:iam::aws:policy/CloudWatchAgentServerPolicy\"\n          }\n        ],\n        \"Path\": \"/\",\n        \"Tags\": [\n          {\n            \"Key\": \"Name\",\n            \"Value\": {\n              \"Fn::Sub\": \"${AWS::StackName}/NodeInstanceRole\"\n            }\n          }\n        ]\n      }\n    },\n    \"PolicyAutoScaling\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"autoscaling:DescribeAutoScalingGroups\",\n                \"autoscaling:DescribeAutoScalingInstances\",\n                \"autoscaling:DescribeLaunchConfigurations\",\n                \"autoscaling:DescribeTags\",\n                \"autoscaling:SetDesiredCapacity\",\n                \"autoscaling:TerminateInstanceInAutoScalingGroup\",\n                \"ec2:DescribeLaunchTemplateVersions\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyAutoScaling\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"PolicyCertManagerChangeSet\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"route53:ChangeResourceRecordSets\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": {\n                \"Fn::Sub\": \"arn:${AWS::Partition}:route53:::hostedzone/*\"\n              }\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyCertManagerChangeSet\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"PolicyCertManagerGetChange\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"route53:GetChange\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": {\n                \"Fn::Sub\": \"arn:${AWS::Partition}:route53:::change/*\"\n              }\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyCertManagerGetChange\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"PolicyCertManagerHostedZones\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"route53:ListResourceRecordSets\",\n                \"route53:ListHostedZonesByName\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyCertManagerHostedZones\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"PolicyEBS\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"ec2:CreateSnapshot\",\n                \"ec2:AttachVolume\",\n                \"ec2:DetachVolume\",\n                \"ec2:ModifyVolume\",\n                \"ec2:DescribeAvailabilityZones\",\n                \"ec2:DescribeInstances\",\n                \"ec2:DescribeSnapshots\",\n                \"ec2:DescribeTags\",\n                \"ec2:DescribeVolumes\",\n                \"ec2:DescribeVolumesModifications\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:CreateTags\"\n              ],\n              \"Condition\": {\n                \"StringEquals\": {\n                  \"ec2:CreateAction\": [\n                    \"CreateVolume\",\n                    \"CreateSnapshot\"\n                  ]\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": [\n                \"arn:aws:ec2:*:*:volume/*\",\n                \"arn:aws:ec2:*:*:snapshot/*\"\n              ]\n            },\n            {\n              \"Action\": [\n                \"ec2:DeleteTags\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": [\n                \"arn:aws:ec2:*:*:volume/*\",\n                \"arn:aws:ec2:*:*:snapshot/*\"\n              ]\n            },\n            {\n              \"Action\": [\n                \"ec2:CreateVolume\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"aws:RequestTag/ebs.csi.aws.com/cluster\": \"true\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:CreateVolume\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"aws:RequestTag/CSIVolumeName\": \"*\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:CreateVolume\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"aws:RequestTag/kubernetes.io/cluster/*\": \"owned\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:DeleteVolume\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"ec2:ResourceTag/ebs.csi.aws.com/cluster\": \"true\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:DeleteVolume\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"ec2:ResourceTag/CSIVolumeName\": \"*\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:DeleteVolume\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"ec2:ResourceTag/kubernetes.io/cluster/*\": \"owned\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:DeleteSnapshot\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"ec2:ResourceTag/CSIVolumeSnapshotName\": \"*\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            },\n            {\n              \"Action\": [\n                \"ec2:DeleteSnapshot\"\n              ],\n              \"Condition\": {\n                \"StringLike\": {\n                  \"ec2:ResourceTag/ebs.csi.aws.com/cluster\": \"true\"\n                }\n              },\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyEBS\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"PolicyExternalDNSChangeSet\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"route53:ChangeResourceRecordSets\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": {\n                \"Fn::Sub\": \"arn:${AWS::Partition}:route53:::hostedzone/*\"\n              }\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyExternalDNSChangeSet\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"PolicyExternalDNSHostedZones\": {\n      \"Type\": \"AWS::IAM::Policy\",\n      \"Properties\": {\n        \"PolicyDocument\": {\n          \"Statement\": [\n            {\n              \"Action\": [\n                \"route53:ListHostedZones\",\n                \"route53:ListResourceRecordSets\",\n                \"route53:ListTagsForResource\"\n              ],\n              \"Effect\": \"Allow\",\n              \"Resource\": \"*\"\n            }\n          ],\n          \"Version\": \"2012-10-17\"\n        },\n        \"PolicyName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-PolicyExternalDNSHostedZones\"\n        },\n        \"Roles\": [\n          {\n            \"Ref\": \"NodeInstanceRole\"\n          }\n        ]\n      }\n    },\n    \"SSH\": {\n      \"Type\": \"AWS::EC2::SecurityGroup\",\n      \"Properties\": {\n        \"GroupDescription\": \"Allow SSH access\",\n        \"GroupName\": {\n          \"Fn::Sub\": \"${AWS::StackName}-remoteAccess\"\n        },\n        \"SecurityGroupIngress\": [\n          {\n            \"CidrIp\": \"*.*.*.*/23\",\n            \"Description\": \"Allow SSH access to managed worker nodes in group nodeGroupName (private, only inside VPC)\",\n            \"FromPort\": 22,\n            \"IpProtocol\": \"tcp\",\n            \"ToPort\": 22\n          }\n        ],\n        \"Tags\": [\n          {\n            \"Key\": \"Name\",\n            \"Value\": {\n              \"Fn::Sub\": \"${AWS::StackName}/SSH\"\n            }\n          }\n        ],\n        \"VpcId\": {\n          \"Fn::ImportValue\": \"eksctl-clusterName-cluster::VPC\"\n        }\n      }\n    }\n  }\n}"
}
2021-12-16 17:20:26 [ℹ]  deploying stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:20:26 [▶]  start waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:20:26 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:20:43 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:21:02 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:21:20 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:21:41 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:21:57 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:22:17 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:22:36 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:22:55 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:23:13 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:23:31 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:23:53 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:24:11 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:24:30 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:24:46 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:25:06 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:25:25 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:25:41 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:25:58 [ℹ]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:26:00 [✖]  unexpected status "CREATE_FAILED" while waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName"
2021-12-16 17:26:00 [ℹ]  fetching stack events in attempt to troubleshoot the root cause of the failure
2021-12-16 17:26:00 [✖]  AWS::CloudFormation::Stack/eksctl-clusterName-nodegroup-nodeGroupName: CREATE_FAILED – "The following resource(s) failed to create: [PolicyEBS]. "
2021-12-16 17:26:00 [▶]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerGetChange: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyExternalDNSHostedZones: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerHostedZones: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerChangeSet: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyAutoScaling: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyExternalDNSChangeSet: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [✖]  AWS::IAM::Policy/PolicyEBS: CREATE_FAILED – "Partition \"aws\" is not valid for resource \"arn:aws:ec2:*:*:volume/*\". (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 6afd146e-b804-4349-8de6-8e657ec1a06b; Proxy: null)"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerGetChange: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyExternalDNSHostedZones: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerHostedZones: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerChangeSet: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyAutoScaling: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyExternalDNSChangeSet: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyEBS: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerGetChange: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerHostedZones: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyExternalDNSHostedZones: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyCertManagerChangeSet: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyAutoScaling: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Policy/PolicyExternalDNSChangeSet: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::EKS::Nodegroup/ManagedNodeGroup: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::IAM::Role/NodeInstanceRole: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::EC2::LaunchTemplate/LaunchTemplate: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::EC2::LaunchTemplate/LaunchTemplate: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::EC2::LaunchTemplate/LaunchTemplate: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::EC2::SecurityGroup/SSH: CREATE_COMPLETE
2021-12-16 17:26:00 [▶]  AWS::EC2::SecurityGroup/SSH: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS – "Resource creation Initiated"
2021-12-16 17:26:00 [▶]  AWS::IAM::Role/NodeInstanceRole: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::EC2::SecurityGroup/SSH: CREATE_IN_PROGRESS
2021-12-16 17:26:00 [▶]  AWS::CloudFormation::Stack/eksctl-clusterName-nodegroup-nodeGroupName: CREATE_IN_PROGRESS – "User Initiated"
2021-12-16 17:26:00 [▶]  failed task: create managed nodegroup "nodeGroupName" (will continue until other parallel tasks are completed)
2021-12-16 17:26:00 [▶]  failed task: create managed nodegroup "nodeGroupName" (will not run other sequential tasks)
2021-12-16 17:26:00 [▶]  failed task:
    2 sequential sub-tasks: {
        6 sequential sub-tasks: {
            wait for control plane to become ready,
            update cluster VPC endpoint access configuration,
            associate IAM OIDC provider,
            no tasks,
            restart daemonset "kube-system/aws-node",
            1 task: { create addons },
        },
        create managed nodegroup "nodeGroupName",
    }
 (will not run other sequential tasks)
2021-12-16 17:26:00 [!]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
2021-12-16 17:26:00 [ℹ]  to cleanup resources, run 'eksctl delete cluster --region=us-gov-west-1 --name=clusterName'
2021-12-16 17:26:00 [✖]  waiting for CloudFormation stack "eksctl-clusterName-nodegroup-nodeGroupName": ResourceNotReady: failed waiting for successful resource state
Error: failed to create cluster "clusterName"

This is all running on an Amazon Linux 2 AMI using a standard binary and being run using a service account.

Versions

$ eksctl info
eksctl version: 0.75.0
kubectl version: v1.22.4
OS: linux

[Skarlso]

Thanks for opening this issue. Yes, we have a few places still where we don't properly use the right Partition. We have to fix those too.

[kbrian187]

Good to know. Do you guys have a timeline? I ask only because my team has a need for it. It's not ultra critical but we'd like to be able to tell our customer when it is we can hit full automation. Thanks

…

On Thu, Dec 16, 2021, 16:59 Gergely Brautigam ***@***.***> wrote: Thanks for opening this issue. Yes, we have a few places still where we don't properly use the right Partition. We have to fix those too. — Reply to this email directly, view it on GitHub <#4545 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLXKMZ5ZKUOI72TSSKD4ELURJOMHANCNFSM5KHJUVNA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[Skarlso]

Sorry, no idea. :/ Especially with holidays coming up. We'll try to triage it, and see what we can do about it. But I have no timeline right now.

[Skarlso]

I'll talk it over with the team, and try to get you a possible ETA.

[Skarlso]

@kbrian187 never mind. I just did in my spare time.

Please try the linked PR if it works. We won't be able to release it just yet, I think ( because of the holidays ) but if you could pull the fork, build it, and try it and see if it works, at least you'll have a working thing until it's officially released.

Hope this helps! Happy holidays!

[kbrian187]

Thanks for the quick turnaround, much appreciated.