[Bug] eksctl removed the "allow all" outbound rule of the control plane security group

[xiangyanw]

What were you trying to accomplish?

Don't remove the default "allow all" outbound rule of the control plane security group.

What happened?

When I use eksctl to create a self-managed node group, it will add two SecurityGroupEgress rules to the control plane security group, and in the mean time remove the “allow all” outbound rule. If my EKS cluster was not created by eksctl and has only one security group, this will become an issue.

How to reproduce it?

1. Create an EKS cluster from AWS console or AWS CLI without providing a security group;
2. Create a self-managed node group by eksctl;

Logs
The cloud formation of the self-managed node group will update the SecurityGroupEgress of the control plane security.

  "Resources": {
    "EgressInterCluster": {
      "Type": "AWS::EC2::SecurityGroupEgress",
      "Properties": {
        "Description": "Allow control plane to communicate with worker nodes in group umng (kubelet and workload TCP ports)",
        "DestinationSecurityGroupId": {
          "Ref": "SG"
        },
        "FromPort": 1025,
        "GroupId": "sg-xxxxxxxxxxxxxxxx",
        "IpProtocol": "tcp",
        "ToPort": 65535
      }
    },
    "EgressInterClusterAPI": {
      "Type": "AWS::EC2::SecurityGroupEgress",
      "Properties": {
        "Description": "Allow control plane to communicate with worker nodes in group umng (workloads using HTTPS port, commonly used with extension API servers)",
        "DestinationSecurityGroupId": {
          "Ref": "SG"
        },
        "FromPort": 443,
        "GroupId": "sg-xxxxxxxxxxxxxxxx",
        "IpProtocol": "tcp",
        "ToPort": 443
      }
    },

But it turns out that it will remove the default "allow all" outbound rule when updating the security group.

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROAZDBM2GWNROF7KU5BW:i-0053c1026xxxxxxxx",
        "arn": "arn:aws:sts::xxxxxxxx:assumed-role/AdminAccessForEC2/i-0053c1026xxxxxxxx",
        "accountId": "xxxxxxxx",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROAZDBM2GWNRxxxxxxxx",
                "arn": "arn:aws:iam::xxxxxxxx:role/AdminAccessForEC2",
                "accountId": "xxxxxxxx",
                "userName": "AdminAccessForEC2"
            },
            "webIdFederationData": {},
            "attributes": {
                "creationDate": "2023-03-23T10:26:50Z",
                "mfaAuthenticated": "false"
            },
            "ec2RoleDelivery": "2.0"
        },
        "invokedBy": "cloudformation.amazonaws.com"
    },
    "eventTime": "2023-03-23T11:12:50Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "RevokeSecurityGroupEgress",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "requestParameters": {
        "groupId": "sg-0dce1a0c7xxxxxxxx",
        "ipPermissions": {
            "items": [
                {
                    "ipProtocol": "-1",
                    "groups": {},
                    "ipRanges": {
                        "items": [
                            {
                                "cidrIp": "0.0.0.0/0"
                            }
                        ]
                    },
                    "ipv6Ranges": {},
                    "prefixListIds": {}
                }
            ]
        }
    },
    "responseElements": {
        "requestId": "32fb32c7-2843-42d1-954e-b3b8b0df3846",
        "_return": true
    },
    "requestID": "32fb32c7-2843-42d1-954e-b3b8b0df3846",
    "eventID": "2493d4ec-9b04-4a6c-96ad-b9c0caf955f5",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "xxxxxxxx",
    "eventCategory": "Management"
}

Anything else we need to know?

Versions

$ eksctl info
eksctl version: 0.133.0
kubectl version: v1.24.0
OS: linux

[github-actions]

Hello xiangyanw 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[Sathyam-Unicommerce]

Facing the same issue, the existing security group's outbound rules are not appended. Instead it deletes the existing rules and adds/remove the rules upon creation and deletion of nodegroups.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[berry2012]

faced the same issue
EKS Cluster version: 1.22

eksctl info
 eksctl version: 0.154.0
 kubectl version: v1.22.17
 OS: linux

Same result as

eksctl info
eksctl version: 0.146.0
kubectl version: v1.25.4
OS: darwin

---

Same test on EKS Cluster version: 1.27 doesn't overwrite the "allow all" outbound rule of the control plane security group

[cPu1]

I was able to reproduce this issue. It appears that you passed the default EKS cluster security group (clusterSecurityGroupId field) to the vpc.securityGroup field. eksctl adds ingress and egress rules to the security group passed in vpc.securityGroup to facilitate communication between the control plane and self-managed nodegroups. However, it appears that CloudFormation is revoking existing egress rules attached to the security group instead of appending new rules. I find this behaviour of CloudFormation a bit strange, but even if as a workaround eksctl were to include the existing egress rules in a nodegroup's CloudFormation stack so as to prevent revoking existing rules, the egress rules will still get revoked when the nodegroup is deleted as the egress rules will become part of the nodegroup's CloudFormation stack.

This feature was not designed to be used with the default cluster security group. For now, I'd recommend creating a new security group dedicated to allowing communication between the control plane and self-managed nodegroups, instead of passing a security group with existing egress rules. Alternatively, you can switch to EKS managed nodegroups if you're not using these features. eksctl can improve this experience by disallowing passing a security group in vpc.securityGroup that has any egress rules attached.

BTW, thanks for the detailed issue, @xiangyanw.