diff --git a/docs/providers-guide.md b/docs/providers-guide.md index 5bf3d2ddd..842c328b8 100644 --- a/docs/providers-guide.md +++ b/docs/providers-guide.md @@ -307,6 +307,106 @@ Provider type: `codebuild`. > Note: Either specify the `spec_inline` or the `spec_filename` in the > properties block. If both are supplied, the pipeline generator will throw > an error instead. +- *vpc_id* *(String)* defaults to none. + > Configure the `vpc_id` if the CodeBuild instance needs to connect through + > a VPC. You will need to set the `subnet_ids` property as well. Plus, + > optionally, you can configure the `security_group_ids` to specify what + > security groups the instance should use. + > + > Please note: + > VPC support can be added to a CodeBuild step in the pipeline, but cannot + > be removed that easily. + > + > In case you want to remove VPC support after adding it first: + > You need to delete the pipeline CloudFormation stack of the pipeline that + > should be updated. Then *release a change* in the + > `aws-deployment-framework-pipelines` in CodePipeline to regenerate the + > stack without the VPC support. + > + > An example of a `vpc_id` value: `vpc-01234567890abcdef` +- *subnet_ids* *(List of Strings)* **(with VPC usage only)** defaults to none. + > The list of subnet ids that the CodeBuild instance is configured to use. + > These subnets need to be part of the VPC that is configured by the `vpc_id` + > property of the same provider. + > + > Please note: + > Only configure the `subnet_ids` when the `vpc_id` is also configured. + > Make sure there are multiple subnets listed that are hosted in separate + > availability zones to ensure a reliable service. + > + > An example of a list of `subnet_ids` is: + > `["subnet-1234567890abcdef0", "subnet-bcdef01234567890a"]` +- *security_group_ids* *(List of Strings)* **(with VPC usage only)** defaults to none. + > The list of security group ids that the CodeBuild instance is configured to use. + > These security groups need to be part of the VPC that is configured by the `vpc_id` + > property of the same provider. + > + > ADF will generate a default security group when you configured a `vpc_id` + > but did not configure any `security_group_ids`. The default security + > group has an allow all egress traffic rule. It is recommended that you + > make use of specific security groups instead. + > + > Typically, one security group would be sufficient, unless you need to + > combine multiple security groups to grant the build environment all access + > it needs. + > + > Please note: + > Only configure the `security_group_ids` when the `vpc_id` is also configured. + > To configure access securely, you need to create and specify the exact + > security group to use on a pipeline per pipeline basis. Such that pipelines + > will only have access to the resources they are allowed to access and + > nothing more. + > + > An example of a list of `security_group_ids` is: + > `["sg-234567890abcdef01", "sg-cdef01234567890ab"]` + +#### Setup permissions for CodeBuild VPC usage + +When you want to configure CodeBuild to use a specific VPC, you can make use of +the `vpc_id`, `subnet_ids`, and/or `security_group_ids` properties. + +However, before you do so, you need to make sure that ADF is allowed to deploy +CodeBuild in the specific VPC that you want. + +You need to update the `aws-deployment-framework-bootstrap` repository once +to grant it access to deploy. To grant access, follow these instructions +closely: + +1. Open the `aws-deployment-framework-bootstrap` repository. +2. Navigate to the `adf-bootstrap/deployment` folder. +3. Check whether the following file exists inside that directory + `global-iam.yml`: The full path for this file in that repository would be + `adf-bootstrap/deployment/global-iam.yml`. +4. If it does not exist, you need to create a copy of the + `example-global-iam.yml` that is stored inside that directory and store it + as `global-iam.yml`. You can comment out the `CloudFormationDeploymentPolicy` + block that is added by the example or tweak it to your needs. +5. Compare the content of the `global-iam.yml` file against the + `example-global-iam.yml` file. + The section that you are interested in starts off with: +```yaml +## +# Begin of VPC CodeBuild support IAM permissions +## +``` + Until the end is commented as: +```yaml +## +# End of VPC CodeBuild support IAM permissions +## +``` + + The `PipelineProvisionerResourcePolicy` and `CodeBuildResourcePolicy` + resources should be listed and configured to allow the use of VPCs in the + CodeBuild provider deployed by ADF. Ensure these are not commented out and + match same IAM policy as defined in the `example-global-iam.yml` file. + +7. If necessary, commit the changes you made to the repository and have them + peer reviewed and merged into the main branch of the + `aws-deployment-framework-bootstrap` repository. +8. You should be allowed to use VPCs in CodeBuild once the + `aws-deployment-framework-bootstrap` pipeline finished deploying your + changes. ### Jenkins diff --git a/samples/sample-codebuild-vpc/README.md b/samples/sample-codebuild-vpc/README.md new file mode 100644 index 000000000..0c2d977f8 --- /dev/null +++ b/samples/sample-codebuild-vpc/README.md @@ -0,0 +1,57 @@ +# Sample CodeBuild VPC usage showcasing ADF Pipelines + +This pipeline will demonstrate how-to setup CodeBuild to use a specific VPC. + +**Please note**: Before you can deploy CodeBuild in a VPC, you need to follow the +instructions as described in the CodeBuild provider documentation at: +[docs/providers-guide.md](../../docs/providers-guide.md#setup-permissions-for-codebuild-vpc-usage) +This is only required once to allow the CodeBuild service to locate and create +the required resources. Once configured, the permissions allow any pipeline to +make use of VPCs when running CodeBuild steps. + +Back to the sample: The pipeline deploys a simple S3 bucket without granting +any permissions. The point of this sample is to demonstrate how different +build and deployment stages can use CodeBuild in a VPC to connect to internal +resources. + +Create a new repository that will host the files that are contained inside +this sample folder. + +Update the `vpc_id`, `subnet_ids`, and `security_group_ids` attributes to match +your own VPC and subnets that are operational in the deployment account. + +### Deployment Map example + +```yaml + - name: sample-codebuild-vpc + default_providers: + source: + provider: codecommit + properties: + account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_5_0" + vpc_id: vpc-01234567890abcdef + subnet_ids: + - subnet-1234567890abcdef1 + - subnet-bcdef01234567890a + deploy: + provider: cloudformation + targets: + - /banking/testing + - name: integration-tests + provider: codebuild + properties: + image: "STANDARD_5_0" + spec_filename: testspec.yml + vpc_id: vpc-01234567890abcdef + subnet_ids: + - subnet-1234567890abcdef1 + - subnet-bcdef01234567890a + security_group_ids: + - sg-234567890abcdef01 + - sg-cdef01234567890ab + - /banking/production +``` diff --git a/samples/sample-codebuild-vpc/buildspec.yml b/samples/sample-codebuild-vpc/buildspec.yml new file mode 100644 index 000000000..bdb030cbd --- /dev/null +++ b/samples/sample-codebuild-vpc/buildspec.yml @@ -0,0 +1,25 @@ +version: 0.2 + +phases: + install: + runtime-versions: + python: 3.9 + commands: + # It will connect through the VPC to fetch all the resources. + # Make sure the subnets and security groups are configured such that + # it is able to connect to S3 and fetch the requirements using pip. + # + # If you want to restrict public access, you can create a local copy + # of the pip required packages and use S3 private link. + - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet + - pip install -r adf-build/requirements.txt -q + + build: + commands: + - python adf-build/generate_params.py + +artifacts: + files: + - 'template.yml' + - 'params/*.json' + - 'params/*.yml"' diff --git a/samples/sample-codebuild-vpc/params/global.yml b/samples/sample-codebuild-vpc/params/global.yml new file mode 100644 index 000000000..89f82f3ab --- /dev/null +++ b/samples/sample-codebuild-vpc/params/global.yml @@ -0,0 +1,3 @@ +Tags: + Repository: sample-codebuild-vpc-repo + App: Sample CodeBuild VPC application diff --git a/samples/sample-codebuild-vpc/template.yml b/samples/sample-codebuild-vpc/template.yml new file mode 100644 index 000000000..c84eff6b6 --- /dev/null +++ b/samples/sample-codebuild-vpc/template.yml @@ -0,0 +1,22 @@ +# // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# // SPDX-License-Identifier: Apache-2.0 + +AWSTemplateFormatVersion: '2010-09-09' +Description: ADF CloudFormation Sample Template +Metadata: + License: Apache-2.0 +Resources: + Bucket: + Type: AWS::S3::Bucket + Properties: + BucketEncryption: + ServerSideEncryptionConfiguration: + - ServerSideEncryptionByDefault: + SSEAlgorithm: AES256 + VersioningConfiguration: + Status: Enabled + PublicAccessBlockConfiguration: + BlockPublicAcls: true + BlockPublicPolicy: true + IgnorePublicAcls: true + RestrictPublicBuckets: true diff --git a/samples/sample-codebuild-vpc/testspec.yml b/samples/sample-codebuild-vpc/testspec.yml new file mode 100644 index 000000000..bcadb1b49 --- /dev/null +++ b/samples/sample-codebuild-vpc/testspec.yml @@ -0,0 +1,12 @@ +version: 0.2 + +phases: + install: + runtime-versions: + python: 3.9 + + build: + commands: + # A sample API call to an internal only service to perform the + # integration tests. + - curl https://integration-test-url.internal/test diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/example-global-iam.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/example-global-iam.yml index f571d3793..ac4d3af98 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/example-global-iam.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/example-global-iam.yml @@ -26,3 +26,66 @@ Resources: - "*" Roles: - adf-cloudformation-deployment-role + + ## + # Begin of VPC CodeBuild support IAM permissions + ## + PipelineProvisionerResourcePolicy: + Type: AWS::IAM::Policy + Properties: + PolicyName: "adf-pipeline-provisioner-codebuild-role-policy" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Sid: "CodeBuildVPC" + Action: + - "ec2:AuthorizeSecurityGroupEgress" + - "ec2:AuthorizeSecurityGroupIngress" + - "ec2:CreateSecurityGroup" + - "ec2:CreateTags" + - "ec2:DeleteSecurityGroup" + - "ec2:DeleteSecurityGroup" + - "ec2:DeleteTags" + - "ec2:Describe*" + - "ec2:List*" + - "ec2:RevokeSecurityGroupEgress" + - "ec2:RevokeSecurityGroupIngress" + Resource: + - "*" + Roles: + - adf-pipeline-provisioner-codebuild-role + + CodeBuildResourcePolicy: + Type: AWS::IAM::Policy + Properties: + PolicyName: "adf-codebuild-role-policy" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Sid: "CodeBuildVPC" + Action: + - "ec2:CreateNetworkInterface" + - "ec2:DescribeDhcpOptions" + - "ec2:DescribeNetworkInterfaces" + - "ec2:DeleteNetworkInterface" + - "ec2:DescribeSubnets" + - "ec2:DescribeSecurityGroups" + - "ec2:DescribeVpcs" + Resource: + - "*" + - Effect: Allow + Sid: "CodeBuildENI" + Action: + - "ec2:CreateNetworkInterfacePermission" + Resource: + - "*" + Condition: + StringEquals: + ec2:AuthorizedService: "codebuild.amazonaws.com" + Roles: + - adf-codebuild-role + ## + # End of VPC CodeBuild support IAM permissions + ## diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py index 06c2bbce3..d6b82a15f 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py @@ -11,6 +11,7 @@ aws_iam as _iam, aws_kms as _kms, aws_ecr as _ecr, + aws_ec2 as _ec2, core ) @@ -64,7 +65,7 @@ def __init__(self, scope: core.Construct, id: str, shared_modules_bucket: str, d map_params['default_providers']['deploy'].get('properties', {}), target, ) - _codebuild.PipelineProject( + self.pipeline_project = _codebuild.PipelineProject( self, 'project', environment=_env, @@ -75,6 +76,10 @@ def __init__(self, scope: core.Construct, id: str, shared_modules_bucket: str, d role=_iam.Role.from_role_arn(self, 'build_role', role_arn=_build_role, mutable=False), build_spec=build_spec, ) + self._setup_vpc( + map_params['default_providers']['deploy'], + target=target, + ) self.deploy = Action( name=id, provider="CodeBuild", @@ -107,7 +112,7 @@ def __init__(self, scope: core.Construct, id: str, shared_modules_bucket: str, d id, map_params['default_providers']['build'].get('properties', {}) ) - _codebuild.PipelineProject( + self.pipeline_project = _codebuild.PipelineProject( self, 'project', environment=_env, @@ -118,6 +123,7 @@ def __init__(self, scope: core.Construct, id: str, shared_modules_bucket: str, d build_spec=build_spec, role=_iam.Role.from_role_arn(self, 'default_build_role', role_arn=_build_role, mutable=False) ) + self._setup_vpc(map_params['default_providers']['build']) self.build = _codepipeline.CfnPipeline.StageDeclarationProperty( name="Build", actions=[ @@ -132,6 +138,63 @@ def __init__(self, scope: core.Construct, id: str, shared_modules_bucket: str, d ] ) + def _setup_vpc(self, default_provider, target=None): + default_props = default_provider.get('properties', {}) + # This will either be empty (build stage) or configured (deploy stage) + target_props = (target or {}).get('properties', {}) + vpc_id = target_props.get('vpc_id', default_props.get('vpc_id')) + subnet_ids = target_props.get( + 'subnet_ids', + default_props.get('subnet_ids', []), + ) + security_group_ids = target_props.get( + 'security_group_ids', + default_props.get('security_group_ids', []), + ) + if vpc_id: + if not subnet_ids: + raise Exception( + "CodeBuild environment of " + f"{self.pipeline_project.project_name} has a " + f"VPC Id ({vpc_id}) set, but no subnets are configured. " + "When specifying the VPC Id for a given CodeBuild " + "environment, you also need to specify the subnet_ids " + "and optionally the security_group_ids that should be " + "used by the CodeBuild instance." + ) + if not security_group_ids: + default_security_group = _ec2.CfnSecurityGroup( + self, + 'sg', + group_description=( + f"The default security group for {self.node.id}" + ), + security_group_egress=[ + { + "cidrIp": "0.0.0.0/0", + "ipProtocol": "-1", + } + ], + vpc_id=vpc_id, + ) + security_group_ids = [ + default_security_group.get_att("GroupId"), + ] + self.pipeline_project.node.default_child.add_property_override( + "VpcConfig", + { + "VpcId": vpc_id, + "Subnets": subnet_ids, + "SecurityGroupIds": security_group_ids, + }, + ) + elif subnet_ids or security_group_ids: + raise Exception( + "CodeBuild environment of " + f"{self.pipeline_project.project_name} requires a VPC Id when " + "configured to connect to specific subnets." + ) + @staticmethod def _determine_stage_build_spec(codebuild_id, props, stage_name, default_filename): filename = props.get('spec_filename') diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/generate_pipeline_stacks.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/generate_pipeline_stacks.py index f414a27bf..6022890b8 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/generate_pipeline_stacks.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/generate_pipeline_stacks.py @@ -26,7 +26,6 @@ def main(): LOGGER.info("ADF Version %s", ADF_VERSION) LOGGER.info("ADF Log Level is %s", ADF_LOG_LEVEL) - _threads = [] _templates = glob.glob("cdk_inputs/*.json") for template_path in _templates: with open(template_path, encoding="utf-8") as template: diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/schema_validation.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/schema_validation.py index 9ba17cc03..5ee5cd458 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/schema_validation.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/schema_validation.py @@ -108,6 +108,9 @@ Optional("tag"): str, # defaults to latest } CODEBUILD_PROPS = { + Optional("vpc_id"): str, + Optional("subnet_ids"): [str], + Optional("security_group_ids"): [str], Optional("image"): Or(str, CODEBUILD_IMAGE_PROPS), Optional("size"): Or('small', 'medium', 'large'), Optional("spec_filename"): str,