Add support for the CodeBuild provider to run inside a VPC

docs/providers-guide.md

@@ -307,6 +307,106 @@ Provider type: `codebuild`.
   > Note: Either specify the `spec_inline` or the `spec_filename` in the
   > properties block. If both are supplied, the pipeline generator will throw
   > an error instead.
+- *vpc_id* *(String)* defaults to none.
+  > Configure the `vpc_id` if the CodeBuild instance needs to connect through
+  > a VPC. You will need to set the `subnet_ids` property as well. Plus,
+  > optionally, you can configure the `security_group_ids` to specify what
+  > security groups the instance should use.
+  >
+  > Please note:
+  > VPC support can be added to a CodeBuild step in the pipeline, but cannot
+  > be removed that easily.
+  >
+  > In case you want to remove VPC support after adding it first:
+  > You need to delete the pipeline CloudFormation stack of the pipeline that
+  > should be updated. Then *release a change* in the
+  > `aws-deployment-framework-pipelines` in CodePipeline to regenerate the
+  > stack without the VPC support.
+  >
+  > An example of a `vpc_id` value: `vpc-01234567890abcdef`
+- *subnet_ids* *(List of Strings)* **(with VPC usage only)** defaults to none.
+  > The list of subnet ids that the CodeBuild instance is configured to use.
+  > These subnets need to be part of the VPC that is configured by the `vpc_id`
+  > property of the same provider.
+  >
+  > Please note:
+  > Only configure the `subnet_ids` when the `vpc_id` is also configured.
+  > Make sure there are multiple subnets listed that are hosted in separate
+  > availability zones to ensure a reliable service.
+  >
+  > An example of a list of `subnet_ids` is:
+  > `["subnet-1234567890abcdef0", "subnet-bcdef01234567890a"]`
+- *security_group_ids* *(List of Strings)* **(with VPC usage only)** defaults to none.
+  > The list of security group ids that the CodeBuild instance is configured to use.
+  > These security groups need to be part of the VPC that is configured by the `vpc_id`
+  > property of the same provider.
+  >
+  > ADF will generate a default security group when you configured a `vpc_id`
+  > but did not configure any `security_group_ids`. The default security
+  > group has an allow all egress traffic rule. It is recommended that you
+  > make use of specific security groups instead.
+  >
+  > Typically, one security group would be sufficient, unless you need to
+  > combine multiple security groups to grant the build environment all access
+  > it needs.
+  >
+  > Please note:
+  > Only configure the `security_group_ids` when the `vpc_id` is also configured.
+  > To configure access securely, you need to create and specify the exact
+  > security group to use on a pipeline per pipeline basis. Such that pipelines
+  > will only have access to the resources they are allowed to access and
+  > nothing more.
+  >
+  > An example of a list of `security_group_ids` is:
+  > `["sg-234567890abcdef01", "sg-cdef01234567890ab"]`
+
+#### Setup permissions for CodeBuild VPC usage
+
+When you want to configure CodeBuild to use a specific VPC, you can make use of
+the `vpc_id`, `subnet_ids`, and/or `security_group_ids` properties.
+
+However, before you do so, you need to make sure that ADF is allowed to deploy
+CodeBuild in the specific VPC that you want.
+
+You need to update the `aws-deployment-framework-bootstrap` repository once
+to grant it access to deploy. To grant access, follow these instructions
+closely:
+
+1. Open the `aws-deployment-framework-bootstrap` repository.
+2. Navigate to the `adf-bootstrap/deployment` folder.
+3. Check whether the following file exists inside that directory
+   `global-iam.yml`: The full path for this file in that repository would be
+   `adf-bootstrap/deployment/global-iam.yml`.
+4. If it does not exist, you need to create a copy of the
+   `example-global-iam.yml` that is stored inside that directory and store it
+   as `global-iam.yml`. You can comment out the `CloudFormationDeploymentPolicy`
+   block that is added by the example or tweak it to your needs.
+5. Compare the content of the `global-iam.yml` file against the
+   `example-global-iam.yml` file.
+   The section that you are interested in starts off with:
+```yaml
+##
+# Begin of VPC CodeBuild support IAM permissions
+##
+```
+  Until the end is commented as:
+```yaml
+##
+# End of VPC CodeBuild support IAM permissions
+##
+```
+
+  The `PipelineProvisionerResourcePolicy` and `CodeBuildResourcePolicy`
+  resources should be listed and configured to allow the use of VPCs in the
+  CodeBuild provider deployed by ADF. Ensure these are not commented out and
+  match same IAM policy as defined in the `example-global-iam.yml` file.
+
+7. If necessary, commit the changes you made to the repository and have them
+   peer reviewed and merged into the main branch of the
+   `aws-deployment-framework-bootstrap` repository.
+8. You should be allowed to use VPCs in CodeBuild once the
+   `aws-deployment-framework-bootstrap` pipeline finished deploying your
+   changes.
 
 ### Jenkins
 

samples/sample-codebuild-vpc/README.md

@@ -0,0 +1,57 @@
+# Sample CodeBuild VPC usage showcasing ADF Pipelines
+
+This pipeline will demonstrate how-to setup CodeBuild to use a specific VPC.
+
+**Please note**: Before you can deploy CodeBuild in a VPC, you need to follow the
+instructions as described in the CodeBuild provider documentation at:
+[docs/providers-guide.md](../../docs/providers-guide.md#setup-permissions-for-codebuild-vpc-usage)
+This is only required once to allow the CodeBuild service to locate and create
+the required resources. Once configured, the permissions allow any pipeline to
+make use of VPCs when running CodeBuild steps.
+
+Back to the sample: The pipeline deploys a simple S3 bucket without granting
+any permissions. The point of this sample is to demonstrate how different
+build and deployment stages can use CodeBuild in a VPC to connect to internal
+resources.
+
+Create a new repository that will host the files that are contained inside
+this sample folder.
+
+Update the `vpc_id`, `subnet_ids`, and `security_group_ids` attributes to match
+your own VPC and subnets that are operational in the deployment account.
+
+### Deployment Map example
+
+```yaml
+  - name: sample-codebuild-vpc
+    default_providers:
+      source:
+        provider: codecommit
+        properties:
+          account_id: 111111111111
+      build:
+        provider: codebuild
+        properties:
+          image: "STANDARD_5_0"
+          vpc_id: vpc-01234567890abcdef
+          subnet_ids:
+            - subnet-1234567890abcdef1
+            - subnet-bcdef01234567890a
+      deploy:
+        provider: cloudformation
+    targets:
+      - /banking/testing
+      - name: integration-tests
+        provider: codebuild
+        properties:
+          image: "STANDARD_5_0"
+          spec_filename: testspec.yml
+          vpc_id: vpc-01234567890abcdef
+          subnet_ids:
+            - subnet-1234567890abcdef1
+            - subnet-bcdef01234567890a
+          security_group_ids:
+            - sg-234567890abcdef01
+            - sg-cdef01234567890ab
+      - /banking/production
+```

samples/sample-codebuild-vpc/buildspec.yml

@@ -0,0 +1,25 @@
+version: 0.2
+
+phases:
+  install:
+    runtime-versions:
+      python: 3.9
+    commands:
+      # It will connect through the VPC to fetch all the resources.
+      # Make sure the subnets and security groups are configured such that
+      # it is able to connect to S3 and fetch the requirements using pip.
+      #
+      # If you want to restrict public access, you can create a local copy
+      # of the pip required packages and use S3 private link.
+      - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet
+      - pip install -r adf-build/requirements.txt -q
+
+  build:
+    commands:
+      - python adf-build/generate_params.py
+
+artifacts:
+  files:
+    - 'template.yml'
+    - 'params/*.json'
+    - 'params/*.yml"'

samples/sample-codebuild-vpc/params/global.yml

@@ -0,0 +1,3 @@
+Tags:
+  Repository: sample-codebuild-vpc-repo
+  App: Sample CodeBuild VPC application

samples/sample-codebuild-vpc/template.yml

@@ -0,0 +1,22 @@
+# // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# // SPDX-License-Identifier: Apache-2.0
+
+AWSTemplateFormatVersion: '2010-09-09'
+Description: ADF CloudFormation Sample Template
+Metadata:
+  License: Apache-2.0
+Resources:
+  Bucket:
+    Type: AWS::S3::Bucket
+    Properties:
+      BucketEncryption:
+        ServerSideEncryptionConfiguration:
+          - ServerSideEncryptionByDefault:
+              SSEAlgorithm: AES256
+      VersioningConfiguration:
+        Status: Enabled
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: true
+        BlockPublicPolicy: true
+        IgnorePublicAcls: true
+        RestrictPublicBuckets: true

samples/sample-codebuild-vpc/testspec.yml

@@ -0,0 +1,12 @@
+version: 0.2
+
+phases:
+  install:
+    runtime-versions:
+      python: 3.9
+
+  build:
+    commands:
+      # A sample API call to an internal only service to perform the
+      # integration tests.
+      - curl https://integration-test-url.internal/test

src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/example-global-iam.yml

@@ -26,3 +26,66 @@ Resources:
               - "*"
       Roles:
         - adf-cloudformation-deployment-role
+
+  ##
+  # Begin of VPC CodeBuild support IAM permissions
+  ##
+  PipelineProvisionerResourcePolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: "adf-pipeline-provisioner-codebuild-role-policy"
+      PolicyDocument:
+        Version: "2012-10-17"
+        Statement:
+          - Effect: Allow
+            Sid: "CodeBuildVPC"
+            Action:
+              - "ec2:AuthorizeSecurityGroupEgress"
+              - "ec2:AuthorizeSecurityGroupIngress"
+              - "ec2:CreateSecurityGroup"
+              - "ec2:CreateTags"
+              - "ec2:DeleteSecurityGroup"
+              - "ec2:DeleteSecurityGroup"
+              - "ec2:DeleteTags"
+              - "ec2:Describe*"
+              - "ec2:List*"
+              - "ec2:RevokeSecurityGroupEgress"
+              - "ec2:RevokeSecurityGroupIngress"
+            Resource:
+              - "*"
+      Roles:
+        - adf-pipeline-provisioner-codebuild-role
+
+  CodeBuildResourcePolicy:
+    Type: AWS::IAM::Policy
+    Properties:
+      PolicyName: "adf-codebuild-role-policy"
+      PolicyDocument:
+        Version: "2012-10-17"
+        Statement:
+          - Effect: Allow
+            Sid: "CodeBuildVPC"
+            Action:
+              - "ec2:CreateNetworkInterface"
+              - "ec2:DescribeDhcpOptions"
+              - "ec2:DescribeNetworkInterfaces"
+              - "ec2:DeleteNetworkInterface"
+              - "ec2:DescribeSubnets"
+              - "ec2:DescribeSecurityGroups"
+              - "ec2:DescribeVpcs"
+            Resource:
+              - "*"
+          - Effect: Allow
+            Sid: "CodeBuildENI"
+            Action:
+              - "ec2:CreateNetworkInterfacePermission"
+            Resource:
+              - "*"
+            Condition:
+              StringEquals:
+                ec2:AuthorizedService: "codebuild.amazonaws.com"
+      Roles:
+        - adf-codebuild-role
+  ##
+  # End of VPC CodeBuild support IAM permissions
+  ##