From f514cce5496e065732081d12a38a1f5d968a2be5 Mon Sep 17 00:00:00 2001 From: Simon Kok Date: Thu, 1 Oct 2020 18:56:09 +0200 Subject: [PATCH 1/3] Upgrade to py3.8 + latest CDK, pylint, and others Upgrade to Python 3.8, CDK v1.88 and latest version of pylint and other libraries. Upgraded to use STANDARD:5.0 CodeBuild image. Due to requirements defined in aws-sam-cli, boto3 and botocore had to be set to 1.14.63 and 1.17.63 respectively. Upgrade of pylint resulted in new errors as listed below. These errors have been fixed as part of this commit too. ```bash find src/ -iname "*.py" | xargs pylint --rcfile .pylintrc ************* Module adf-build.organization_policy src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/organization_policy.py:23:0: R1721: Unnecessary use of a comprehension (unnecessary-comprehension) ************* Module adf-build.shared.cdk.cdk_stacks.main src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/main.py:86:16: R1724: Unnecessary "elif" after "continue" (no-else-continue) ************* Module adf-build.shared.cdk.clean_pipelines src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/clean_pipelines.py:77:16: W1309: Using an f-string that does not have any interpolated variables (f-string-without-interpolation) ************* Module adf-build.shared.python.parameter_store src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/parameter_store.py:66:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/parameter_store.py:82:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) ************* Module adf-build.shared.python.cloudformation src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py:206:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) ************* Module adf-build.shared.python.organizations src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py:160:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) ************* Module adf-build.shared.python.codepipeline src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/codepipeline.py:28:0: R1721: Unnecessary use of a comprehension (unnecessary-comprehension) ************* Module adf-build.shared.resolver src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/resolver.py:39:12: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/resolver.py:92:16: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) ************* Module main src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py:13:0: C0411: third party import "from organizations import Organizations" should be placed before "from src import read_config_files, delete_default_vpc, Support" (wrong-import-order) src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py:14:0: C0411: third party import "from logger import configure_logger" should be placed before "from src import read_config_files, delete_default_vpc, Support" (wrong-import-order) src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py:15:0: C0411: third party import "from parameter_store import ParameterStore" should be placed before "from src import read_config_files, delete_default_vpc, Support" (wrong-import-order) src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py:16:0: C0411: third party import "from sts import STS" should be placed before "from src import read_config_files, delete_default_vpc, Support" (wrong-import-order) ************* Module src.vpc src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/src/vpc.py:60:16: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from) src/lambda_codebase/account/main.py:113:0: E0012: Bad option value 'bad-continuation' (bad-option-value) ``` --- .nvmrc | 1 + .travis.yml | 5 +- docs/user-guide.md | 6 +- requirements.txt | 19 +-- samples/sample-cdk-app/README.md | 2 +- samples/sample-cdk-app/buildspec.yml | 4 +- samples/sample-cdk-app/index.ts | 2 +- .../sample-ec2-java-app-codedeploy/README.md | 2 +- .../buildspec.yml | 2 +- samples/sample-ec2-with-codedeploy/README.md | 4 + .../sample-ec2-with-codedeploy/buildspec.yml | 4 +- samples/sample-ecr-repository/README.md | 4 + samples/sample-ecr-repository/buildspec.yml | 4 +- samples/sample-ecs-cluster/README.md | 4 + samples/sample-ecs-cluster/buildspec.yml | 4 +- samples/sample-expunge-vpc/README.md | 2 + samples/sample-expunge-vpc/buildspec.yml | 4 +- samples/sample-expunge-vpc/template.yml | 4 +- samples/sample-fargate-node-app/README.md | 2 +- samples/sample-fargate-node-app/buildspec.yml | 4 +- samples/sample-iam/README.md | 4 + samples/sample-iam/buildspec.yml | 4 +- samples/sample-mono-repo/README.md | 2 + .../sample-mono-repo/apps/alpha/buildspec.yml | 2 + .../sample-mono-repo/apps/beta/buildspec.yml | 2 + samples/sample-serverless-app/README.md | 3 +- samples/sample-serverless-app/buildspec.yml | 4 +- samples/sample-serverless-app/template.yml | 4 +- .../sample-service-catalog-product/README.md | 4 + .../buildspec.yml | 4 +- samples/sample-terraform/README.md | 2 +- samples/sample-terraform/my_test_spec.yml | 5 +- samples/sample-vpc/README.md | 4 + samples/sample-vpc/buildspec.yml | 4 +- src/lambda_codebase/account/main.py | 1 - .../adf-bootstrap/deployment/global.yml | 16 +-- .../initial_commit/requirements.txt | 4 +- .../adf-build/organization_policy.py | 8 +- .../adf-build/provisioner/main.py | 2 +- .../adf-build/provisioner/src/vpc.py | 4 +- .../adf-build/requirements.txt | 22 ++-- .../adf-build/shared/cdk/cdk_stacks/main.py | 2 +- .../adf-build/shared/cdk/clean_pipelines.py | 2 +- .../shared/helpers/package_transform.sh | 2 - .../adf-build/shared/python/cloudformation.py | 2 +- .../adf-build/shared/python/codepipeline.py | 5 +- .../adf-build/shared/python/organizations.py | 4 +- .../shared/python/parameter_store.py | 8 +- .../adf-build/shared/requirements.txt | 114 +++++++++--------- .../adf-build/shared/resolver.py | 17 ++- .../initial_commit/requirements.txt | 4 +- src/template.yml | 76 ++++++------ tox.ini | 2 +- 53 files changed, 246 insertions(+), 180 deletions(-) create mode 100644 .nvmrc diff --git a/.nvmrc b/.nvmrc new file mode 100644 index 000000000..48082f72f --- /dev/null +++ b/.nvmrc @@ -0,0 +1 @@ +12 diff --git a/.travis.yml b/.travis.yml index 258bcfd85..b6fd1800c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,11 +1,12 @@ # Enable container based builds -sudo: false +dist: focal language: python python: - - "3.7" + - "3.8" install: + - nvm install # Install the code requirements - make init diff --git a/docs/user-guide.md b/docs/user-guide.md index e554a792a..aa051c8b5 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -125,7 +125,7 @@ pipelines: account_id: 111112233332 build: provider: codebuild - image: "STANDARD_2_0" # Use a specific docker image (defaults to Python 3.7) for the build stage in this pipeline -> https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-codebuild.LinuxBuildImage.html + image: "STANDARD_4_0" # Use a specific docker image (supports Python 3.7 and Python 3.8) for the build stage in this pipeline -> https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-codebuild.LinuxBuildImage.html deploy: provider: codedeploy targets: @@ -556,8 +556,8 @@ version: 0.2 phases: install: runtime-versions: - python: 3.7 - nodejs: 10 + python: 3.8 + nodejs: 12 pre_build: commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet diff --git a/requirements.txt b/requirements.txt index 6fae53f6f..9b58f6e55 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,9 +1,10 @@ -tox==2.2.1 -pylint==2.2.2 -pytest==3.0.7 -isort==4.3.21 -mock==2.0.0 -boto3~=1.10, >=1.10.47 -pyyaml>=5.1 -astroid==2.1.0 -schema==0.7.1 +astroid~=2.4.2 +botocore==1.17.63 +boto3==1.14.63 +isort==5.6.4 +mock~=4.0.3 +pylint~=2.6.0 +pytest~=6.2.1 +pyyaml>=5.3 +schema~=0.7.2 +tox==3.20.1 diff --git a/samples/sample-cdk-app/README.md b/samples/sample-cdk-app/README.md index 6801733c8..c3e4c2c3b 100644 --- a/samples/sample-cdk-app/README.md +++ b/samples/sample-cdk-app/README.md @@ -14,7 +14,7 @@ This pipeline is expecting *(in the example case)* a AWS CodeCommit repository o build: provider: codebuild properties: - image: "STANDARD_2_0" + image: "STANDARD_4_0" targets: - /banking/testing - /banking/production diff --git a/samples/sample-cdk-app/buildspec.yml b/samples/sample-cdk-app/buildspec.yml index e458b4896..953847be3 100644 --- a/samples/sample-cdk-app/buildspec.yml +++ b/samples/sample-cdk-app/buildspec.yml @@ -3,8 +3,8 @@ version: 0.2 phases: install: runtime-versions: - python: 3.7 - nodejs: 10 + python: 3.8 + nodejs: 12 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q diff --git a/samples/sample-cdk-app/index.ts b/samples/sample-cdk-app/index.ts index 100ac1a6f..34fa3a967 100644 --- a/samples/sample-cdk-app/index.ts +++ b/samples/sample-cdk-app/index.ts @@ -13,7 +13,7 @@ export class LambdaCronStack extends cdk.Stack { code: new lambda.InlineCode(fs.readFileSync('handler.py', { encoding: 'utf-8' })), handler: 'index.main', timeout: cdk.Duration.seconds(300), - runtime: lambda.Runtime.PYTHON_3_7 + runtime: lambda.Runtime.PYTHON_3_8 }); // Run every day at 6PM UTC // See https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html diff --git a/samples/sample-ec2-java-app-codedeploy/README.md b/samples/sample-ec2-java-app-codedeploy/README.md index d9f3a8dc2..647934494 100644 --- a/samples/sample-ec2-java-app-codedeploy/README.md +++ b/samples/sample-ec2-java-app-codedeploy/README.md @@ -14,7 +14,7 @@ This example is coupled with the `sample-ec2-with-codedeploy` repository and is build: provider: codebuild properties: - image: "STANDARD_2_0" # Since we're building a Java application here we want to use STANDARD_2_0 (Ubuntu) as our base CodeBuild Image, that way we can tell it to have Java ready for us so we can build, compile and test our application. + image: "STANDARD_4_0" # Since we're building a Java application here we want to use STANDARD_4_0 (Ubuntu) as our base CodeBuild Image, that way we can tell it to have Java ready for us so we can build, compile and test our application. deploy: provider: codedeploy # We will deploy out application with AWS CodeDeploy. targets: diff --git a/samples/sample-ec2-java-app-codedeploy/buildspec.yml b/samples/sample-ec2-java-app-codedeploy/buildspec.yml index aaab93ecc..6c51511eb 100644 --- a/samples/sample-ec2-java-app-codedeploy/buildspec.yml +++ b/samples/sample-ec2-java-app-codedeploy/buildspec.yml @@ -3,7 +3,7 @@ version: 0.2 phases: install: runtime-versions: - java: openjdk8 + java: corretto8 build: commands: - mvn clean package --quiet diff --git a/samples/sample-ec2-with-codedeploy/README.md b/samples/sample-ec2-with-codedeploy/README.md index aefd3f412..022f31fdd 100644 --- a/samples/sample-ec2-with-codedeploy/README.md +++ b/samples/sample-ec2-with-codedeploy/README.md @@ -19,6 +19,10 @@ This stack also requires `sample-vpc` and `sample-iam` to be in deployed as it i provider: codecommit properties: account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_4_0" # So we can specify which Python version we need targets: - /banking/testing - /banking/production diff --git a/samples/sample-ec2-with-codedeploy/buildspec.yml b/samples/sample-ec2-with-codedeploy/buildspec.yml index 8682854ab..56cc38834 100644 --- a/samples/sample-ec2-with-codedeploy/buildspec.yml +++ b/samples/sample-ec2-with-codedeploy/buildspec.yml @@ -2,9 +2,11 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q - python adf-build/generate_params.py artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-ecr-repository/README.md b/samples/sample-ecr-repository/README.md index 312dc288f..a6480158c 100644 --- a/samples/sample-ecr-repository/README.md +++ b/samples/sample-ecr-repository/README.md @@ -9,6 +9,10 @@ provider: codecommit properties: account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_4_0" # So we can specify which Python version we need targets: - /deployment ``` diff --git a/samples/sample-ecr-repository/buildspec.yml b/samples/sample-ecr-repository/buildspec.yml index 8682854ab..56cc38834 100644 --- a/samples/sample-ecr-repository/buildspec.yml +++ b/samples/sample-ecr-repository/buildspec.yml @@ -2,9 +2,11 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q - python adf-build/generate_params.py artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-ecs-cluster/README.md b/samples/sample-ecs-cluster/README.md index 9fe5858d2..cb7fd83d2 100644 --- a/samples/sample-ecs-cluster/README.md +++ b/samples/sample-ecs-cluster/README.md @@ -9,6 +9,10 @@ provider: codecommit properties: account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_4_0" # So we can specify which Python version we need targets: - 222222222222 - path: 333333333333 diff --git a/samples/sample-ecs-cluster/buildspec.yml b/samples/sample-ecs-cluster/buildspec.yml index 8682854ab..56cc38834 100644 --- a/samples/sample-ecs-cluster/buildspec.yml +++ b/samples/sample-ecs-cluster/buildspec.yml @@ -2,9 +2,11 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q - python adf-build/generate_params.py artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-expunge-vpc/README.md b/samples/sample-expunge-vpc/README.md index 689afd8d9..c41f74e76 100644 --- a/samples/sample-expunge-vpc/README.md +++ b/samples/sample-expunge-vpc/README.md @@ -15,8 +15,10 @@ Upon stack deletion the default VPCs will be recreated. build: provider: codebuild properties: + image: "STANDARD_4_0" # So we can specify which Python version we need environment_variables: CONTAINS_TRANSFORM: true # Required for templates that contain transforms. (eg SAM Templates) + params: - restart_execution_on_update: true targets: diff --git a/samples/sample-expunge-vpc/buildspec.yml b/samples/sample-expunge-vpc/buildspec.yml index df8e74fb6..1180ccf47 100644 --- a/samples/sample-expunge-vpc/buildspec.yml +++ b/samples/sample-expunge-vpc/buildspec.yml @@ -1,6 +1,8 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q @@ -9,4 +11,4 @@ phases: commands: - bash adf-build/helpers/package_transform.sh artifacts: - files: "**/*" \ No newline at end of file + files: "**/*" diff --git a/samples/sample-expunge-vpc/template.yml b/samples/sample-expunge-vpc/template.yml index d37aa3fab..f986ec6d9 100644 --- a/samples/sample-expunge-vpc/template.yml +++ b/samples/sample-expunge-vpc/template.yml @@ -32,7 +32,7 @@ Resources: Handler: lambda_function.lambda_handler MemorySize: 128 Role: !GetAtt LambdaVPCPolicyRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 600 Environment: Variables: @@ -40,4 +40,4 @@ Resources: DeleteVPCCustom: Type: Custom::DeleteVPC Properties: - ServiceToken: !GetAtt DeleteVPCLambda.Arn \ No newline at end of file + ServiceToken: !GetAtt DeleteVPCLambda.Arn diff --git a/samples/sample-fargate-node-app/README.md b/samples/sample-fargate-node-app/README.md index 2b6c1ef79..5050d3391 100644 --- a/samples/sample-fargate-node-app/README.md +++ b/samples/sample-fargate-node-app/README.md @@ -12,7 +12,7 @@ build: provider: codebuild properties: - image: "STANDARD_2_0" + image: "STANDARD_4_0" privileged: true # Required for Docker in Docker to work as expected (Since CodeBuild will run our docker commands to create and push our image) targets: # Example Targets - These accounts/regions have had the sample-vpc deployed - 222222222222 diff --git a/samples/sample-fargate-node-app/buildspec.yml b/samples/sample-fargate-node-app/buildspec.yml index 7b878ae1a..debdfd822 100644 --- a/samples/sample-fargate-node-app/buildspec.yml +++ b/samples/sample-fargate-node-app/buildspec.yml @@ -4,7 +4,7 @@ phases: install: runtime-versions: docker: 18 - python: 3.7 + python: 3.8 build: commands: - bash build/docker.sh @@ -13,4 +13,4 @@ phases: - bash build/generate_parameters.sh artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-iam/README.md b/samples/sample-iam/README.md index 3e7398078..5e439ed7f 100644 --- a/samples/sample-iam/README.md +++ b/samples/sample-iam/README.md @@ -11,6 +11,10 @@ This pipeline is expecting *(in the example case)* a AWS CodeCommit repository o provider: codecommit properties: account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_4_0" params: restart_execution_on_update: True targets: diff --git a/samples/sample-iam/buildspec.yml b/samples/sample-iam/buildspec.yml index 8682854ab..56cc38834 100644 --- a/samples/sample-iam/buildspec.yml +++ b/samples/sample-iam/buildspec.yml @@ -2,9 +2,11 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q - python adf-build/generate_params.py artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-mono-repo/README.md b/samples/sample-mono-repo/README.md index 1c3ebeaec..0953b0294 100644 --- a/samples/sample-mono-repo/README.md +++ b/samples/sample-mono-repo/README.md @@ -33,6 +33,7 @@ repository. build: provider: codebuild properties: + image: "STANDARD_4_0" spec_filename: apps/alpha/buildspec.yml deploy: provider: cloudformation @@ -52,6 +53,7 @@ repository. build: provider: codebuild properties: + image: "STANDARD_4_0" spec_filename: apps/beta/buildspec.yml deploy: provider: cloudformation diff --git a/samples/sample-mono-repo/apps/alpha/buildspec.yml b/samples/sample-mono-repo/apps/alpha/buildspec.yml index c02377b65..d9113f0e0 100644 --- a/samples/sample-mono-repo/apps/alpha/buildspec.yml +++ b/samples/sample-mono-repo/apps/alpha/buildspec.yml @@ -6,6 +6,8 @@ env: phases: install: + runtime-versions: + python: 3.8 commands: - cd $INFRASTRUCTURE_ROOT_DIR - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet diff --git a/samples/sample-mono-repo/apps/beta/buildspec.yml b/samples/sample-mono-repo/apps/beta/buildspec.yml index 0a62fdea8..8bad5885d 100644 --- a/samples/sample-mono-repo/apps/beta/buildspec.yml +++ b/samples/sample-mono-repo/apps/beta/buildspec.yml @@ -6,6 +6,8 @@ env: phases: install: + runtime-versions: + python: 3.8 commands: - cd $INFRASTRUCTURE_ROOT_DIR - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet diff --git a/samples/sample-serverless-app/README.md b/samples/sample-serverless-app/README.md index ba29ab0ee..80f123b98 100644 --- a/samples/sample-serverless-app/README.md +++ b/samples/sample-serverless-app/README.md @@ -10,8 +10,9 @@ properties: account_id: 111111111111 build: - provider: STANDARD_2_0 + provider: codebuild properties: + image: "STANDARD_4_0" environment_variables: CONTAINS_TRANSFORM: True targets: diff --git a/samples/sample-serverless-app/buildspec.yml b/samples/sample-serverless-app/buildspec.yml index 2e606ab41..623c077f2 100644 --- a/samples/sample-serverless-app/buildspec.yml +++ b/samples/sample-serverless-app/buildspec.yml @@ -3,10 +3,10 @@ version: 0.2 phases: install: runtime-versions: - python: 3.7 + python: 3.8 build: commands: - bash build/generate_parameters.sh - bash adf-build/helpers/package_transform.sh artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-serverless-app/template.yml b/samples/sample-serverless-app/template.yml index f3af785d8..b5e0dfd2f 100644 --- a/samples/sample-serverless-app/template.yml +++ b/samples/sample-serverless-app/template.yml @@ -11,7 +11,7 @@ Resources: Type: 'AWS::Serverless::Function' Properties: Handler: lambda_function.handler - Runtime: python3.7 + Runtime: python3.8 CodeUri: . Description: Sample Lambda Function MemorySize: 128 @@ -21,4 +21,4 @@ Resources: Type: Api Properties: Path: /resource/{id} - Method: get \ No newline at end of file + Method: get diff --git a/samples/sample-service-catalog-product/README.md b/samples/sample-service-catalog-product/README.md index dac111652..54265c8f0 100644 --- a/samples/sample-service-catalog-product/README.md +++ b/samples/sample-service-catalog-product/README.md @@ -10,6 +10,10 @@ This stack imports values from `sample-vpc` and `sample-iam`. provider: codecommit properties: account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_4_0" targets: - /banking/testing - path: /banking/production diff --git a/samples/sample-service-catalog-product/buildspec.yml b/samples/sample-service-catalog-product/buildspec.yml index 8682854ab..56cc38834 100644 --- a/samples/sample-service-catalog-product/buildspec.yml +++ b/samples/sample-service-catalog-product/buildspec.yml @@ -2,9 +2,11 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q - python adf-build/generate_params.py artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/samples/sample-terraform/README.md b/samples/sample-terraform/README.md index 633401a7c..095c794d5 100644 --- a/samples/sample-terraform/README.md +++ b/samples/sample-terraform/README.md @@ -12,7 +12,7 @@ deploy: provider: codebuild properties: - image: "STANDARD_2_0" + image: "STANDARD_4_0" targets: - properties: spec_filename: my_test_spec.yml diff --git a/samples/sample-terraform/my_test_spec.yml b/samples/sample-terraform/my_test_spec.yml index 075a0ae84..c27464bf2 100644 --- a/samples/sample-terraform/my_test_spec.yml +++ b/samples/sample-terraform/my_test_spec.yml @@ -9,6 +9,9 @@ env: TF_CLI_ARGS: "-no-color" phases: + install: + runtime-versions: + python: 3.8 build: commands: - export PATH=$PATH:$(pwd) @@ -20,4 +23,4 @@ phases: - terraform validate - bash adf-build/helpers/sts.sh $TF_VAR_TARGET_ACCOUNT_ID $TF_VAR_TARGET_ACCOUNT_ROLE - terraform plan -out $ADF_PROJECT_NAME - - terraform apply $ADF_PROJECT_NAME \ No newline at end of file + - terraform apply $ADF_PROJECT_NAME diff --git a/samples/sample-vpc/README.md b/samples/sample-vpc/README.md index b1ce91fbd..2f6b41a80 100644 --- a/samples/sample-vpc/README.md +++ b/samples/sample-vpc/README.md @@ -9,6 +9,10 @@ provider: codecommit properties: account_id: 111111111111 + build: + provider: codebuild + properties: + image: "STANDARD_4_0" params: restart_execution_on_update: True targets: diff --git a/samples/sample-vpc/buildspec.yml b/samples/sample-vpc/buildspec.yml index 8682854ab..56cc38834 100644 --- a/samples/sample-vpc/buildspec.yml +++ b/samples/sample-vpc/buildspec.yml @@ -2,9 +2,11 @@ version: 0.2 phases: install: + runtime-versions: + python: 3.8 commands: - aws s3 cp s3://$S3_BUCKET_NAME/adf-build/ adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q - python adf-build/generate_params.py artifacts: - files: '**/*' \ No newline at end of file + files: '**/*' diff --git a/src/lambda_codebase/account/main.py b/src/lambda_codebase/account/main.py index 904390aaf..a77c898a8 100644 --- a/src/lambda_codebase/account/main.py +++ b/src/lambda_codebase/account/main.py @@ -110,7 +110,6 @@ def delete_(event, _context): return -# pylint: disable=bad-continuation # https://github.com/PyCQA/pylint/issues/747 def ensure_account(existing_account_id: str, account_name: str, account_email: str, diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml index d5baab6e2..13b287087 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml @@ -26,7 +26,7 @@ Parameters: Image: Description: The Image you wish to use for CodeBuild (defaults to ubuntu). Type: String - Default: "aws/codebuild/standard:2.0" + Default: "aws/codebuild/standard:5.0" ComputeType: Description: The Compute Type to use for AWS CodeBuild Type: String @@ -654,11 +654,11 @@ Resources: phases: install: runtime-versions: - python: 3.7 - nodejs: 10 + python: 3.8 + nodejs: 12 pre_build: commands: - - npm install cdk@1.32 -g -y --quiet --no-progress + - npm install cdk@1.88 -g -y --quiet --no-progress - aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --quiet - pip install -r adf-build/requirements.txt -q -t ./adf-build build: @@ -930,7 +930,7 @@ Resources: Variables: ADF_PIPELINE_PREFIX: !Ref PipelinePrefix ADF_LOG_LEVEL: !Ref ADFLogLevel - Runtime: python3.7 + Runtime: python3.8 Timeout: 10 EnableCrossAccountAccess: Type: "AWS::Serverless::Function" @@ -948,7 +948,7 @@ Resources: FunctionName: UpdateCrossAccountIAM Handler: enable_cross_account_access.lambda_handler Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 900 CheckPipelineStatus: Type: "AWS::Serverless::Function" @@ -965,7 +965,7 @@ Resources: FunctionName: CheckPipelineStatus Handler: update_pipelines.lambda_handler Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 120 LambdaRole: Type: "AWS::IAM::Role" @@ -1172,7 +1172,7 @@ Resources: - codecommit:DeleteBranch Resource: !GetAtt CodeCommitRepository.Arn FunctionName: PipelinesCreateInitialCommitFunction - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 KmsKeyArnParameter: Type: "AWS::SSM::Parameter" diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/requirements.txt index aa5cab371..05aca1e04 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/requirements.txt +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/requirements.txt @@ -1,3 +1,3 @@ -Jinja2~=2.10.1 +Jinja2~=2.11.2 +boto3==1.14.63 cfn-custom-resource~=1.0.1 -boto3~=1.10, >=1.10.47 \ No newline at end of file diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/organization_policy.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/organization_policy.py index fb01b61f2..51c778b56 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/organization_policy.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/organization_policy.py @@ -19,10 +19,10 @@ def __init__(self): @staticmethod def _find_all(policy): - _files = [ - p for p in glob.iglob( - './adf-bootstrap/**/{0}.json'.format(policy), - recursive=True)] + _files = list(glob.iglob( + './adf-bootstrap/**/{0}.json'.format(policy), + recursive=True, + )) return [f.replace('./adf-bootstrap', '.') for f in _files] def _compare_ordered_policy(self, obj): diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py index 7cf0466f6..445cfb107 100755 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/main.py @@ -9,11 +9,11 @@ import os from concurrent.futures import ThreadPoolExecutor import boto3 -from src import read_config_files, delete_default_vpc, Support from organizations import Organizations from logger import configure_logger from parameter_store import ParameterStore from sts import STS +from src import read_config_files, delete_default_vpc, Support LOGGER = configure_logger(__name__) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/src/vpc.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/src/vpc.py index 554be83ba..301222938 100755 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/src/vpc.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/src/vpc.py @@ -57,7 +57,9 @@ def delete_default_vpc(client, account_id, region, role): max_retry_seconds = + 2 sleep(2) if max_retry_seconds <= 0: - raise Exception("Could not describe VPCs within retry limit.") + raise Exception( + "Could not describe VPCs within retry limit.", + ) from e for vpc in vpc_response["Vpcs"]: if vpc["IsDefault"] is True: diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt index b570fec4a..0845fdc52 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt @@ -1,10 +1,14 @@ # Install libs here that you might want in AWS CodeBuild (On Master Account) -boto3==1.12.46 -pylint~=2.2.2 -pytest~=3.0.7 -mock~=2.0.0 -pyyaml~=5.1 -astroid~=2.1.0 -six~=1.11.0 -aws-sam-cli==0.46.2 -pip==19.1.1 +astroid~=2.4.2 +awscli==1.18.140 +aws-sam-cli==1.15.0 +botocore==1.17.63 +boto3==1.14.63 +jsii<2.0.0,>=1.16.0 +mock~=4.0.3 +pip~=20.2.3 +pylint~=2.6.0 +pytest~=6.2.1 +pyyaml>=5.3 +six~=1.15.0 +urllib3~=1.25.11 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/main.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/main.py index 19e373d9b..faaade4ff 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/main.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/main.py @@ -96,7 +96,7 @@ def __init__(self, scope: core.Construct, stack_input: dict, **kwargs) -> None: ).config ]) continue - elif 'codebuild' in target_stage_override: + if 'codebuild' in target_stage_override: _actions.extend([ adf_codebuild.CodeBuild( self, diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/clean_pipelines.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/clean_pipelines.py index 386f388a1..2eea47f8e 100755 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/clean_pipelines.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/clean_pipelines.py @@ -74,7 +74,7 @@ def main(): ADF_PIPELINE_PREFIX ) - LOGGER.info(f'Cleaning Stale Deployment Map entries') + LOGGER.info('Cleaning Stale Deployment Map entries') clean(parameter_store, deployment_map) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/package_transform.sh b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/package_transform.sh index ff82e7398..500468af8 100755 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/package_transform.sh +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/package_transform.sh @@ -24,8 +24,6 @@ do esac done -pip install --upgrade awscli aws-sam-cli -q - if [[ $SKIP_BUILD == 0 ]]; then echo "Perform build step" # Build our template and its potential dependencies diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py index 7d7dc5db9..b750167b1 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py @@ -203,7 +203,7 @@ def _create_change_set(self): return True return False except ClientError as error: - raise GenericAccountConfigureError(error) + raise GenericAccountConfigureError(error) from error except WaiterError as error: err = error.last_response if CloudFormation._change_set_failed_due_to_empty(err["Status"], err["StatusReason"]): diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/codepipeline.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/codepipeline.py index f0b33cbc0..83316c93e 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/codepipeline.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/codepipeline.py @@ -25,8 +25,9 @@ def get_pipeline_status(self, pipeline_name): name=pipeline_name ) - return [i for i in response.get( - 'stageStates')][0]['latestExecution']['status'] + return list( + response.get('stageStates') + )[0]['latestExecution']['status'] except KeyError: LOGGER.error('Pipeline status for %s could not be determined', pipeline_name) return None diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py index dbad8eacd..6a6daf4ce 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/organizations.py @@ -156,8 +156,8 @@ def describe_ou_name(self, ou_id): OrganizationalUnitId=ou_id ) return response['OrganizationalUnit']['Name'] - except ClientError: - raise RootOUIDError("OU is the Root of the Organization") + except ClientError as error: + raise RootOUIDError("OU is the Root of the Organization") from error @staticmethod def determine_ou_path(ou_path, ou_child_name): diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/parameter_store.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/parameter_store.py index ebdaf54c4..fc80d067b 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/parameter_store.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/parameter_store.py @@ -62,10 +62,10 @@ def fetch_parameters_by_path(self, path): Recursive=True, WithDecryption=False ) - except self.client.exceptions.ParameterNotFound: + except self.client.exceptions.ParameterNotFound as error: raise ParameterNotFoundError( 'Parameter Path {0} Not Found'.format(path) - ) + ) from error def fetch_parameter(self, name, with_decryption=False): @@ -78,7 +78,7 @@ def fetch_parameter(self, name, with_decryption=False): WithDecryption=with_decryption ) return response['Parameter']['Value'] - except self.client.exceptions.ParameterNotFound: + except self.client.exceptions.ParameterNotFound as error: raise ParameterNotFoundError( 'Parameter {0} Not Found'.format(name) - ) + ) from error diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt index 02b26dee9..048e1dd0f 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt @@ -1,56 +1,60 @@ # Install libs here that you might want in AWS CodeBuild -pytest==3.0.7 -mock==2.0.0 -boto3~=1.10, >=1.10.47 -pyyaml>=5.1 -schema==0.7.1 -jsii==1.1.0 -aws_cdk.aws_codepipeline==1.32 -aws_cdk.aws_lambda_event_sources==1.32 -aws-cdk.aws-codepipeline-actions==1.32 -aws_cdk.aws_codebuild==1.32 -aws_cdk.aws_codecommit==1.32 -aws-cdk.aws-s3==1.32 -aws-cdk-assets==1.32 -aws-cdk.aws-iam==1.32 -aws-cdk.aws-kms==1.32 -aws-cdk.aws-lambda==1.32 -aws-cdk.aws-sns==1.32 -aws-cdk.aws_secretsmanager==1.32 -aws-cdk.aws_ssm==1.32 -aws-cdk.aws_events==1.32 -aws_cdk.aws_cloudformation==1.32 -aws_cdk.aws_certificatemanager==1.32 -aws_cdk.aws_sqs==1.32 -aws_cdk.aws_ec2==1.32 -aws_cdk.aws_sns==1.32 -aws_cdk.aws_elasticloadbalancing==1.32 -aws_cdk.aws_s3_notifications==1.32 -aws_cdk.aws_codedeploy==1.32 -aws_cdk.aws_ecr==1.32 -aws_cdk.aws_stepfunctions==1.32 -aws_cdk.aws_servicediscovery==1.32 -aws_cdk.aws_autoscaling_hooktargets==1.32 -aws_cdk.aws_cloudfront==1.32 -aws_cdk.aws_route53_targets==1.32 -aws_cdk.aws_route53==1.32 -aws_cdk.aws_ecs==1.32 -aws_cdk.aws_events_targets==1.32 -aws_cdk.aws_dynamodb==1.32 -aws_cdk.aws_applicationautoscaling==1.32 -aws_cdk.aws_autoscaling==1.32 -aws_cdk.aws_autoscaling_common==1.32 -aws_cdk.aws_s3_assets==1.32 -aws_cdk.aws_ecr_assets==1.32 -aws_cdk.aws_elasticloadbalancingv2==1.32 -aws_cdk.aws_apigateway==1.32 -aws_cdk.aws_logs==1.32 -aws_cdk.aws_ecr==1.32 -aws_cdk.aws_cloudwatch==1.32 -aws_cdk.aws_kinesis==1.32 -aws_cdk.aws_s3_assets==1.32 -aws_cdk.aws_sns_subscriptions==1.32 -aws-cdk.aws-sam==1.32 -aws-cdk.cx-api==1.32 -aws-cdk.region-info==1.32 -aws-cdk.core==1.32 +aws-cdk-assets==1.88 +aws-cdk.aws-apigateway==1.88 +aws-cdk.aws-applicationautoscaling==1.88 +aws-cdk.aws-autoscaling==1.88 +aws-cdk.aws-autoscaling_common==1.88 +aws-cdk.aws-autoscaling_hooktargets==1.88 +aws-cdk.aws-certificatemanager==1.88 +aws-cdk.aws-cloudformation==1.88 +aws-cdk.aws-cloudfront==1.88 +aws-cdk.aws-cloudwatch==1.88 +aws-cdk.aws-codebuild==1.88 +aws-cdk.aws-codecommit==1.88 +aws-cdk.aws-codedeploy==1.88 +aws-cdk.aws-codepipeline-actions==1.88 +aws-cdk.aws-codepipeline==1.88 +aws-cdk.aws-dynamodb==1.88 +aws-cdk.aws-ec2==1.88 +aws-cdk.aws-ecr==1.88 +aws-cdk.aws-ecr==1.88 +aws-cdk.aws-ecr_assets==1.88 +aws-cdk.aws-ecs==1.88 +aws-cdk.aws-elasticloadbalancing==1.88 +aws-cdk.aws-elasticloadbalancingv2==1.88 +aws-cdk.aws-events==1.88 +aws-cdk.aws-events_targets==1.88 +aws-cdk.aws-iam==1.88 +aws-cdk.aws-kinesis==1.88 +aws-cdk.aws-kms==1.88 +aws-cdk.aws-lambda==1.88 +aws-cdk.aws-lambda_event_sources==1.88 +aws-cdk.aws-logs==1.88 +aws-cdk.aws-route53==1.88 +aws-cdk.aws-route53_targets==1.88 +aws-cdk.aws-s3==1.88 +aws-cdk.aws-s3_assets==1.88 +aws-cdk.aws-s3_assets==1.88 +aws-cdk.aws-s3_notifications==1.88 +aws-cdk.aws-sam==1.88 +aws-cdk.aws-secretsmanager==1.88 +aws-cdk.aws-servicediscovery==1.88 +aws-cdk.aws-sns==1.88 +aws-cdk.aws-sns==1.88 +aws-cdk.aws-sns_subscriptions==1.88 +aws-cdk.aws-sqs==1.88 +aws-cdk.aws-ssm==1.88 +aws-cdk.aws-stepfunctions==1.88 +aws-cdk.core==1.88 +aws-cdk.cx-api==1.88 +aws-cdk.region-info==1.88 +awscli==1.18.140 +aws-sam-cli==1.15.0 +boto3==1.14.63 +botocore==1.17.63 +jsii<2.0.0,>=1.16.0 +mock~=4.0.3 +pytest~=6.2.1 +pyyaml>=5.3 +schema~=0.7.2 +urllib3~=1.25.11 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/resolver.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/resolver.py index b2c4a066c..3fea010df 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/resolver.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/resolver.py @@ -35,11 +35,13 @@ def _is_optional(value): def fetch_stack_output(self, value, key, optional=False): # pylint: disable=too-many-statements try: [_, account_id, region, stack_name, output_key] = str(value).split(':') - except ValueError: + except ValueError as error: raise ValueError( - "{0} is not a valid import string." - "syntax should be import:account_id:region:stack_name:output_key".format(str(value)) - ) + "{0} is not a valid import string. Syntax should be " + "import:account_id:region:stack_name:output_key".format( + str(value), + ) + ) from error if Resolver._is_optional(output_key): LOGGER.info("Parameter %s is considered optional", output_key) optional = True @@ -84,12 +86,15 @@ def fetch_stack_output(self, value, key, optional=False): # pylint: disable=too- ) ) self.stage_parameters[parent_key][key] = stack_output - except IndexError: + except IndexError as error: if stack_output: if self.stage_parameters.get(key): self.stage_parameters[key] = stack_output else: - raise Exception("Could not determine the structure of the file in order to import from CloudFormation") + raise Exception( + "Could not determine the structure of the file in order " + "to import from CloudFormation", + ) from error return True def upload(self, value, key, file_name): diff --git a/src/lambda_codebase/initial_commit/requirements.txt b/src/lambda_codebase/initial_commit/requirements.txt index 9a0d8237e..05aca1e04 100644 --- a/src/lambda_codebase/initial_commit/requirements.txt +++ b/src/lambda_codebase/initial_commit/requirements.txt @@ -1,3 +1,3 @@ -Jinja2~=2.10.1 +Jinja2~=2.11.2 +boto3==1.14.63 cfn-custom-resource~=1.0.1 -boto3~=1.10, >=1.10.47 diff --git a/src/template.yml b/src/template.yml index 1792eaca8..19880619e 100644 --- a/src/template.yml +++ b/src/template.yml @@ -95,7 +95,7 @@ Resources: Type: "AWS::Serverless::LayerVersion" Properties: ContentUri: "./lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/" - CompatibleRuntimes: + CompatibleRuntimes: - python3.7 - python3.8 Description: "Shared Lambda Layer between master and deployment account" @@ -109,16 +109,16 @@ Resources: Principal: "*" LambdaRole: Type: "AWS::IAM::Role" - Properties: - AssumeRolePolicyDocument: + Properties: + AssumeRolePolicyDocument: Version: "2012-10-17" - Statement: + Statement: - Effect: "Allow" - Principal: - Service: + Principal: + Service: - "states.amazonaws.com" - "lambda.amazonaws.com" - Action: + Action: - "sts:AssumeRole" LambdaPolicy: Type: "AWS::IAM::ManagedPolicy" @@ -176,7 +176,7 @@ Resources: ADF_LOG_LEVEL: INFO FunctionName: StackWaiter Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 DetermineEventFunction: Type: 'AWS::Serverless::Function' @@ -197,7 +197,7 @@ Resources: ADF_LOG_LEVEL: INFO FunctionName: DetermineEventFunction Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 CrossAccountExecuteFunction: Type: 'AWS::Serverless::Function' @@ -218,7 +218,7 @@ Resources: ADF_LOG_LEVEL: INFO FunctionName: CrossAccountExecuteFunction Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 600 RoleStackDeploymentFunction: Type: 'AWS::Serverless::Function' @@ -237,7 +237,7 @@ Resources: ADF_LOG_LEVEL: INFO FunctionName: RoleStackDeploymentFunction Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 MovedToRootActionFunction: Type: 'AWS::Serverless::Function' @@ -256,7 +256,7 @@ Resources: ADF_LOG_LEVEL: INFO FunctionName: MovedToRootActionFunction Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 900 UpdateResourcePoliciesFunction: Type: 'AWS::Serverless::Function' @@ -275,11 +275,11 @@ Resources: ADF_LOG_LEVEL: INFO FunctionName: UpdateResourcePoliciesFunction Role: !GetAtt LambdaRole.Arn - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 CloudWatchEventsRule: Type: "AWS::Events::Rule" - Properties: + Properties: Description: Triggers StateMachine on Move OU EventPattern: source: @@ -333,9 +333,9 @@ Resources: Statement: - Effect: "Allow" Principal: - Service: + Service: - "codebuild.amazonaws.com" - Action: + Action: - "sts:AssumeRole" ManagedPolicyArns: - !Ref "CodeBuildPolicy" @@ -417,7 +417,7 @@ Resources: - "arn:aws:s3:::adf-shared-modules-*-*/*" - Effect: "Allow" Action: - - "codebuild:*" + - "codebuild:*" Resource: - !Sub "arn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:project/aws-deployment-framework-base-templates" - Effect: "Allow" @@ -446,7 +446,7 @@ Resources: Environment: ComputeType: "BUILD_GENERAL1_LARGE" PrivilegedMode: false - Image: "aws/codebuild/standard:2.0" + Image: "aws/codebuild/standard:5.0" EnvironmentVariables: - Name: ADF_VERSION Value: 3.0.6 @@ -460,20 +460,20 @@ Resources: Value: !Ref AWS::AccountId - Name: DEPLOYMENT_ACCOUNT_BUCKET Value: !GetAtt SharedModulesBucketName.Value - - Name: ORGANIZATION_ID + - Name: ORGANIZATION_ID Value: !GetAtt Organization.OrganizationId - - Name: ADF_LOG_LEVEL + - Name: ADF_LOG_LEVEL Value: INFO Type: LINUX_CONTAINER Name: 'aws-deployment-framework-base-templates' ServiceRole: !Ref CodeBuildRole Source: - BuildSpec: !Sub | + BuildSpec: !Sub | version: 0.2 phases: install: runtime-versions: - python: 3.7 + python: 3.8 pre_build: commands: - apt-get update -qq @@ -734,9 +734,9 @@ Resources: Description: "ADF Lambda Function - BootstrapCreateInitialCommitFunction" Policies: - Version: "2012-10-17" - Statement: + Statement: - Effect: Allow - Action: + Action: - codecommit:CreateBranch - codecommit:CreateCommit - codecommit:CreatePullRequest @@ -745,7 +745,7 @@ Resources: - codecommit:GetDifferences Resource: !GetAtt CodeCommitRepository.Arn FunctionName: BootstrapCreateInitialCommitFunction - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 SharedModulesBucket: Type: Custom::CrossRegionBucket @@ -773,14 +773,14 @@ Resources: - cloudformation.amazonaws.com SharedModulesBucketName: Type: AWS::SSM::Parameter - Properties: + Properties: Description: DO NOT EDIT - Used by The AWS Deployment Framework Name: shared_modules_bucket Type: String Value: !GetAtt SharedModulesBucket.BucketName LogLevelSetting: Type: AWS::SSM::Parameter - Properties: + Properties: Description: DO NOT EDIT - Used by The AWS Deployment Framework Name: adf_log_level Type: String @@ -793,7 +793,7 @@ Resources: Description: "ADF Lambda Function - Create Deployment Bucket in Main Deployment Region" Policies: - Version: "2012-10-17" - Statement: + Statement: - Effect: Allow Action: s3:CreateBucket Resource: "*" @@ -809,7 +809,7 @@ Resources: - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/shared_modules_bucket" - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/deployment_account_region" FunctionName: CrossRegionBucketHandler - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 Organization: Type: Custom::Organization @@ -823,7 +823,7 @@ Resources: Description: "ADF Lambda Function - Enable AWS Organizations" Policies: - Version: "2012-10-17" - Statement: + Statement: - Effect: Allow Action: - "organizations:CreateOrganization" @@ -835,7 +835,7 @@ Resources: Action: "iam:CreateServiceLinkedRole" Resource: "arn:aws:iam::*:role/aws-service-role/*" FunctionName: AwsOrganizationsHandler - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 DeploymentOrganizationUnit: Type: Custom::OrganizationUnit @@ -851,7 +851,7 @@ Resources: Description: "ADF Lambda Function - Create Organization Unit" Policies: - Version: "2012-10-17" - Statement: + Statement: - Effect: Allow Action: - "organizations:CreateOrganizationalUnit" @@ -859,7 +859,7 @@ Resources: - "organizations:ListOrganizationalUnitsForParent" Resource: "*" FunctionName: OrganizationUnitHandler - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 DeploymentAccount: Type: Custom::Account @@ -878,7 +878,7 @@ Resources: Description: "ADF Lambda Function - Create Account" Policies: - Version: "2012-10-17" - Statement: + Statement: - Effect: Allow Action: - "organizations:CreateAccount" @@ -888,7 +888,7 @@ Resources: Action: ssm:GetParameter Resource: !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/deployment_account_id" FunctionName: AccountHandler - Runtime: python3.7 + Runtime: python3.8 Timeout: 300 PipelineCloudWatchEventRole: Type: AWS::IAM::Role @@ -909,7 +909,7 @@ Resources: Statement: - Effect: Allow Action: codepipeline:StartPipelineExecution - Resource: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref CodePipeline ] ] + Resource: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref CodePipeline ] ] PipelineCloudWatchEventRule: Type: AWS::Events::Rule Properties: @@ -929,7 +929,7 @@ Resources: referenceName: - master Targets: - - Arn: + - Arn: !Join [ '', [ 'arn:aws:codepipeline:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':', !Ref CodePipeline ] ] RoleArn: !GetAtt PipelineCloudWatchEventRole.Arn Id: adf-codepipeline-trigger-bootstrap @@ -952,4 +952,4 @@ Outputs: Description: "The CodeCommit SSH Url" Value: !GetAtt CodeCommitRepository.CloneUrlSsh Export: - Name: "BaseTemplatesRepoSSHURL" \ No newline at end of file + Name: "BaseTemplatesRepoSSHURL" diff --git a/tox.ini b/tox.ini index 484b891cf..26c89502b 100644 --- a/tox.ini +++ b/tox.ini @@ -4,7 +4,7 @@ # and then run "tox" from this directory. [tox] -envlist = py37 +envlist = py38 skipsdist = True From 21be1c139be4c9e32bdeda570106420afe5a4975 Mon Sep 17 00:00:00 2001 From: Simon Kok Date: Thu, 4 Feb 2021 14:59:15 +0100 Subject: [PATCH 2/3] Limit to jsii to <1.20.0 **Why** CDK generate the CloudFormation changes, using jsii v1.20.0 generated an issue where it reported: `ValueError: write to closed file`. **What?** CDK 1.88.0 is compatible with older versions of jsii, so relying on that instead. --- .../bootstrap_repository/adf-build/requirements.txt | 2 +- .../bootstrap_repository/adf-build/shared/requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt index 0845fdc52..00f676e6b 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/requirements.txt @@ -4,7 +4,7 @@ awscli==1.18.140 aws-sam-cli==1.15.0 botocore==1.17.63 boto3==1.14.63 -jsii<2.0.0,>=1.16.0 +jsii<1.20.0,>=1.16.0 mock~=4.0.3 pip~=20.2.3 pylint~=2.6.0 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt index 048e1dd0f..bc55bccf5 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/requirements.txt @@ -52,7 +52,7 @@ awscli==1.18.140 aws-sam-cli==1.15.0 boto3==1.14.63 botocore==1.17.63 -jsii<2.0.0,>=1.16.0 +jsii<1.20.0,>=1.16.0 mock~=4.0.3 pytest~=6.2.1 pyyaml>=5.3 From c78ac9cfb597f8bf3188e6ad70f67962b32e8043 Mon Sep 17 00:00:00 2001 From: Simon Kok Date: Thu, 4 Feb 2021 13:26:42 +0100 Subject: [PATCH 3/3] Remove apt-get update, rely on repeatable packages in the image **Why?** The `apt-get update` could fail as it fetches the latest updates that might break the build. Instead, we should rely on the packages that have been installed in the CodeBuild image instead. **What?** Changed to rely on the latest STANDARD:5.0 container image packages. **References** * Fix for issue #325 --- src/template.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/src/template.yml b/src/template.yml index 19880619e..297af4cf0 100644 --- a/src/template.yml +++ b/src/template.yml @@ -476,7 +476,6 @@ Resources: python: 3.8 pre_build: commands: - - apt-get update -qq - pip install -r adf-build/requirements.txt --quiet - pytest -vvv build: