Skip to content

Sync Step Function input files when content changed only with exec id metadata #530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sbkok merged 8 commits into from
Sep 20, 2022
Merged

Sync Step Function input files when content changed only with exec id metadata #530

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
dfd65e8
Sync SFn input files when content changed only with exec id metadata
sbkok Sep 12, 2022
cfffdc7
Move sync_to_s3 to shared helpers
sbkok Sep 14, 2022
7f4f158
Fix linting issues
sbkok Sep 14, 2022
35070e9
Add helper requirements
sbkok Sep 14, 2022
8131878
Code review changes
sbkok Sep 15, 2022
162c264
Add support for syncing multiple file extensions with sync_to_s3.py
sbkok Sep 15, 2022
ddab966
Add CodePipeline Execution Id to accounts & pipeline gen
sbkok Sep 16, 2022
6e14fe2
Add ADF Version to S3 file sync, such that an ADF update triggers upd…
sbkok Sep 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ test:
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pytest.ini
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/pytest.ini
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/determine_default_branch -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/determine_default_branch/pytest.ini
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/pytest.ini
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/pytest.ini
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/pytest.ini
pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/pytest.ini
Expand Down
17 changes: 14 additions & 3 deletions src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -766,11 +766,14 @@ Resources:
commands:
- npm install [email protected] -g -y --quiet --no-progress
- aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --quiet
- pip install -r adf-build/requirements.txt -q -t ./adf-build
- pip install -r adf-build/requirements.txt -r adf-build/helpers/requirements.txt -q -t ./adf-build
pre_build:
commands:
- mkdir -p deployment_maps
build:
commands:
- bash -c "[[ -e deployment_map.yml ]] && echo 'Copying deployment_map.yml' && aws s3 cp deployment_map.yml s3://$ADF_PIPELINES_BUCKET/deployment_map.yml || echo 'No deployment_map.yml, skipping copy'"
- bash -c "[[ -e deployment_maps ]] && echo 'Syncing deployment_maps folder' && aws s3 sync deployment_maps s3://$ADF_PIPELINES_BUCKET/deployment_maps || echo 'No deployment_maps folder, skipping sync'"
- python adf-build/helpers/sync_to_s3.py --metadata adf_version=${!ADF_VERSION} --upload-with-metadata execution_id=${!CODEPIPELINE_EXECUTION_ID} deployment_map.yml s3://$ADF_PIPELINES_BUCKET/deployment_map.yml
- python adf-build/helpers/sync_to_s3.py --extension .yml --extension .yaml --metadata adf_version=${!ADF_VERSION} --upload-with-metadata execution_id=${!CODEBUILD_BUILD_NUMBER} --recursive deployment_maps s3://$ADF_PIPELINES_BUCKET/deployment_maps
post_build:
commands:
- echo "Pipelines are updated in the AWS Step Functions ADFPipelineManagementStateMachine."
Expand Down Expand Up @@ -820,6 +823,14 @@ Resources:
- Name: "Source"
Configuration:
ProjectName: !Ref CodeBuildProject
EnvironmentVariables: >-
[
{
"name": "CODEPIPELINE_EXECUTION_ID",
"value": "#{codepipeline.PipelineExecutionId}",
"type": "PLAINTEXT"
}
]
RunOrder: 1

PipelineSNSTopic:
Expand Down
2 changes: 1 addition & 1 deletion ...sitory/adf-bootstrap/deployment/lambda_codebase/determine_default_branch/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
boto3==1.18.2
boto3==1.24.59
cfn-custom-resource~=1.0.1
2 changes: 2 additions & 0 deletions src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0
3 changes: 3 additions & 0 deletions src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/pytest.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[pytest]
testpaths = tests
norecursedirs = terraform
Loading