Merge branch 'master' into feature/387

Author: sbkok

docs/admin-guide.md

@@ -320,18 +320,37 @@ pipelines:
 ```
 
 ## Service Control Policies
-Service control policies *(SCPs)* are one type of policy that you can use to manage your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization, allowing you to ensure your accounts stay within your organization’s access control guidelines. ADF allows SCPs to be applied in a similar fashion as base stacks. You can define your SCP definition in a file named `scp.json` and place it in a folder that represents your Organizational Unit within the `adf-bootstrap` folder from the `aws-deployment-framework-bootstrap` repository on the Master Account.
+Service control policies *(SCPs)* are one type of policy that you can use to manage your organization. SCPs offer central control over the maximum available permissions for all accounts in your organization, allowing you to ensure your accounts stay within your organization’s access control guidelines. ADF allows SCPs to be applied in a similar fashion as base stacks. You can define your SCP definition in a file named `scp.json` and place it in a folder that represents your Organizational Unit (or OU/AccountName path if you are wanting to apply an account-specific SCP) within the `adf-bootstrap` folder from the `aws-deployment-framework-bootstrap` repository on the Master Account.
+
+For example, if you have an account named `my_banking_account` under the `banking/dev` OU that needs a specific SCP, and another SCP defined for the whole `deployment` OU, the folder structure would look like this:
+
+```
+adf-bootstrap <-- This folder lives in the aws-deployment-framework-bootstrap repository on the master account.
+│
+└─── deployment
+│    |
+│    └─── scp.json
+│
+│───banking
+│   │
+│   └─── dev
+│         │
+│         └─── my_banking_account
+│               └─── scp.json
+│
+```
+The file `adf-bootstrap/deployment/scp.json` applies the defined SCP to the `deployment` *OU*, while the file `adf-bootstrap/banking/dev/my_backing_account/scp.json` applies the defined SCP to the `my_banking_account` *account*.
 
 SCPs are available only in an organization that has [all features enabled](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_org_support-all-features.html). Once you have enabled all features within your Organization, ADF can manage and automate the application and updating process of the SCPs.
 
 ## Tagging Policies
-Tag Policies are a feature that allows you to define rules on how tags can be used on AWS resources in your accounts in AWS Organizations. You can use Tag Policies to easily adopt a standardized approach for tagging AWS resources. You can define your Tagging Policy definition in a file named `tagging-policy.json` and place it in a folder that represents your Organizational Unit within the `adf-bootstrap` folder from the `aws-deployment-framework-bootstrap` repository on the Master Account.
+Tag Policies are a feature that allows you to define rules on how tags can be used on AWS resources in your accounts in AWS Organizations. You can use Tag Policies to easily adopt a standardized approach for tagging AWS resources. You can define your Tagging Policy definition in a file named `tagging-policy.json` and place it in a folder that represents your Organizational Unit within the `adf-bootstrap` folder from the `aws-deployment-framework-bootstrap` repository on the Master Account. Tagging policies can also be applied to single account using the same approach described above for SCPs.
 
 Tag Policies are available only in an organization that has [all features enabled](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_org_support-all-features.html). Once you have enabled all features within your Organization, ADF can manage and automate the application and updating process of the Tag Policies. For more information, see [here](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies.html).
 
 
 ## Integrating Slack
-
+### Integrating with Slack using Lambda
 The ADF allows alternate *notification_endpoint* values that can be used to notify the status of a specific pipeline *(in deployment_map.yml)*. You can specify an email address in the deployment map and notifications will be emailed directly to that address. However, if you specify a slack channel name *(eg team-bugs)* as the value, the notifications will be forwarded to that channel. In order to setup this integration you will need to create a [Slack App](https://api.slack.com/apps). When you create your Slack app, you can create multiple Webhook URL's *(Incoming Webhook)* that are each associated with their own channel. Create a webhook for each channel you plan on using throughout your Organization. Once created, copy the webhook URL and create a new secret in Secrets Manager on the Deployment Account:
 
 1. In AWS Console, click _Store a new secret_ and select type 'Other type of secrets' *(eg API Key)*.
@@ -362,6 +381,22 @@ pipelines:
         name: omg_production
 ```
 
+### Integrating with Slack with AWS ChatBot
+The ADF also supports integrating pipeline notifications with Slack via the AWS ChatBot. This allows pipeline notifications to scale and provides a consistent Slack notification across different AWS services. 
+
+In order to use AWS ChatBot, first you must configure an (AWS ChatBot Client)[https://us-east-2.console.aws.amazon.com/chatbot/home?region=eu-west-1#/chat-clients] for your desired Slack workspace. Once the client has been created. You will need to manually create a channel configuration that will be used by the ADF. 
+
+Currently, dynamically creating channel configurations is not supported. In the deployment map, you can configure a unique channel via the notification endpoint parameter for each pipeline separately. Add the `params` section if that is missing and add the following configuration to the pipeline:
+```
+pipelines:
+  - name: some-pipeline
+    # ...
+    params:
+      notification_endpoint: 
+        type: chat_bot
+        target: my_channel_config
+```
+
 ## Check Current Version
 
 To determine the current version, follow these steps:

docs/user-guide.md

@@ -195,8 +195,16 @@ Pipelines also have parameters that don't relate to a specific stage but rather
 
 The following are the available pipeline parameters:
 
-- *notification_endpoint* *(String)* defaults to none.
-  > Can either be a valid email address or a string that represents the name of a Slack Channel. In order to integrate ADF with Slack see [Integrating with Slack](./admin-guide.md) in the admin guide. By Default, Notifications will be sent when pipelines Start, Complete or Fail.
+- *notification_endpoint* *(String) | (Dict) * defaults to none.
+  > Can either be a valid email address or a string that represents the name of a Slack Channel. 
+  > A more complex configuration can be provided to integrate with Slack via AWS ChatBot. 
+  > ```yaml
+  > notification_endpoint:
+  >   type: chat_bot
+  >   target: example_slack_channel  # This is the name of an slack channel configuration you created within the AWS Chat Bot service. This needs to be created before you apply the changes to the deployment map.
+  > ```
+  >
+  > In order to integrate ADF with Slack see [Integrating with Slack](./admin-guide.md#integrating-with-slack-with-aws-chatbot) in the admin guide. By default, notifications will be sent when pipelines Start, Complete, or Fail.
 
 - *schedule* *(String)* defaults to none.
   > If the Pipeline should execute on a specific Schedule. Schedules are defined by using a Rate or an Expression. See [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html#RateExpressions) for more information on how to define Rate or an Expression.

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/provisioner/src/configparser.py

@@ -8,17 +8,41 @@
 
 import os
 import yaml
+from logger import configure_logger
 from .account import Account
 
 
+LOGGER = configure_logger(__name__)
+
+
 def read_config_files(folder):
     files = [os.path.join(folder, f) for f in os.listdir(folder)]
     accounts = []
     for filename in files:
-        if filename.endswith(".yml"):
-            with open(filename, 'r') as stream:
-                config = yaml.safe_load(stream)
-                for account in config.get('accounts', []):
-                    accounts.append(Account.load_from_config(account))
+        if not filename.endswith(".yml"):
+            # Skipping files that do not end with .yml
+            continue
+        accounts.extend(_read_config_file(filename))
 
     return accounts
+
+
+def _read_config_file(filename):
+    accounts = []
+    try:
+        with open(filename, 'r') as stream:
+            config = yaml.safe_load(stream)
+            for account in config.get('accounts', []):
+                accounts.append(Account.load_from_config(account))
+        return accounts
+    except Exception as error:
+        LOGGER.error(
+            "Could not process %s due to an error: %s. ",
+            filename,
+            error
+        )
+        LOGGER.error(
+            "Make sure the content of YAML files (.yml) are not empty and "
+            "contain a valid YAML data structure.",
+        )
+        raise

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_chatbot.py

@@ -0,0 +1,68 @@
+# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+"""Construct related to Notifications Codepipeline Input
+"""
+
+import os
+from aws_cdk import (
+    aws_codestarnotifications as cp_notifications,
+    aws_codepipeline as codepipeline,
+    core,
+)
+from logger import configure_logger
+
+ADF_DEPLOYMENT_REGION = os.environ["AWS_REGION"]
+ADF_DEPLOYMENT_ACCOUNT_ID = os.environ["ACCOUNT_ID"]
+
+LOGGER = configure_logger(__name__)
+
+EVENT_TYPE_IDS = [
+    "codepipeline-pipeline-stage-execution-succeeded",
+    "codepipeline-pipeline-stage-execution-failed",
+    "codepipeline-pipeline-pipeline-execution-started",
+    "codepipeline-pipeline-pipeline-execution-failed",
+    "codepipeline-pipeline-pipeline-execution-succeeded",
+    "codepipeline-pipeline-manual-approval-needed",
+    "codepipeline-pipeline-manual-approval-succeeded",
+]
+
+
+class PipelineNotifications(core.Construct):
+    def __init__(
+        self,
+        scope: core.Construct,
+        id: str,
+        pipeline: codepipeline.CfnPipeline,
+        notification_config,
+        **kwargs,
+    ):  # pylint: disable=W0622
+        super().__init__(scope, id, **kwargs)
+        slack_channel_arn = f"arn:aws:chatbot::{ADF_DEPLOYMENT_ACCOUNT_ID}:chat-configuration/slack-channel/{notification_config.get('target')}"
+        pipeline_arn = f"arn:aws:codepipeline:{ADF_DEPLOYMENT_REGION}:{ADF_DEPLOYMENT_ACCOUNT_ID}:{pipeline.ref}"
+        cp_notifications.CfnNotificationRule(
+            scope,
+            "pipeline-notification",
+            detail_type="FULL",
+            event_type_ids=EVENT_TYPE_IDS,
+            name=pipeline.ref,
+            resource=pipeline_arn,
+            targets=[
+                cp_notifications.CfnNotificationRule.TargetProperty(
+                    target_type=PipelineNotifications.get_target_type_from_config(
+                        scope, notification_config
+                    ),
+                    target_address=slack_channel_arn,
+                )
+            ],
+        )
+
+    @staticmethod
+    def get_target_type_from_config(scope, config):
+        target_type = config.get("type", "chat_bot")
+        if target_type == "chat_bot":
+            return "AWSChatbotSlack"
+        scope.node.add_error(
+            f"{target_type} is not supported for CodePipeline notifications."
+        )
+        return None

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_notifications.py

@@ -21,7 +21,9 @@
 
 
 class Notifications(core.Construct):
-    def __init__(self, scope: core.Construct, id: str, map_params: dict, **kwargs): #pylint: disable=W0622
+    def __init__(
+        self, scope: core.Construct, id: str, map_params: dict, **kwargs
+    ):  # pylint: disable=W0622
         super().__init__(scope, id, **kwargs)
         LOGGER.debug('Notification configuration required for %s', map_params['name'])
         stack = core.Stack.of(self)
@@ -32,42 +34,36 @@ def __init__(self, scope: core.Construct, id: str, map_params: dict, **kwargs):
             f'arn:{stack.partition}:lambda:{ADF_DEPLOYMENT_REGION}:'
             f'{ADF_DEPLOYMENT_ACCOUNT_ID}:function:SendSlackNotification'
         )
-        _topic = _sns.Topic(self, 'PipelineTopic')
+        _topic = _sns.Topic(self, "PipelineTopic")
         _statement = _iam.PolicyStatement(
             actions=["sns:Publish"],
             effect=_iam.Effect.ALLOW,
             principals=[
-                _iam.ServicePrincipal(
-                    'sns.amazonaws.com'
-                ),
-                _iam.ServicePrincipal(
-                    'codecommit.amazonaws.com'
-                ),
-                _iam.ServicePrincipal(
-                    'events.amazonaws.com'
-                )
+                _iam.ServicePrincipal("sns.amazonaws.com"),
+                _iam.ServicePrincipal("codecommit.amazonaws.com"),
+                _iam.ServicePrincipal("events.amazonaws.com"),
             ],
-            resources=["*"]
+            resources=["*"],
         )
         _topic.add_to_resource_policy(_statement)
-        _lambda.CfnPermission(
-            self,
-            'slack_notification_sns_permissions',
-            principal='sns.amazonaws.com',
-            action='lambda:InvokeFunction',
-            source_arn=_topic.topic_arn,
-            function_name='SendSlackNotification'
-        )
-        _endpoint = map_params.get('params', {}).get('notification_endpoint', '')
+        _endpoint = map_params.get("params", {}).get("notification_endpoint", "")
         _sub = _sns.Subscription(
             self,
-            'sns_subscription',
+            "sns_subscription",
             topic=_topic,
-            endpoint=_endpoint if '@' in _endpoint else _slack_func.function_arn,
-            protocol=_sns.SubscriptionProtocol.EMAIL if '@' in _endpoint else _sns.SubscriptionProtocol.LAMBDA
+            endpoint=_endpoint if "@" in _endpoint else _slack_func.function_arn,
+            protocol=_sns.SubscriptionProtocol.EMAIL
+            if "@" in _endpoint
+            else _sns.SubscriptionProtocol.LAMBDA,
         )
-        if '@' not in _endpoint:
-            _slack_func.add_event_source(
-                source=_event_sources.SnsEventSource(_topic)
+        if "@" not in _endpoint:
+            _lambda.CfnPermission(
+                self,
+                "slack_notification_sns_permissions",
+                principal="sns.amazonaws.com",
+                action="lambda:InvokeFunction",
+                source_arn=_topic.topic_arn,
+                function_name="SendSlackNotification",
             )
+            _slack_func.add_event_source(source=_event_sources.SnsEventSource(_topic))
         self.topic_arn = _topic.topic_arn

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/adf_default_pipeline.py

@@ -16,6 +16,7 @@
 from cdk_constructs import adf_s3
 from cdk_constructs import adf_cloudformation
 from cdk_constructs import adf_notifications
+from cdk_constructs import adf_chatbot
 from logger import configure_logger
 
 ADF_DEPLOYMENT_REGION = os.environ["AWS_REGION"]
@@ -29,10 +30,10 @@
 def generate_adf_default_pipeline(scope: core.Stack, stack_input):
     _stages = []
 
-    if stack_input["input"].get("params", {}).get("notification_endpoint"):
-        stack_input["input"]["topic_arn"] = adf_notifications.Notifications(
-            scope, "adf_notifications", stack_input["input"]
-        ).topic_arn
+    notification_config = stack_input["input"].get("params", {}).get("notification_endpoint", {})
+
+    if isinstance(notification_config, str) or notification_config.get('type', '') == "lambda":
+        stack_input["input"]["topic_arn"] = adf_notifications.Notifications(scope, "adf_notifications", stack_input["input"]).topic_arn
 
     _source_name = generate_source_stage_for_pipeline(_stages, scope, stack_input)
     generate_build_stage_for_pipeline(_stages, scope, stack_input)
@@ -41,9 +42,12 @@ def generate_adf_default_pipeline(scope: core.Stack, stack_input):
     _pipeline = adf_codepipeline.Pipeline(
         scope, "code_pipeline", stack_input["input"], stack_input["ssm_params"], _stages
     )
+
     if "github" in _source_name:
         adf_github.GitHub.create_webhook_when_required(scope, _pipeline.cfn, stack_input["input"])
 
+    if isinstance(notification_config, dict) and notification_config.get('type', '') == 'chat_bot':
+        adf_chatbot.PipelineNotifications(scope, "adf_chatbot_notifications", _pipeline.cfn, notification_config)
 
 def generate_source_stage_for_pipeline(_stages, scope, stack_input):
     _source_name = stack_input["input"]["default_providers"]["source"][

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_stacks/tests/test_pipeline_creation.py

@@ -194,3 +194,50 @@ def test_pipeline_creation_outputs_as_expected_when_source_is_codecommit_with_co
 
     assert len(build_stage['Actions']) == 1
 
+
+def test_pipeline_creation_outputs_as_expected_when_notification_endpoint_is_chatbot():
+    region_name = "eu-central-1"
+    acount_id = "123456789012"
+
+    stack_input = {
+        "input": {"params": {"notification_endpoint": {"target": "fake-config", "type": "chat_bot"}}, "default_providers": {}, "regions": {}, },
+        "ssm_params": {"fake-region": {}},
+    }
+
+    stack_input["input"]["name"] = "test-stack"
+
+    stack_input["input"]["default_providers"]["source"] = {
+        "provider": "codecommit",
+        "properties": {"account_id": "123456789012"},
+    }
+    stack_input["input"]["default_providers"]["build"] = {
+        "provider": "codebuild",
+        "properties": {"account_id": "123456789012"},
+    }
+
+    stack_input["ssm_params"][region_name] = {
+        "modules": "fake-bucket-name",
+        "kms": f"arn:aws:kms:{region_name}:{acount_id}:key/my-unique-kms-key-id",
+    }
+    app = core.App()
+    PipelineStack(app, stack_input)
+
+    cloud_assembly = app.synth()
+    resources = {k[0:-8]: v for k, v in cloud_assembly.stacks[0].template['Resources'].items()}
+    pipeline_notification = resources['pipelinenoti']['Properties']
+
+    target = pipeline_notification["Targets"][0]
+    
+    assert resources["pipelinenoti"]["Type"] == "AWS::CodeStarNotifications::NotificationRule"
+    assert target["TargetAddress"] == "arn:aws:chatbot::111111111111:chat-configuration/slack-channel/fake-config"
+    assert target["TargetType"] == "AWSChatbotSlack"
+    assert pipeline_notification["EventTypeIds"] == [
+            "codepipeline-pipeline-stage-execution-succeeded",
+            "codepipeline-pipeline-stage-execution-failed",
+            "codepipeline-pipeline-pipeline-execution-started",
+            "codepipeline-pipeline-pipeline-execution-failed",
+            "codepipeline-pipeline-pipeline-execution-succeeded",
+            "codepipeline-pipeline-manual-approval-needed",
+            "codepipeline-pipeline-manual-approval-succeeded"
+         ]
+    assert pipeline_notification["DetailType"] == "FULL"