Service Discovery Design

[kohidave]

I'm really interested in integrating service discovery, by default for our apps. This design doesn't include App Mesh integration, but I think that'd be additive.

What is service discovery?

Service discovery allows two services to communicate using an internal (within the same VPC) DNS.

An example would be, if you have two apps, backend and frontend, the frontend app might have a public loadbancer, but the backend might be isolated in a private subnet.

The frontend would be able to communicate with the backend service using a VPC-local DNS like backend.local. The backend would also be able to contact the frontend through a VPC-local DNS in the same way. This has the benefit of the data not transiting the internet and the inter-service communicate doesn't rely on the LB url.

What services to use?

• AWS CloudMap
• ECS's ServiceDiscovery (same thing, but handles the registering of tasks into CloudMap)

What do we have to do?

It's actually really easy.

1. In the environment, create a ServiceDiscovery Namespace (this is the DNS suffix). I'm thinking .local.

  ServiceDiscoveryNamespace:
    Type: AWS::ServiceDiscovery::PrivateDnsNamespace
    Properties:
        Name: local
        Vpc: !Ref VPC

2. Export the Namespace ID

3. In the app stack, create a discovery service (this is the actual DNS for this app):

  DiscoveryService:
    Type: AWS::ServiceDiscovery::Service
    Properties:
      Description: Discovery Service for the ECS CLI V2 services
      DnsConfig:
        RoutingPolicy: MULTIVALUE
        DnsRecords:
          - TTL: 60
            Type: A
          - TTL: 60
            Type: SRV
      HealthCheckCustomConfig:
        FailureThreshold: 1
      Name:  !Ref AppName
      NamespaceId:
               Fn::ImportValue:
               !Sub '${ProjectName}-${EnvName}-NamespaceID'

4. Update the service to use ServiceRegistries

      ServiceRegistries:
        - RegistryArn: !GetAtt DiscoveryService.Arn
          Port: !Ref ContainerPort

5. This will set up everything, but one tricky thing is our security groups don't permit inter-app communication right now (since each app gets its own security group and each security group only allows traffic from the ALB and itself. So, what we could do is move the security group up to the env, and have each app use the same security group. This is definitely the trickiest part.

6. Pass in dnsSearchDomains to the Task Definition. When we create the .local namespace it'd be rad if customers didn't have to include it in their actual http requests. A feature called Search Domains (which we can set in our task definitions) lets us add automatic suffixes if a request is made without a fully qualified domain. This means if we add a .local search domain, if the customer's container makes a request to backend - it'll be resolved to backend.local.

Example

Calling a backend app from the frontend app. The backend is running on port 8080:

resp, err := http.Get("http://backend:8080/")

[efekarakus]

I am dumping a few questions for me to get a better understanding. I went through the docs but stuff is still not clear to me 😅:

In the environment, create a ServiceDiscovery Namespace (this is the DNS suffix). I'm thinking .local.

Did you fix it to a value like "local" on purpose (instead of say the env name) so that users testing their applications locally will get the same behavior once deployed?

I wonder if we should pass this value as an environment variable to the containers in the task definition?

In the app stack, create a discovery service (this is the actual DNS for this app):

Looking at Nathan's example there is only an SRV record with a much more aggressive TTL. The tutorial instead only has an A record. Finally, the cfn guide which i think this design is based on has both. (I think the default RoutingPolicy is MULTIVALUE so we can drop it, at least that's what happens when I test with the CLI)

Few things that I don't understand:

1. When do we need the SRV records?
   1. What happens when a port that's not 80 is exposed by the container? Is that why we need it?
2. If the RoutingPolicy is MULTIVALUE, how does the application code pick one of the 8 ip addresses ?_?

It is recommended to use container-level health checks managed by Amazon ECS for your service discovery service.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

Does this mean we need to set a healthcheck value for the container in the task? I think we need this for a "backend" pattern, where tasks are not behind a LB.

[efekarakus]

Ok, I think I have a better understanding of the concepts now :D I didn't know anything about DNS, so I hope this helps others too.

1. Service discovery is neat, and allows the application code to make requests like this:

resp, err := http.Get("http://backend.local:8080/")

2. We need to resolve backend.local to a real IP address. Route53 helps us do this by returning to us up to 8 ip addresses (because of MULTIVALUErouting policy) for the service, each record with a TTL set (60 seconds in David's post).
3. One of these ip addresses gets cached locally in the task for the duration of the TTL. Once the TTL expires, another request to Route53 is made.
4. If the downstream task goes down until the TTL expires, our application code will keep receiving errors. So it becomes important to balance the TTL value with cost.

It's important for Route53 to store only ip addresses that are healthy. This is achieved by setting a HealthCheck config to the important containers in our task definition. Whenever a container becomes unhealthy the ECS scheduler reports that back to CloudMap. CloudMap then removes the entry from Route53 after 30 seconds * FailureThreshold:

HealthCheckCustomConfig:
     FailureThreshold: 1

I'm still not sure what's the point of SRV records but it appears that they're used if we were to choose a weighted routing policy instead.

Note that the application still needs to specify the port that's exposed from the container it's making a request to. Service discovery only helps you resolve IP addresses from a logical name.

[kohidave]

Ok! Well, after working with the App Mesh team, we think CloudMap is a good first step. App Mesh features can be added at a later time, and probably as opt in by customers.