Nomad Enterprise HVD on AWS EC2

Terraform module aligned with HashiCorp Validated Designs (HVD) to deploy Nomad Enterprise on Amazon Web Services (AWS) using EC2 instances.

Prerequisites

General

• Knowledge of Terraform and AWS
• Terraform CLI >= 1.9 installed on client/workstations
• An AWS account with permissions to provision resources via Terraform
• git CLI and Visual Studio Code recommended on workstations

Networking

• AWS VPC ID and the following subnets:
  • Subnets for the EC2 instances (compute)

📝 Note: Specify at least two subnets for high availability.

Security groups

• This module will create security groups for EC2 instances.
• Define CIDR ranges for Nomad access, managed via the input variable cidr_allow_ingress_nomad.

TLS certificates

TLS certificates for Nomad have very specific requirements for server and client nodes. Nomad Agent Certificates Let's Encrypt Certificates can not be used for Nomad due to the Nomad specific requirements. This module has the option to not enable TLS with nomad_tls_enabled however this should never be used outside a lab environment.

• TLS certificate (e.g. cert.pem) and private key (e.g. privkey.pem) for the Nomad web UI, in PEM format.
  • TLS private key must not be password-protected.
• TLS certificate authority (CA) bundle (e.g. ca_bundle.pem) in PEM format.
• These TLS files will be stored as secrets in AWS Secrets Manager.

📝 Store these TLS files as AWS Secrets Manager secrets to securely manage your certificates.

Gossip encryption

Gossip Encryption is required for this module. A Gossip Encryption Key can be generated by the Nomad CLI with nomad operator gossip keyring generate. Create this before deploying the module and store this in AWS Secrets Manager.

Secrets management

The following secrets must be stored in AWS Secrets Manager to bootstrap the Nomad Server deployment:

• Nomad license - text string used for Nomad Enterprise licensing - This is only needed for Nomad Server deployment
• Nomad Gossip Encryption Key - text string generated by nomad operator gossip keyring generate - This is only needed for Nomad Server deployment
• Nomad TLS certificate and private key - stored as base64-encoded PEM secrets in AWS Secrets Manager

📝 See the Nomad documentation for more details on securing these secrets in AWS Secrets Manager.

Usage

1. Configure the prerequisites.

2. In the examples directory, you’ll find subdirectories with ready-made Terraform configurations for deploying this module. Select an example that matches your use case, and copy its contents to a new directory.

   📝 Example structure for managing multiple Nomad deployments:

   .
   └── environments
       ├── production
       │   ├── backend.tf
       │   ├── main.tf
       │   ├── outputs.tf
       │   ├── terraform.tfvars
       │   └── variables.tf
       └── sandbox
           ├── backend.tf
           ├── main.tf
           ├── outputs.tf
           ├── terraform.tfvars
           └── variables.tf
   

   📝 This example has two separate Nomad deployments: one for a sandbox environment and one for a production environment.

3. (Optional) If using S3 for remote state, configure the backend.tf file with custom values.

4. Update the terraform.tfvars file with your custom values, then run terraform init, terraform plan, and terraform apply.

5. After terraform apply completes successfully, connect to the Nomad EC2 instance shell using SSH or AWS SSM to monitor the cloud-init logs:

   Connecting to EC2 instance:

   ssh -i /path/to/ec2_ssh_key_pair.pem ubuntu@<ec2-private-ip>

   Viewing logs:

   tail -f /var/log/nomad-cloud-init.log

6. Once the installation finishes, verify the health of the Nomad cluster by checking the Nomad Web UI or by running:

   nomad server members

7. If nomad_acl_enabled is true the Nomad cluster will need to be bootstrapped with the command nomad acl bootstrap. This will generate a bootstrap token that can be used to login to the CLI or UI.

Docs

Additional documentation for managing and customizing your Nomad deployment is available in the docs folder:

• Nomad Version Upgrades
• Nomad TLS Certificate Rotation
• Nomad Configuration Settings
• Nomad Deployment Customizations

Troubleshooting

During deployment the output of the user_data script can be traced in /var/log/cloud-init.log, /var/log/cloud-init-output.log and /var/log/vault-cloud-init.log due to set -xeuo pipefail in the default install-vault.sh.tpl For help debugging cloud init and user data scripts

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#userdata-linux
• https://cloudinit.readthedocs.io/en/latest/howto/debugging.html#cloud-init-ran-but-didn-t-do-what-i-want-it-to

Module support

This open source software is maintained by the HashiCorp Technical Field Organization, independently of our enterprise products. While our Support Engineering team provides dedicated support for our enterprise offerings, this open source software is not included.

• For help using this open source software, please engage your account team.
• To report bugs/issues with this open source software, please open them directly against this code repository using the GitHub issues feature.

Please note that there is no official Service Level Agreement (SLA) for support of this software as a HashiCorp customer. This software falls under the definition of Community Software/Versions in your Agreement. We appreciate your understanding and collaboration in improving our open source projects.

Requirements

Name	Version
terraform	>= 1.9
aws	~> 5.0

Providers

Name	Version
aws	~> 5.0

Resources

Name	Type
aws_autoscaling_group.nomad	resource
aws_iam_instance_profile.nomad_ec2	resource
aws_iam_role.nomad_ec2	resource
aws_iam_role_policy.nomad_ec2	resource
aws_iam_role_policy_attachment.aws_ssm	resource
aws_launch_template.nomad	resource
aws_lb.nlb	resource
aws_lb_listener.nlb_443	resource
aws_lb_listener.nlb_80	resource
aws_lb_target_group.nlb_4646	resource
aws_placement_group.nomad	resource
aws_route53_record.alias_record	resource
aws_security_group.ec2_allow_ingress	resource
aws_security_group.egress	resource
aws_security_group.lb_allow_egress	resource
aws_security_group.lb_allow_ingress	resource
aws_security_group.nomad_rpc	resource
aws_security_group_rule.ec2_allow_ingress_nomad_from_cidr	resource
aws_security_group_rule.ec2_allow_ingress_nomad_from_lb	resource
aws_security_group_rule.lb_allow_egress_all	resource
aws_security_group_rule.lb_allow_ingress_nomad_from_cidr	resource
aws_security_group_rule.lb_allow_ingress_nomad_from_ec2	resource
aws_ami.al2023	data source
aws_ami.provided	data source
aws_ami.rhel	data source
aws_ami.ubuntu	data source
aws_iam_policy_document.assume_role_policy	data source
aws_iam_policy_document.combined	data source
aws_iam_policy_document.gossip_encryption_key	data source
aws_iam_policy_document.license	data source
aws_iam_policy_document.nomad_discovery	data source
aws_iam_policy_document.tls_ca	data source
aws_iam_policy_document.tls_cert	data source
aws_iam_policy_document.tls_privkey	data source
aws_route53_zone.nomad	data source
aws_vpc.cluster	data source

Inputs

Name	Description	Type	Default	Required
friendly_name_prefix	Friendly name prefix used for uniquely naming AWS resources.	string	n/a	yes
instance_subnets	List of AWS subnet IDs for instance(s) to be deployed into.	list(string)	n/a	yes
key_name	SSH key name, already registered in AWS, to use for instance access	string	n/a	yes
nomad_client	Boolean to enable the Nomad client agent.	bool	n/a	yes
nomad_datacenter	Specifies the data center of the local agent. A datacenter is an abstract grouping of clients within a region. Clients are not required to be in the same datacenter as the servers they are joined with, but do need to be in the same region.	string	n/a	yes
nomad_server	Boolean to enable the Nomad server agent.	bool	n/a	yes
region	AWS region where Nomad will be deployed.	string	n/a	yes
vpc_id	ID of the AWS VPC resources are deployed into.	string	n/a	yes
additional_package_names	List of additional repository package names to install	set(string)	[]	no
additional_security_group_ids	List of AWS security group IDs to apply to all cluster nodes.	list(string)	[]	no
asg_health_check_grace_period	The amount of time to wait for a new Nomad EC2 instance to become healthy. If this threshold is breached, the ASG will terminate the instance and launch a new one.	number	600	no
associate_public_ip	Whether public IPv4 addresses should automatically be attached to cluster nodes.	bool	false	no
autopilot_health_enabled	Whether autopilot upgrade migration validation is performed for server nodes at boot-time	bool	true	no
cidr_allow_ingress_nomad	List of CIDR ranges to allow ingress traffic on port 443 or 80 to Nomad server or load balancer.	list(string)	[ "0.0.0.0/0" ]	no
cni_version	Version of CNI plugin to install.	string	"1.6.0"	no
common_tags	Map of common tags for taggable AWS resources.	map(string)	{}	no
create_nlb	Boolean to create a Network Load Balancer for Nomad.	bool	true	no
create_route53_nomad_dns_record	Boolean to create Route53 Alias Record for nomad_hostname resolving to Load Balancer DNS name. If true, route53_hosted_zone_nomad is also required.	bool	false	no
custom_install_template	Filename of a custom Install script template to use in place of of the built-in user_data script. The file must exist within a directory named './templates' in your current working directory.	string	null	no
data_ebs_iops	Amount of IOPS to configure when EBS volume type is gp3. Must be greater than or equal to 3000 and less than or equal to 16000.	number	3000	no
data_ebs_throughput	Throughput (MB/s) to configure when data EBS volume type is gp3. Must be greater than or equal to 125 and less than or equal to 1000.	number	250	no
data_ebs_volume_size	Size (GB) of the data EBS volume for Nomad EC2 instances. Must be greater than or equal to 50 and less than or equal to 16000.	number	50	no
data_ebs_volume_type	EBS volume type for data Nomad EC2 instances vol.	string	"gp3"	no
ebs_is_encrypted	Boolean to encrypt the EBS root block device of the Nomad EC2 instance(s). An AWS managed key will be used when true unless a value is also specified for ebs_kms_key_arn.	bool	true	no
ebs_kms_key_arn	ARN of KMS customer managed key (CMK) to encrypt Nomad EC2 EBS volumes.	string	null	no
ec2_allow_ssm	Boolean to attach the AmazonSSMManagedInstanceCore policy to the Nomad instance role, allowing the SSM agent (if present) to function.	bool	false	no
ec2_ami_id	AMI to launch ASG instances from.	string	null	no
ec2_os_distro	Linux OS distribution type for Nomad EC2 instance. Choose from al2023, ubuntu, rhel, centos.	string	"ubuntu"	no
instance_type	EC2 instance type to launch.	string	"m5.large"	no
lb_is_internal	Boolean to create an internal (private) load balancer. The lb_subnet_ids must be private subnets when this is true.	bool	true	no
lb_subnet_ids	List of subnet IDs to use for the load balancer. If lb_is_internal is false, then these should be public subnets. Otherwise, these should be private subnets.	list(string)	null	no
nomad_acl_enabled	Boolean to enable ACLs for Nomad.	bool	true	no
nomad_architecture	Architecture of the Nomad binary to install.	string	"amd64"	no
nomad_fqdn	Fully qualified domain name of the Nomad Cluster. This name should resolve to the load balancer IP address and will be what admins will use to access Nomad.	string	null	no
nomad_gossip_encryption_key_secret_arn	ARN of AWS Secrets Manager secret for Nomad gossip encryption key.	string	null	no
nomad_license_secret_arn	ARN of AWS Secrets Manager secret for Nomad license file.	string	null	no
nomad_nodes	Number of Nomad nodes to deploy.	number	6	no
nomad_region	Specifies the region of the local agent. A region is an abstract grouping of datacenters. Clients are not required to be in the same region as the servers they are joined with, but do need to be in the same datacenter. If not specified, the region is set AWS region.	string	null	no
nomad_tls_ca_bundle_secret_arn	ARN of AWS Secrets Manager secret for private/custom TLS Certificate Authority (CA) bundle in PEM format. Secret must be stored as a base64-encoded string.	string	null	no
nomad_tls_cert_secret_arn	ARN of AWS Secrets Manager secret for Nomad TLS certificate in PEM format. Secret must be stored as a base64-encoded string.	string	null	no
nomad_tls_enabled	Boolean to enable TLS for Nomad.	bool	true	no
nomad_tls_privkey_secret_arn	ARN of AWS Secrets Manager secret for Nomad TLS private key in PEM format. Secret must be stored as a base64-encoded string.	string	null	no
nomad_ui_enabled	Boolean to enable the Nomad UI.	bool	true	no
nomad_upstream_servers	List of Nomad server addresses to join the Nomad client with.	list(string)	null	no
nomad_upstream_tag_key	String of the tag key the Nomad client should look for in AWS to join with. Only needed for auto-joining the Nomad client.	string	null	no
nomad_upstream_tag_value	String of the tag value the Nomad client should look for in AWS to join with. Only needed for auto-joining the Nomad client.	string	null	no
nomad_version	Version of Nomad to install.	string	"1.9.0+ent"	no
permit_all_egress	Whether broad (0.0.0.0/0) egress should be permitted on cluster nodes. If disabled, additional rules must be added to permit HTTP(S) and other necessary network access.	bool	true	no
root_ebs_iops	Amount of IOPS to configure when root EBS volume type is gp3. Must be greater than or equal to 3000 and less than or equal to 16000.	number	3000	no
root_ebs_throughput	Throughput (MB/s) to configure when root EBS volume type is gp3. Must be greater than or equal to 125 and less than or equal to 1000.	number	250	no
root_ebs_volume_size	Size (GB) of the root EBS volume for Nomad EC2 instances. Must be greater than or equal to 50 and less than or equal to 16000.	number	50	no
root_ebs_volume_type	EBS volume type for root Nomad EC2 instances vol.	string	"gp3"	no
route53_nomad_hosted_zone_is_private	Boolean indicating if route53_nomad_hosted_zone_name is a private hosted zone.	bool	false	no
route53_nomad_hosted_zone_name	Route53 Hosted Zone name to create nomad_hostname Alias record in. Required if create_nomad_alias_record is true.	string	null	no

Outputs

Name	Description
nomad_cli_config	Environment variables to configure the nomad CLI
nomad_url	URL to access Nomad application based on value of nomad_fqdn input.