-
Notifications
You must be signed in to change notification settings - Fork 17
dev-docs: document setup on eks #2239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| # Contrast on EKS | ||
|
|
||
| This is based on the setup presented in https://github.com/aws-samples/howto-runtime-attestation-on-aws. | ||
|
|
||
| ## Resources | ||
|
|
||
| Replace the placeholder in eks-managed-sev-snp-metal-ubuntu-template.yaml with your SSH key name. | ||
|
|
||
| Create cluster and node group with | ||
|
|
||
| ``` | ||
| eksctl create cluster --without-nodegroup -f eks-cluster-template.yaml | ||
| eksctl create nodegroup -f eks-managed-sev-snp-metal-ubuntu-template.yaml | ||
| ``` | ||
|
|
||
| ## Prevent autoscaling of the node | ||
|
|
||
| This is needed as the node needs to be rebooted. | ||
|
|
||
| ```sh | ||
| aws autoscaling suspend-processes \ | ||
| --auto-scaling-group-name eksctl-raas-nodegroup-selfmanaged-NodeGroup-c2opXigZr6N9 \ | ||
| --scaling-processes ReplaceUnhealthy | ||
| ``` | ||
|
|
||
| ## Setup the node | ||
|
|
||
| Get ssh access, then | ||
|
|
||
| ```sh | ||
| sudo apt update | ||
| sudo apt upgrade -y | ||
| ``` | ||
|
|
||
| ```bash | ||
| sudo apt install dracut -y | ||
| sudo tee -a /etc/dracut.conf.d/20-omit-ccp.conf <<EOF | ||
| omit_drivers+=" ccp " | ||
| EOF | ||
| sudo dracut --force | ||
| sudo tee -a /etc/modprobe.d/60-ccp.conf <<EOF | ||
| options ccp init_ex_path=/SEV_metadata | ||
| EOF | ||
| ``` | ||
|
|
||
| ```bash | ||
| sudo sed 's/\(GRUB_CMD.*\)"/\1 mem_encrypt=on kvm_amd.sev=1 iommu=nopt"/' -i /etc/default/grub.d/50-cloudimg-settings.cfg | ||
| grep GRUB_CMD /etc/default/grub.d/50-cloudimg-settings.cfg # To validate | ||
| sudo update-grub | ||
| sudo grep sev /boot/grub/grub.cfg # Check | ||
| ``` | ||
|
|
||
| Reboot the node | ||
|
|
||
| ```sh | ||
| sudo reboot | ||
| ``` | ||
|
|
||
| ## Get the reference values | ||
|
|
||
| Access the node again via ssh, then | ||
|
|
||
| ```sh | ||
| $ sh <(curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install) --daemon | ||
| ``` | ||
|
|
||
| As root | ||
|
|
||
| ```sh | ||
| nix-shell -p snphost | ||
| ``` | ||
|
|
||
| Reference values and check of host configuration via | ||
|
|
||
| ```sh | ||
| modprobe msr | ||
| snphost ok | ||
| ``` | ||
|
|
||
| ```sh | ||
| lscpu | grep "Model name\|CPU family" | ||
| ``` | ||
|
|
||
| CPU family = 25 → Milan | ||
| CPU family = 26 → Genoa | ||
|
|
||
| ## Get the kubeconfig of the cluster | ||
|
|
||
| ```sh | ||
| aws eks update-kubeconfig \ | ||
| --region eu-central-1 \ | ||
| --name raas \ | ||
| --kubeconfig kubeconf | ||
| export KUBECONFIG=$(realpath kubeconf) | ||
| ``` | ||
|
|
||
| ## Make `ebs` default storage class | ||
|
|
||
| ```sh | ||
| kubectl patch storageclass gp2 \ | ||
| -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' | ||
| ``` | ||
|
|
||
| ## Configure the Contrast Manifest | ||
|
|
||
| - Remove the non-matching SNP processor family | ||
| - Update the TCB values with the output of `snphost ok` | ||
| - Set `TSMEEnabled` to `true` | ||
|
|
||
| ## Update settings.json | ||
|
|
||
| Set the correct pause image | ||
|
|
||
| ```json | ||
| "cluster_config": { | ||
| "pause_container_image": "public.ecr.aws/eks-distro/kubernetes/pause:3.5", | ||
| ``` | ||
|
|
||
| ## Getting the coordinator service IP | ||
|
|
||
| Notice EKS isn't giving a LoadBalancer IP by default but a DNS name. | ||
| The instructions from the docs won't work to get it, describe the service and use the ingress field. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| apiVersion: eksctl.io/v1alpha5 | ||
| kind: ClusterConfig | ||
|
|
||
| metadata: | ||
| version: "1.31" | ||
| region: eu-central-1 | ||
| name: contrast-testing | ||
|
|
||
| iam: | ||
| withOIDC: true | ||
| serviceAccounts: | ||
| - metadata: | ||
| name: aws-load-balancer-controller | ||
| namespace: kube-system | ||
| wellKnownPolicies: | ||
| awsLoadBalancerController: true | ||
|
|
||
| addons: | ||
| - name: aws-ebs-csi-driver | ||
| wellKnownPolicies: # Adds an IAM service account | ||
| ebsCSIController: true |
33 changes: 33 additions & 0 deletions
33
dev-docs/frozen/eks-managed-sev-snp-metal-ubuntu-template.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| apiVersion: eksctl.io/v1alpha5 | ||
| kind: ClusterConfig | ||
|
|
||
| metadata: | ||
| name: contrast-testing | ||
| region: eu-central-1 | ||
| version: "1.31" | ||
|
|
||
| iam: | ||
| withOIDC: true | ||
|
|
||
| nodeGroups: | ||
| - name: selfmanaged | ||
| instanceType: "m6a.metal" | ||
| overrideBootstrapCommand: | | ||
| #!/bin/bash | ||
| source /var/lib/cloud/scripts/eksctl/bootstrap.helper.sh | ||
| # Note "--node-labels=${NODE_LABELS}" needs the above helper sourced to work, otherwise will have to be defined manually. | ||
| /etc/eks/bootstrap.sh ${CLUSTER_NAME} --container-runtime containerd --kubelet-extra-args "--node-labels=${NODE_LABELS}" | ||
| desiredCapacity: 1 | ||
| maxSize: 1 | ||
| amiFamily: Ubuntu2404 | ||
| volumeType: gp3 | ||
| volumeSize: 300 | ||
| iam: | ||
| attachPolicyARNs: | ||
| - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy | ||
| - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly | ||
| - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore | ||
| - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy | ||
| ssh: | ||
| allow: true | ||
| publicKeyName: ssh-key-name # replace with your actual key name |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you try the
kexecapproach described here: https://github.com/aws-samples/howto-runtime-attestation-on-aws/blob/a3e1361a7fb94dbad7af69e9ad4669f39835883e/README.md?plain=1#L155-L161? That might make the reboot a lot faster.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I tried but cannot remember the complication with it.