Experiment: CAS

[yhakbar]

Share your feedback on the cas experiment.

If you believe that we're ready to end the experiment, and support the feature by default, share that opinion!

[FlorinAndrei]

How does CAS interact with parallel runs? I assume the cache is shared somehow, but what if two parallel runs want to create the same entry?

How are repo tag versions handled? E.g. I might use this block for some resource:

terraform {
  source = "${include.compcommon.locals.base_source_url}?ref=v5.21.0"
}

But I might use this block for some other resource:

terraform {
  source = "${include.compcommon.locals.base_source_url}?ref=v5.20.0"
}

[yhakbar]

How does CAS interact with parallel runs? I assume the cache is shared somehow, but what if two parallel runs want to create the same entry?

It's designed to be safe to run concurrently. The CAS store gets populated with every unique file cloned from Git for all runs being performed, and each unit gets an independent directory created with files that are hard links to the shared CAS store.

How are repo tag versions handled?

If you only change one file between version v5.20.0 and v5.21.0, then you'll have a copy of every file in v5.20.0 and an extra copy of the file that changed between them in your store.

[wjam]

How are SSH credentials picked up by the CAS experiment? I ask because attempting to run with --experiment cas added to the plan command line ends up with this comment for each unit:

CAS download failed: git_ls_remote: no matching references: no matching reference. Falling back to standard getter.

[yhakbar]

Hmmm... It should just be using the native git binary installed on your system to perform git ls-remote there. Is there any way you could provide a way to reproduce the error you're experiencing in a test repo referenced in an issue?

[wjam]

It's a private repository, so it'll take a bit to get a test repository.

But looking at the Terragrunt code, I think the issue is that the ref part of the module URL is a commit SHA rather than a branch or a tag. This'd cause issues if the SHA isn't at the HEAD of a branch. I'll need to be on my work laptop to confirm it, but I'm guessing this command doesn't include the SHA it's expecting to download.

terragrunt/internal/cas/git.go

Line 75 in 0b18393

cmd := g.prepareCommand(ctx, "ls-remote", args...)

[wjam]

This does appear to be the issue (again without having got round to creating a minimal test case).

If ref in the source URL was main, then the ls-remote command would be this, which succeeds:

$ git ls-remote git@github.com:Ortus-Telematics-Ltd/terraform-database-module.git main
ed89dc01d8c34c41e0abe72aacd1c6a7f592ba78        refs/heads/main

If the ref in the source URL was a specific SHA, then the ls-remote command will return this (note that Git doesn't error out, just returns nothing):

$ git ls-remote git@github.com:gruntwork-io/terragrunt.git 4d7bb3816c0dae06e7eee6541cecbd2a49d3c7c3

[atheiman]

Is this supported on Windows? Running w/ git-bash and the below terragrunt/terraform versions:

terragrunt version v0.92.1
Terraform v1.13.3
on windows_amd64

Running terragrunt run --experiment cas --all -- version to generate units and stacks and run terraform version in units below my current directory. This isnt speeding anything up compared to without --experiment cas. There are 3 units below $PWD, including stack { } and unit { } declarations all pulling from different paths within the same source git repo. I have unset any TG_ and TF_ env vars I typically use for this testing. Im also not seeing ~/.cache/terragrunt/cas generate, and dont see any mention of cas in the logs w/ --log-level debug.

Maybe there is more setup needed beyond just adding --experiment cas to my run --all command?

[yhakbar]

I think this is likely due to permissions issues on Windows. If you don't mind, please open a bug report on this if you're still experiencing the issue. We'll need exact steps for reproduction to investigate further.

[j2udev]

Is it expected that CAS doesn't work when using https git sources instead of ssh? At my workplace, I can't easily use ssh git cloning (of private repos).

For example, when we use this as the source for our vpc unit, the ~/.cache/terragrunt/cas folder is not created:

source = "git::https://github.com/terraform-aws-modules/terraform-aws-vpc?ref=v5.9.0"

But when we use this the cas folder is created:

source = "git@github.com:terraform-aws-modules/terraform-aws-vpc.git?ref=v5.9.0"

[yhakbar]

Did you end up reporting this as a bug, @j2udev ? I vaguely remember you bringing this up in Terragrunt Office Hours and that we might have addressed it. Is that right?

[nbjohnson]

Is this feature expected to work with nested modules? We have a source module that then sources several other modules. I am seeing the ~/.cache/terragrunt/cas/store/ folder populated, when initing a second project I see a log for it attempting to use CAS for the top level source module, but then it seems to download all other nested modules seemingly without using the cache. The docs make mention of CAS linking CAS entries to the new repository when using the cache, but I have not yet seen any links in my caches.

[yhakbar]

No, modules referenced in OpenTofu/Terraform code is going to be downloaded by OpenTofu/Terraform, and Terragrunt won't interfere with those downloads.

[nbjohnson]

Thank you for clarifying that, at least that helps me understand why the caching doesn't seem to be helping me. Would there be any possibility of terragrunt being able to do that? We re-use submodules and having to download them multiple times even though they are the same version is a significant slowdown and having someway to be able to cache those would be a huge boost.

[yhakbar]

I would recommend checking to see if there are any existing OpenTofu issues that explore that kind of feature request, and if none exist to create one (and to thumbs up any issues that do exist if they do already). It can help to give concrete numbers on the pain this is causing you. The OpenTofu team is really responsive, and they really do care if something has significant impact to users.

In the interim, one thing you can look to do is refactor your infrastructure so that more of your cloning happens from Terragrunt. There's a guide here for how you can restructure your infrastructure like that. Hope this helps!

[yhakbar]

The cas experiment is now feature-complete 🎉

You can now use the CAS wherever you clone source code with Terragrunt, and it will be on by default in Terragrunt 1.1. In addition to de-duplicating clones, it will also serve as a mechanism for improved ergonomics in catalog design. You will be able to use the update_source_with_cas attribute to have paths rewritten on disk after cloning to synthetic trees in your CAS. This allows for more conveniently authored Terragrunt configurations in catalog repositories, as you will no longer need to use self-referential remote references in catalog definitions of module/unit/stack sources.

You will be able to use the completed experiment in the next patch release of Terragrunt (v1.0.7), or use it today in the latest tip build. We'll be preparing v1.1-rc1 soon, and with the start of that release, you will no longer need to use the cas experiment to leverage the CAS. You can always opt out of using the CAS with the --no-cas option.

You are heavily encouraged to share your feedback early and often as we go through the stabilization process. We will continue to update the behavior of the CAS throughout v1 releases with new features and optimizations, but we'll look to keep the default behavior stable for the duration of v1.

[TColl]

Possibly a bit of an edge case, but we're seeing issues with the CAS experiment where our stacks have dynamically defined unit source paths, e.g. in a catalog stack file:

unit "bar" {
  source                 = "../../..//units/foo/bar/${try(values.parent_config_path, "") != "" ? "nested" : "top-level"}"
  update_source_with_cas = true
  path                   = "bar"
  ...
}

Where we have two possible catalog units to choose from depending on the values provided in the calling stack ( units/foo/bar/nested/terragrunt.hcl and units/foo/bar/top-level/terragrunt.hcl). This is happening due to having to bake dependencies in at unit level, but if there's a cleaner pattern for having a single unit which can handle an optional input like that I'd be very keen to see it! We can also work around this by duplicating our stacks but that feels like we're introducing a lot of scope for human copy/paste error in keeping the two files aligned in future if we do that.

Right now, when we try to generate our stacks with CAS enabled, .terragrunt-stack/bar/terragrunt.stack.hcl generates a copy of the catalog stack file without CAS references being replaced, then its internal units fail to generate at all:

$ terragrunt --experiment cas stack run plan -- -concise
WARN   CAS processing failed for stack "baz": failed to process directory for CAS: failed to build synthetic tree for unit "bar": lstat /var/folders/5b/_nhq3tl95td98mjyq94d3qh40000gp/T/terragrunt-cas-stack-*584066472/repo/units/foo/bar/nestedtop-level: no such file or directory. Falling back to standard getter.

The calling stack in the live repo does deploy a separate unit alongside this stack (without any nasty dynamic source refs), and that one is getting generated properly with a CAS reference in place so I can see the overall setup working, just not on this specific catalog stack.

[yhakbar]

Hey @TColl ,

I'd recommend leveraging the experimental autoinclude block from the stack-dependencies experiment.

You can inject your inputs from the unit block on the terragrunt.stack.hcl file that generates it. We need the path to be static, unfortunately, as we need to have a synthetic CAS tree that we can reliably point you at after generation.

[TColl]

Thanks @yhakbar, that makes sense. I've been keeping an eye on the autoinclude experiment precisely for that reason! I'm not keen on using either feature in production until the experiments conclude though, so I guess we'll have to hold off from upgrading to 1.1 onwards until the autoinclude piece is available too for replacing our setup cleanly in one go.

I know it's an edge case, but I think this technically makes the CAS a breaking change - granted we've been using terragrunt in unconventional/undocumented ways so no major concerns from me there, it's just a mildly annoying setback given we made the upgrade to v1 primarily for the compatibility guarantees and the hope of an easy life 😆

[TColl]

On second thoughts, if we just leave the relative paths / update_source_with_cas flags off that stack we should be safe to move forward with the CAS for everything else, right?

[yhakbar]

Yup! If you don't use update_source_with_cas, there's no requirement that you have statically defined relative source paths.