Title
#general
s

Sylvain Witmeyer

09/12/2022, 6:56 AM
Hi folks. I’m struggling on an error I can’t fix
Error: Failed to parse the Terragrunt code using the Terragrunt library:
1 error occurred:
	* error downloading '<https://github.com/aws-ia/terraform-aws-control_tower_account_factory.git?ref=1.6.2>': /usr/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git
I don’t know why it’s looking for the master branch I’m using terragrunt and calling the module like this
terraform {
  source = "<http://github.com/aws-ia/terraform-aws-control_tower_account_factory//.?ref=1.6.2|github.com/aws-ia/terraform-aws-control_tower_account_factory//.?ref=1.6.2>"
}
Any idea what’s wrong ? Thank you
7:30 AM
Follow up
# Remove the '!${{ env.TF_ROOT }}...' path below if you're using Terragrunt and seeing
          # issues downloading from remote git sources.
          path: |
            ${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**
            !${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**/.git/**
the git folders were commented and I fixed the issue with re-enabling them. So doing the opposite of the comment fix my issue for the first time and when I rerun my action, it failed again. I think the error comes from this
# If there's no cached record for this commit, pull in the latest cached record anyway
          # Internally infracost will downloaded any additional required modules if required
          restore-keys: infracost-terraform-modules-${{ runner.os }}-
I updated my module from 1.6.1 to 1.6.2, so the cache should contains my previous version
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:12 AM
hmm, interesting. What was the error the second time?
s

Sylvain Witmeyer

09/12/2022, 9:13 AM
Exactly the same
Error: Failed to parse the Terragrunt code using the Terragrunt library:
1 error occurred:
	* error downloading '<https://github.com/aws-ia/terraform-aws-control_tower_account_factory.git?ref=1.6.2>': /usr/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:21 AM
I’m wondering if the cache now contains some bad data. You could try re-running it with the line commented out a different cache key prefix to see if that helps.
s

Sylvain Witmeyer

09/12/2022, 9:35 AM
even if the cache has a bad tag, I still don’t understand why it’s looking for master.
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:41 AM
I’m not exactly sure tbh, it’s something in the Terragrunt source code that is doing it (see https://github.com/gruntwork-io/terragrunt/issues/764).
s

Sylvain Witmeyer

09/12/2022, 9:44 AM
I can try to remove the .terragrunt-cache from the cache. What do you think about
!${{ env.TF_ROOT }}/**/.terragrunt-cache/**
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:46 AM
yeah that could work!
s

Sylvain Witmeyer

09/12/2022, 9:47 AM
But I’m not sure what will be left in the cache if we remove git and terragrunt cache 🙂
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:49 AM
yeah fair, the point of the cache is to make it faster when using remote modules so this makes it less useful
s

Sylvain Witmeyer

09/12/2022, 9:49 AM
Maybe we should just use the branch name instead of the sha for the cache key, and not restoring the more global one if we don’t have one
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:50 AM
yeah, it will mean new PRs still need to redownload all modules, but will at least speed up future runs on the same PR
s

Sylvain Witmeyer

09/12/2022, 9:52 AM
yes because this is very specific
key: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.base.sha || github.sha }}
Actually we never can reuse this key so we always end up with
infracost-terraform-modules-${{ runner.os }}-
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 9:57 AM
The pipeline will re-use that the cache key when it re-runs the PR since it’s taking from the base commit. But maybe adding the head ref like something like
key: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.head.ref }}-${{ github.event.pull_request.base.sha || github.sha }}
and then setting the restore key to
restore-keys: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.head.ref }}-
could help. What do you think?
s

Sylvain Witmeyer

09/12/2022, 10:01 AM
I think we shouldn’t use the sha in the key. Usually we would use the signature of the dependencies file sha256(packages.json) for instance, hence if the file is the same the cache is restored but with the commit sha, this key is unique so it can’t be reused. I would try something more like
key: infracost-terraform-modules-${{ runner.os }}-${{ github.ref }}
10:10 AM
1st what do you think of
- name: Cache the Infracost baseline JSON result
        id: cache-infracost-base-json
        uses: actions/cache@v3
        with:
          path: '/tmp/infracost-base.json'
          key: infracost-base-json-${{ runner.os }}-${{ github.base_ref || github.ref }}
if it’s a pull request we create the key with the target branch “main”, if it’s a push we use the current branch. That way every PR targetting “main” will be able to reuse the cache
10:17 AM
restore-keys
- An ordered list of keys to use for restoring stale cache if no cache hit occurred for key. Note
cache-hit
returns false in this case. by using the commit sha we can never have a cache hit
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 10:20 AM
yeah but using the base ref would mean that it hits each time for every subsequent run of a PR unless it’s rebased right?
s

Sylvain Witmeyer

09/12/2022, 10:22 AM
Cache not found for input keys: infracost-base-json-Linux-main
So yes we want to hit the cache build from the main branch for every subsequent run. This cache will alway be up to date with the last build, which is fine I guess. I can add commits on my PR, I will reuse the same cache, or I can create another PR it will also use this cache for the base json
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 10:28 AM
yeah, I’m wondering if there’s a nice way we could use a manifest file or something. For terraform repos we could use the lock file, but not sure if there’s an equivalent for Terragrunt.
s

Sylvain Witmeyer

09/12/2022, 10:33 AM
will be harder with tg. But I think I have something nicely working now. Using the branch and remove the restore key because if we restore a key with a change it seems to break tg.
- name: Cache the Infracost baseline JSON result
        id: cache-infracost-base-json
        uses: actions/cache@v3
        with:
          path: '/tmp/infracost-base.json'
          key: infracost-base-json-${{ runner.os }}-${{ github.base_ref || github.ref }}

      # Checkout the base branch of the pull request (e.g. main/master).
      - name: Checkout base branch
        uses: actions/checkout@v3
        with:
          ref: '${{ github.event.pull_request.base.ref }}'

      # Downloading remote Terraform modules can be slow, so we add them to the GitHub cache.
      # We skip this for pushes to the main/master branch that already have the baseline generated.
      - name: Cache .infracost/terraform_modules for target branch
        uses: actions/cache@v3
        with:
          # Remove the '!${{ env.TF_ROOT }}...' path below if you're using Terragrunt and seeing
          # issues downloading from remote git sources.
          path: |
            ${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**
            !${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**/.git/**
          key: infracost-terraform-modules-${{ runner.os }}-${{ github.head_ref || github.ref }}
          # If there's no cached record for this commit, pull in the latest cached record anyway
          # Internally infracost will downloaded any additional required modules if required
#          restore-keys: infracost-terraform-modules-${{ runner.os }}-
        if: github.event_name == 'pull_request' || steps.cache-infracost-base-json.outputs.cache-hit != 'true'
Alistair (Infracost)

Alistair (Infracost)

09/12/2022, 10:52 AM
ok nice, thanks! I’ll give this a shot as well with some of our test terragrunt projects.