https://infracost.io logo
Join Slack
Powered by
Hi folks. I’m struggling on an error I can’t fix `...
# general
p
Hi folks. I’m struggling on an error I can’t fix
Copy code
Error: Failed to parse the Terragrunt code using the Terragrunt library:
1 error occurred:
	* error downloading '<https://github.com/aws-ia/terraform-aws-control_tower_account_factory.git?ref=1.6.2>': /usr/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git
I don’t know why it’s looking for the master branch I’m using terragrunt and calling the module like this
Copy code
terraform {
  source = "<http://github.com/aws-ia/terraform-aws-control_tower_account_factory//.?ref=1.6.2|github.com/aws-ia/terraform-aws-control_tower_account_factory//.?ref=1.6.2>"
}
Any idea what’s wrong ? Thank you
Follow up
Copy code
# Remove the '!${{ env.TF_ROOT }}...' path below if you're using Terragrunt and seeing
          # issues downloading from remote git sources.
          path: |
            ${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**
            !${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**/.git/**
the git folders were commented and I fixed the issue with re-enabling them. So doing the opposite of the comment fix my issue for the first time and when I rerun my action, it failed again. I think the error comes from this
Copy code
# If there's no cached record for this commit, pull in the latest cached record anyway
          # Internally infracost will downloaded any additional required modules if required
          restore-keys: infracost-terraform-modules-${{ runner.os }}-
I updated my module from 1.6.1 to 1.6.2, so the cache should contains my previous version
l
hmm, interesting. What was the error the second time?
p
Exactly the same
Copy code
Error: Failed to parse the Terragrunt code using the Terragrunt library:
1 error occurred:
	* error downloading '<https://github.com/aws-ia/terraform-aws-control_tower_account_factory.git?ref=1.6.2>': /usr/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git
l
I’m wondering if the cache now contains some bad data. You could try re-running it with the line commented out a different cache key prefix to see if that helps.
p
even if the cache has a bad tag, I still don’t understand why it’s looking for master.
l
I’m not exactly sure tbh, it’s something in the Terragrunt source code that is doing it (see https://github.com/gruntwork-io/terragrunt/issues/764).
p
I can try to remove the .terragrunt-cache from the cache. What do you think about 
!${{ env.TF_ROOT }}/**/.terragrunt-cache/**
l
yeah that could work!
p
But I’m not sure what will be left in the cache if we remove git and terragrunt cache 🙂
l
yeah fair, the point of the cache is to make it faster when using remote modules so this makes it less useful
p
Maybe we should just use the branch name instead of the sha for the cache key, and not restoring the more global one if we don’t have one
l
yeah, it will mean new PRs still need to redownload all modules, but will at least speed up future runs on the same PR
p
yes because this is very specific 
key: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.base.sha || github.sha }}
Actually we never can reuse this key so we always end up with
Copy code
infracost-terraform-modules-${{ runner.os }}-
l
The pipeline will re-use that the cache key when it re-runs the PR since it’s taking from the base commit. But maybe adding the head ref like something like 
key: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.head.ref }}-${{ github.event.pull_request.base.sha || github.sha }}
and then setting the restore key to 
restore-keys: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.head.ref }}-
could help. What do you think?
p
I think we shouldn’t use the sha in the key. Usually we would use the signature of the dependencies file sha256(packages.json) for instance, hence if the file is the same the cache is restored but with the commit sha, this key is unique so it can’t be reused. I would try something more like 
key: infracost-terraform-modules-${{ runner.os }}-${{ github.ref }}
1st what do you think of
Copy code
- name: Cache the Infracost baseline JSON result
        id: cache-infracost-base-json
        uses: actions/cache@v3
        with:
          path: '/tmp/infracost-base.json'
          key: infracost-base-json-${{ runner.os }}-${{ github.base_ref || github.ref }}
if it’s a pull request we create the key with the target branch “main”, if it’s a push we use the current branch. That way every PR targetting “main” will be able to reuse the cache
• 
restore-keys
- An ordered list of keys to use for restoring stale cache if no cache hit occurred for key. Note 
cache-hit
returns false in this case. by using the commit sha we can never have a cache hit
l
yeah but using the base ref would mean that it hits each time for every subsequent run of a PR unless it’s rebased right?
p
Cache not found for input keys: infracost-base-json-Linux-main
So yes we want to hit the cache build from the main branch for every subsequent run. This cache will alway be up to date with the last build, which is fine I guess. I can add commits on my PR, I will reuse the same cache, or I can create another PR it will also use this cache for the base json
l
yeah, I’m wondering if there’s a nice way we could use a manifest file or something. For terraform repos we could use the lock file, but not sure if there’s an equivalent for Terragrunt.
p
will be harder with tg. But I think I have something nicely working now. Using the branch and remove the restore key because if we restore a key with a change it seems to break tg.
Copy code
- name: Cache the Infracost baseline JSON result
        id: cache-infracost-base-json
        uses: actions/cache@v3
        with:
          path: '/tmp/infracost-base.json'
          key: infracost-base-json-${{ runner.os }}-${{ github.base_ref || github.ref }}

      # Checkout the base branch of the pull request (e.g. main/master).
      - name: Checkout base branch
        uses: actions/checkout@v3
        with:
          ref: '${{ github.event.pull_request.base.ref }}'

      # Downloading remote Terraform modules can be slow, so we add them to the GitHub cache.
      # We skip this for pushes to the main/master branch that already have the baseline generated.
      - name: Cache .infracost/terraform_modules for target branch
        uses: actions/cache@v3
        with:
          # Remove the '!${{ env.TF_ROOT }}...' path below if you're using Terragrunt and seeing
          # issues downloading from remote git sources.
          path: |
            ${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**
            !${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**/.git/**
          key: infracost-terraform-modules-${{ runner.os }}-${{ github.head_ref || github.ref }}
          # If there's no cached record for this commit, pull in the latest cached record anyway
          # Internally infracost will downloaded any additional required modules if required
#          restore-keys: infracost-terraform-modules-${{ runner.os }}-
        if: github.event_name == 'pull_request' || steps.cache-infracost-base-json.outputs.cache-hit != 'true'
l
ok nice, thanks! I’ll give this a shot as well with some of our test terragrunt projects.
12 Views
Open in Slack
PreviousNext
Powered by