https://infracost.io logo
Powered by
#general
Title
# general
p

polite-engineer-31217

09/12/2022, 6:56 AM
Hi folks. I’m struggling on an error I can’t fix
Copy code
Error: Failed to parse the Terragrunt code using the Terragrunt library:
1 error occurred:
	* error downloading '<https://github.com/aws-ia/terraform-aws-control_tower_account_factory.git?ref=1.6.2>': /usr/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git
I don’t know why it’s looking for the master branch I’m using terragrunt and calling the module like this
Copy code
terraform {
  source = "<http://github.com/aws-ia/terraform-aws-control_tower_account_factory//.?ref=1.6.2|github.com/aws-ia/terraform-aws-control_tower_account_factory//.?ref=1.6.2>"
}
Any idea what’s wrong ? Thank you
Follow up
Copy code
# Remove the '!${{ env.TF_ROOT }}...' path below if you're using Terragrunt and seeing
          # issues downloading from remote git sources.
          path: |
            ${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**
            !${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**/.git/**
the git folders were commented and I fixed the issue with re-enabling them. So doing the opposite of the comment fix my issue for the first time and when I rerun my action, it failed again. I think the error comes from this
Copy code
# If there's no cached record for this commit, pull in the latest cached record anyway
          # Internally infracost will downloaded any additional required modules if required
          restore-keys: infracost-terraform-modules-${{ runner.os }}-
I updated my module from 1.6.1 to 1.6.2, so the cache should contains my previous version
l

little-author-61621

09/12/2022, 9:12 AM
hmm, interesting. What was the error the second time?
p

polite-engineer-31217

09/12/2022, 9:13 AM
Exactly the same
Copy code
Error: Failed to parse the Terragrunt code using the Terragrunt library:
1 error occurred:
	* error downloading '<https://github.com/aws-ia/terraform-aws-control_tower_account_factory.git?ref=1.6.2>': /usr/bin/git exited with 1: error: pathspec 'master' did not match any file(s) known to git
l

little-author-61621

09/12/2022, 9:21 AM
I’m wondering if the cache now contains some bad data. You could try re-running it with the line commented out a different cache key prefix to see if that helps.
p

polite-engineer-31217

09/12/2022, 9:35 AM
even if the cache has a bad tag, I still don’t understand why it’s looking for master.
l

little-author-61621

09/12/2022, 9:41 AM
I’m not exactly sure tbh, it’s something in the Terragrunt source code that is doing it (see https://github.com/gruntwork-io/terragrunt/issues/764).
p

polite-engineer-31217

09/12/2022, 9:44 AM
I can try to remove the .terragrunt-cache from the cache. What do you think about 
!${{ env.TF_ROOT }}/**/.terragrunt-cache/**
l

little-author-61621

09/12/2022, 9:46 AM
yeah that could work!
p

polite-engineer-31217

09/12/2022, 9:47 AM
But I’m not sure what will be left in the cache if we remove git and terragrunt cache 🙂
l

little-author-61621

09/12/2022, 9:49 AM
yeah fair, the point of the cache is to make it faster when using remote modules so this makes it less useful
p

polite-engineer-31217

09/12/2022, 9:49 AM
Maybe we should just use the branch name instead of the sha for the cache key, and not restoring the more global one if we don’t have one
l

little-author-61621

09/12/2022, 9:50 AM
yeah, it will mean new PRs still need to redownload all modules, but will at least speed up future runs on the same PR
p

polite-engineer-31217

09/12/2022, 9:52 AM
yes because this is very specific 
key: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.base.sha || github.sha }}
Actually we never can reuse this key so we always end up with
Copy code
infracost-terraform-modules-${{ runner.os }}-
l

little-author-61621

09/12/2022, 9:57 AM
The pipeline will re-use that the cache key when it re-runs the PR since it’s taking from the base commit. But maybe adding the head ref like something like 
key: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.head.ref }}-${{ github.event.pull_request.base.sha || github.sha }}
and then setting the restore key to 
restore-keys: infracost-terraform-modules-${{ runner.os }}-${{ github.event.pull_request.head.ref }}-
could help. What do you think?
p

polite-engineer-31217

09/12/2022, 10:01 AM
I think we shouldn’t use the sha in the key. Usually we would use the signature of the dependencies file sha256(packages.json) for instance, hence if the file is the same the cache is restored but with the commit sha, this key is unique so it can’t be reused. I would try something more like 
key: infracost-terraform-modules-${{ runner.os }}-${{ github.ref }}
1st what do you think of
Copy code
- name: Cache the Infracost baseline JSON result
        id: cache-infracost-base-json
        uses: actions/cache@v3
        with:
          path: '/tmp/infracost-base.json'
          key: infracost-base-json-${{ runner.os }}-${{ github.base_ref || github.ref }}
if it’s a pull request we create the key with the target branch “main”, if it’s a push we use the current branch. That way every PR targetting “main” will be able to reuse the cache
• 
restore-keys
- An ordered list of keys to use for restoring stale cache if no cache hit occurred for key. Note 
cache-hit
returns false in this case. by using the commit sha we can never have a cache hit
l

little-author-61621

09/12/2022, 10:20 AM
yeah but using the base ref would mean that it hits each time for every subsequent run of a PR unless it’s rebased right?
p

polite-engineer-31217

09/12/2022, 10:22 AM
Cache not found for input keys: infracost-base-json-Linux-main
So yes we want to hit the cache build from the main branch for every subsequent run. This cache will alway be up to date with the last build, which is fine I guess. I can add commits on my PR, I will reuse the same cache, or I can create another PR it will also use this cache for the base json
l

little-author-61621

09/12/2022, 10:28 AM
yeah, I’m wondering if there’s a nice way we could use a manifest file or something. For terraform repos we could use the lock file, but not sure if there’s an equivalent for Terragrunt.
p

polite-engineer-31217

09/12/2022, 10:33 AM
will be harder with tg. But I think I have something nicely working now. Using the branch and remove the restore key because if we restore a key with a change it seems to break tg.
Copy code
- name: Cache the Infracost baseline JSON result
        id: cache-infracost-base-json
        uses: actions/cache@v3
        with:
          path: '/tmp/infracost-base.json'
          key: infracost-base-json-${{ runner.os }}-${{ github.base_ref || github.ref }}

      # Checkout the base branch of the pull request (e.g. main/master).
      - name: Checkout base branch
        uses: actions/checkout@v3
        with:
          ref: '${{ github.event.pull_request.base.ref }}'

      # Downloading remote Terraform modules can be slow, so we add them to the GitHub cache.
      # We skip this for pushes to the main/master branch that already have the baseline generated.
      - name: Cache .infracost/terraform_modules for target branch
        uses: actions/cache@v3
        with:
          # Remove the '!${{ env.TF_ROOT }}...' path below if you're using Terragrunt and seeing
          # issues downloading from remote git sources.
          path: |
            ${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**
            !${{ env.TF_ROOT }}/**/.infracost/terraform_modules/**/.git/**
          key: infracost-terraform-modules-${{ runner.os }}-${{ github.head_ref || github.ref }}
          # If there's no cached record for this commit, pull in the latest cached record anyway
          # Internally infracost will downloaded any additional required modules if required
#          restore-keys: infracost-terraform-modules-${{ runner.os }}-
        if: github.event_name == 'pull_request' || steps.cache-infracost-base-json.outputs.cache-hit != 'true'
l

little-author-61621

09/12/2022, 10:52 AM
ok nice, thanks! I’ll give this a shot as well with some of our test terragrunt projects.
11 Views
Powered by