Title
#general
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 7:52 AM
Hello, I have a situation where I'm doing a breakdown on my default branch, then a breakdown on my feature branch, then I'm doing a diff, and I'm getting the error
Invalid --compare-to Infracost JSON, found duplicate project name ...
. What exactly does this mean?
7:53 AM
I'm getting that error on the diff
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 8:02 AM
@Malcolm (Terrateam) can you share the commands you are running for breakdown and diff? I'll be able to have more of a look in about an hour.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 8:25 AM
@Alistair (Infracost) Here is where I create the breakdown based on the base branch: https://github.com/terrateamio/action/blob/v1/terrat_runner/workflow_step_infracost_setup_base.py and here is where I do the breakdown on the feature branch and the diff: https://github.com/terrateamio/action/blob/v1/terrat_runner/workflow_step_infracost_setup.py
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 8:56 AM
So this error appears if you have multiple projects in your Infracost JSON that have the same project name. Since Infracost is unable to distinguish between them then when it is doing the diff it can’t tell which one to diff against. So I’m wondering what the config file looks like or the Infracost output JSON that means that there’s two projects with the same name.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 11:17 AM
@Alistair (Infracost) What does "two projects with the same name" mean exactly? Is only the
diff
checking this? Should
breakdown
fail in this case?
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 11:20 AM
It means two projects in the
prev_infracost
JSON file have the same
name
key. The diff is checking this because it assumes unique project names otherwise it doesn’t know which one to diff against. Are you setting project names in the config file when you create it?
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 11:20 AM
11:21 AM
I am not setting project names
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 11:21 AM
ok, do any of the
dirspaces
have the same path and workspace?
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 11:21 AM
I will double check but no, they should not
11:21 AM
That tuple should be unique
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 11:23 AM
the name it is showing in the
here
found duplicate project name ...
is the one it’s finding twice.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 11:24 AM
Yes, I'm trying to setup a test for this. But since I'm not specifying a project name, how is infracost assigning a project name that would be duplicated?
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 11:25 AM
Infracost defaults the name based on the path and the workspace. So the name it’s generating should be
<path> (Workspace: <workspace>)
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 11:26 AM
Does it not include that if the workspace is
default
?
11:26 AM
Does it not include the (workspace) part I mean
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 11:26 AM
correct it just uses the path for that case
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 11:26 AM
Thank you. I'll try some things
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 11:27 AM
You can set your own project name by specifying a
name
key for each project in the config file if you want as well.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 12:34 PM
@Alistair (Infracost) How does infracost handle a configuration like:
projects:
- path: foo
  terraform_workspace: default
- path: foo/bar
  terraform_workspace: default
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 12:36 PM
Ah that might be causing the problem. If
foo
is a Terraform mono repo then it will create a new project for each project underneath it, and so there might be two created for
foo/bar
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 12:37 PM
I think that is what I'm seeing
12:38 PM
Is there any way to tell it to not be recursive?
12:42 PM
Could I set an exclude path for anything underneath?
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 12:43 PM
There’s
exclude_paths
(https://www.infracost.io/docs/features/config_file/#usage) - does that help?
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 12:44 PM
A non-recursive flag would be preferred but I might be able to work with exclude_paths
12:45 PM
Would this work?
projects:
- path: foo
  terraform_workspace: default
  exclude_paths: ['foo/*/*']
- path: foo/bar
  terraform_workspace: default
  exclude_paths: ['foo/bar/*/*']
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 12:46 PM
It could work - it’s not something I’ve used much, so I can’t say for sure. My bigger question is why do you need to specify all the sub-paths separately?
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 12:47 PM
Because it's a monorepo we have precise control over what we consider a terraform dir and what we don't
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 12:48 PM
Okay, so if you pass the parent directory to Infracost it doesn’t properly detect the terraform dirs?
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 12:48 PM
We have our own tooling which detects what the terraform dirs are and which ones we want to run various operations on
1:40 PM
Hrm, excluse_paths does not seem to work
1:43 PM
exclude_paths
1:43 PM
@Alistair (Infracost) If i were to manipulate the output JSON to remove dup projects, would
diff
work fine? Or does it depend on the
summary
information in a
breakdown
to be correct?
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 1:48 PM
yeah that could work. Another option could be to set the
name
manually in the config file.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 1:49 PM
Well a question:
breakdown
is producing duplicate entries in the output, but does that mean the summary contains the duplicate values?
1:50 PM
Or rather does the summary have the a duplicate sum?
2:49 PM
@Alistair (Infracost) I created an Issue as well: https://github.com/infracost/infracost/issues/1952
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 2:58 PM
Sorry for the slow response, thanks for creating an issue. The summaries will probably still contain duplicates if you modify the JSON directly yeah. Just trying to understand a little more. Are there
.tf
files directly in both
/foo
and
/foo/bar
? And are there directories under
/foo
you aren’t wanting to generate Infracost results for?
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 2:59 PM
Yes, and possibly yes.
Alistair (Infracost)

Alistair (Infracost)

08/05/2022, 3:04 PM
Maybe try:
exclude_paths:
      - ./*
This seems to work for me. The exclude_paths is relative to the path parameter.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/05/2022, 3:07 PM
Ah ok I'll try
1:13 PM
@Alistair (Infracost) Sorry for the long delay (and I know it's weekend, so no rush getting back to me), but
exclude_paths
did not work either.
projects:
- exclude_paths:
  - ./*
  path: non-prod
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1/qa
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1/qa/mysql
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1/qa/webserver-cluster
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1/stage
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1/stage/mysql
  terraform_workspace: default
- exclude_paths:
  - ./*
  path: non-prod/us-east-1/stage/webserver-cluster
  terraform_workspace: default
version: '0.1'
Alistair (Infracost)

Alistair (Infracost)

08/06/2022, 9:09 PM
Okay, this is odd, here’s some ideas to debug it further. 1. It would be good to see the output of
tree
on this directory. 2. What is the output of running
infracost breakdown --config-file=infracost.yml --format=json | jq -r '.projects[].name'
3. What is the output of
INFRACOST_PARALLELISM=1 infracost breakdown --config-file=infracost.yml --format=json --log-level=debug 2>&1 | grep 'walking path\|skipping directory'
? 4. If you run with just the first project, i.e.:
projects:
  - exclude_paths:
    - ./*
    path: non-prod
    terraform_workspace: default
What is the output of the above commands from 2 & 3? Random idea that might work for just now. Is there any way of flattening the directories yourself before running Infracost? i.e. having
non-prod
,
non-prod-us-east-1
,
non-prod-us-east-1-qa
. I know this isn’t ideal, just an idea for a workaround.
Malcolm (Terrateam)

Malcolm (Terrateam)

08/07/2022, 5:57 AM
@Alistair (Infracost) I edited the JSON file and removed all duplicate path/workspaces, and that seems to be working as expected. It looks like diff recalcualtes the summary values on its own