• Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Hello, I have a situation where I'm doing a breakdown on my default branch, then a breakdown on my feature branch, then I'm doing a diff, and I'm getting the error
    Invalid --compare-to Infracost JSON, found duplicate project name ...
    . What exactly does this mean?
  • I'm getting that error on the diff
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    @Malcolm (Terrateam) can you share the commands you are running for breakdown and diff? I'll be able to have more of a look in about an hour.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    @Alistair (Infracost) Here is where I create the breakdown based on the base branch: https://github.com/terrateamio/action/blob/v1/terrat_runner/workflow_step_infracost_setup_base.py and here is where I do the breakdown on the feature branch and the diff: https://github.com/terrateamio/action/blob/v1/terrat_runner/workflow_step_infracost_setup.py
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    So this error appears if you have multiple projects in your Infracost JSON that have the same project name. Since Infracost is unable to distinguish between them then when it is doing the diff it can’t tell which one to diff against. So I’m wondering what the config file looks like or the Infracost output JSON that means that there’s two projects with the same name.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    @Alistair (Infracost) What does "two projects with the same name" mean exactly? Is only the
    diff
    checking this? Should
    breakdown
    fail in this case?
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    It means two projects in the
    prev_infracost
    JSON file have the same
    name
    key. The diff is checking this because it assumes unique project names otherwise it doesn’t know which one to diff against. Are you setting project names in the config file when you create it?
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
  • I am not setting project names
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    ok, do any of the
    dirspaces
    have the same path and workspace?
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    I will double check but no, they should not
  • That tuple should be unique
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    the name it is showing in the
    here
    found duplicate project name ...
    is the one it’s finding twice.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Yes, I'm trying to setup a test for this. But since I'm not specifying a project name, how is infracost assigning a project name that would be duplicated?
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    Infracost defaults the name based on the path and the workspace. So the name it’s generating should be
    <path> (Workspace: <workspace>)
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Does it not include that if the workspace is
    default
    ?
  • Does it not include the (workspace) part I mean
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    correct it just uses the path for that case
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Thank you. I'll try some things
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    You can set your own project name by specifying a
    name
    key for each project in the config file if you want as well.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    @Alistair (Infracost) How does infracost handle a configuration like:
    projects:
    - path: foo
      terraform_workspace: default
    - path: foo/bar
      terraform_workspace: default
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    Ah that might be causing the problem. If
    foo
    is a Terraform mono repo then it will create a new project for each project underneath it, and so there might be two created for
    foo/bar
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    I think that is what I'm seeing
  • Is there any way to tell it to not be recursive?
  • Could I set an exclude path for anything underneath?
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    There’s
    exclude_paths
    (https://www.infracost.io/docs/features/config_file/#usage) - does that help?
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    A non-recursive flag would be preferred but I might be able to work with exclude_paths
  • Would this work?
    projects:
    - path: foo
      terraform_workspace: default
      exclude_paths: ['foo/*/*']
    - path: foo/bar
      terraform_workspace: default
      exclude_paths: ['foo/bar/*/*']
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    It could work - it’s not something I’ve used much, so I can’t say for sure. My bigger question is why do you need to specify all the sub-paths separately?
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Because it's a monorepo we have precise control over what we consider a terraform dir and what we don't
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    Okay, so if you pass the parent directory to Infracost it doesn’t properly detect the terraform dirs?
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    We have our own tooling which detects what the terraform dirs are and which ones we want to run various operations on
  • Hrm, excluse_paths does not seem to work
  • exclude_paths
  • @Alistair (Infracost) If i were to manipulate the output JSON to remove dup projects, would
    diff
    work fine? Or does it depend on the
    summary
    information in a
    breakdown
    to be correct?
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    yeah that could work. Another option could be to set the
    name
    manually in the config file.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Well a question:
    breakdown
    is producing duplicate entries in the output, but does that mean the summary contains the duplicate values?
  • Or rather does the summary have the a duplicate sum?
  • @Alistair (Infracost) I created an Issue as well: https://github.com/infracost/infracost/issues/1952
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    Sorry for the slow response, thanks for creating an issue. The summaries will probably still contain duplicates if you modify the JSON directly yeah. Just trying to understand a little more. Are there
    .tf
    files directly in both
    /foo
    and
    /foo/bar
    ? And are there directories under
    /foo
    you aren’t wanting to generate Infracost results for?
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Yes, and possibly yes.
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    Maybe try:
    exclude_paths:
          - ./*
    This seems to work for me. The exclude_paths is relative to the path parameter.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    Ah ok I'll try
  • @Alistair (Infracost) Sorry for the long delay (and I know it's weekend, so no rush getting back to me), but
    exclude_paths
    did not work either.
    projects:
    - exclude_paths:
      - ./*
      path: non-prod
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1/qa
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1/qa/mysql
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1/qa/webserver-cluster
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1/stage
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1/stage/mysql
      terraform_workspace: default
    - exclude_paths:
      - ./*
      path: non-prod/us-east-1/stage/webserver-cluster
      terraform_workspace: default
    version: '0.1'
  • Alistair (Infracost)

    Alistair (Infracost)

    1 week ago
    Okay, this is odd, here’s some ideas to debug it further. 1. It would be good to see the output of
    tree
    on this directory. 2. What is the output of running
    infracost breakdown --config-file=infracost.yml --format=json | jq -r '.projects[].name'
    3. What is the output of
    INFRACOST_PARALLELISM=1 infracost breakdown --config-file=infracost.yml --format=json --log-level=debug 2>&1 | grep 'walking path\|skipping directory'
    ? 4. If you run with just the first project, i.e.:
    projects:
      - exclude_paths:
        - ./*
        path: non-prod
        terraform_workspace: default
    What is the output of the above commands from 2 & 3? Random idea that might work for just now. Is there any way of flattening the directories yourself before running Infracost? i.e. having
    non-prod
    ,
    non-prod-us-east-1
    ,
    non-prod-us-east-1-qa
    . I know this isn’t ideal, just an idea for a workaround.
  • Malcolm (Terrateam)

    Malcolm (Terrateam)

    1 week ago
    @Alistair (Infracost) I edited the JSON file and removed all duplicate path/workspaces, and that seems to be working as expected. It looks like diff recalcualtes the summary values on its own