Hey everyone. I'm struggling with getting a pipeline in Azure DevOps running, bumping into an error with duplicate project detected: Error: Invalid --compare-to Infracost JSON, found duplicate project name entity/functions/Event.Publisher/terraform I've got an ADO pipeline which tries to compare a main branch against a pull request branch that is running into the issue, but I can't reproduce it locally. Here's the highlevel steps in the pipeline:

jobs:
  # Run Infracost on pull requests
  - job: infracost_pull_request_checks
    displayName: Run Infracost on pull requests
    pool:
      vmImage: ubuntu-latest

    steps:
      - bash: |
          curl -fsSL <https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh> | sh
          infracost configure set api_key ${{ parameters.infracostApiKey }}
        displayName: Setup Infracost

      - bash: |
          git clone $(Build.Repository.Uri) --branch=$(System.PullRequest.TargetBranchName) --single-branch /tmp/base --config http.extraheader="AUTHORIZATION: bearer $(System.AccessToken)"
        displayName: Checkout Base Branch

      - bash: |
          infracost breakdown --path=. \
            --format=json \
            --out-file=/tmp/infracost-base.json
        displayName: Generate Infracost Cost Estimate Baseline
        workingDirectory: /tmp/base/${{ parameters.terraformDirectory }}
        env:
          SYSTEM_ACCESSTOKEN: $(System.AccessToken)

      - bash: |
          infracost diff --path=. \
            --format=json \
            --compare-to=/tmp/infracost-base.json \
            --out-file=/tmp/infracost.json
        displayName: Generate Infracost Diff
        workingDirectory: $(System.DefaultWorkingDirectory)/${{ parameters.terraformDirectory }}
        env:
          SYSTEM_ACCESSTOKEN: $(System.AccessToken)

My repo layout is like this: root --> terraform/main.tf --> terraform/(dev|qa|uat|prd).tfvars

When you run

infracost breakdown --path terraform

locally on your machine, does it show something like this in the output? That’ll tell us if the auto-detect is working ok or if we need to add a config file to tell the CLI what the projects are.

INFO Autodetected 4 Terraform projects across 1 root module
INFO Found Terraform project dev, qa, uat, prd at directory terraform

From root of the repo if I run that command locally, I get this output:

INFO Autodetected 4 Terraform projects across 1 root module
INFO Found Terraform project main-dev at directory . using Terraform var files dev.tfvars
INFO Found Terraform project main-prd at directory . using Terraform var files prd.tfvars
INFO Found Terraform project main-qa at directory . using Terraform var files qa.tfvars
INFO Found Terraform project main-uat at directory . using Terraform var files uat.tfvars

Unfortunately changing my Azure Devops pipeline to use the passed in terraform directory (terraform) (and adjust the workingDirectory for the bash commands) I get the following output for Generate Baseline:

2024-08-29T18:46:54Z INFO Autodetected 4 Terraform projects across 1 root module
2024-08-29T18:46:54Z INFO Found Terraform project "main-dev" at directory "." using Terraform var files "dev.tfvars"
2024-08-29T18:46:54Z INFO Found Terraform project "main-prd" at directory "." using Terraform var files "prd.tfvars"
2024-08-29T18:46:54Z INFO Found Terraform project "main-qa" at directory "." using Terraform var files "qa.tfvars"
2024-08-29T18:46:54Z INFO Found Terraform project "main-uat" at directory "." using Terraform var files "uat.tfvars"
2024-08-29T18:46:55Z INFO Output saved to /tmp/infracost-base.json

And the Generate Diff I get:

2024-08-29T18:46:56Z INFO Autodetected 4 Terraform projects across 1 root module
2024-08-29T18:46:56Z INFO Found Terraform project "main-dev" at directory "." using Terraform var files "dev.tfvars"
2024-08-29T18:46:56Z INFO Found Terraform project "main-prd" at directory "." using Terraform var files "prd.tfvars"
2024-08-29T18:46:56Z INFO Found Terraform project "main-qa" at directory "." using Terraform var files "qa.tfvars"
2024-08-29T18:46:56Z INFO Found Terraform project "main-uat" at directory "." using Terraform var files "uat.tfvars"
Error: Invalid --compare-to Infracost JSON, found duplicate project name entity/functions/Event.Publisher/terraform

Does the path

entity/functions/Event.Publisher/terraform

mean anything to you? is there such a path in the repo?

Admittedly it's a bit strange, because that translates to org/project/repo/, and then we're at the root so would expect to see terraform listed next (which checks out, it is)

@gifted-sandwich-69781 the above was going to be my suggestion - thanks for digging in! You can also try this simpler config file:

#infracost.yml.tmp
version: 0.1
autodetect:
  env_names:
    - dev
    - qa
    - uat
    - prd

infracost generate config --repo-path . --template-path infracost.yml.tmp --out-file infracost.yml

infracost breakdown --config-file infracost.yml

If the above works ok locally, you can use that in CI/CD too. If all of your repos have one or many of those 4 envs then you can put that template in the pipeline instead of adding it to every repo.

Not sure if I should split out a separate help, but I was reviewing the docs for integration with Azure DevOps and specifically the private modules (e.g. we have modules in another project/repo that we then share to the devs so they can manage their own code), but it seems that the suggestion is to use an SSH auth token.. which sadly Azure DevOps only supports RSA and not ed25519.. Is there any opportune way to use the baked in System.AccessToken? I've tried adding it to my .git-credentials with a bash step (echo "https://$SYSTEM_ACCESSTOKEN@dev.azure.com" >> ~/.git-credentials) but that wasn't met with any success and I still bomb out an error on 'fataklcould not read Username for 'https://dev.azure.com'' (in the posted PR comment)

@gifted-sandwich-69781 sorry for the delay, how are the modules referenced in the code? SSH? HTTPS? or are they downloaded from a terraform registry? The infracost CLI uses the same method that Terraform uses to download the modules, so if they’re SSH modules then an SSH key is needed. Put another way, whatever creds you have in CI/CD for

terraform init

to work, you can re-use those creds for the Infracost CLI as it uses the same method to download the modules.

We're using the TerraformTaskV4@4.227.24 task, and passing in a service connection to get that working. I will work to sort out how to perhaps pass that in if I can.. e.g.:

jobs:

- deployment: TerraformApply${{parameters.environment}} displayName: Terraform Apply ${{ parameters.environment }} environment: ${{ parameters.environment }} strategy: runOnce: deploy: steps: - checkout: self path: self - ${{ each repository in parameters.module_repositories }}: - checkout: ${{ repository }} persistCredentials: true - script: | git config --global http.https://dev.azure.com.extraheader "Authorization: Bearer $(System.AccessToken)" displayName: Git Authentication - task: TerraformInstaller@1 inputs: terraformVersion: ${{ parameters.terraform_version }} - task: TerraformTaskV4@4.227.24 displayName: Terraform Init inputs: provider: azurerm command: init backendServiceArm: ${{ parameters.backend_service_connection }} backendAzureRmResourceGroupName: ${{ parameters.backend_resource_group }} backendAzureRmStorageAccountName: ${{ parameters.backend_storage_account }} backendAzureRmContainerName: ${{ parameters.backend_container }} backendAzureRmKey: ${{ parameters.environment }}.tfstate workingDirectory: $(Build.Repository.LocalPath)/${{ parameters.terraform_dir }}

@gifted-sandwich-69781 this is the key bit that’s checking out the module repos for your setup so when your terraform init runs, it doesn’t need to download any modules as they have already been done I think. > - ${{ each repository in parameters.module_repositories }}: > - checkout: ${{ repository }} > persistCredentials: true if you grep your TF repo for

module "

what is the

source

param of one of them set to? that’ll tell us how the modules are being accessed (HTTPS or SSH, the following is an SSH example), it might be a mixture of SSH and HTTPS as you also have the

git config --global

command set above > module “my-module” { > source = “git@github.com:my-org/terraform-private-module.git” > ref = “v1.2.3”