https://infracost.io logo
Title
g

glamorous-cat-93012

10/14/2022, 9:02 AM
BTW both infracost usage files have different settings for each environment for testing and demo purposes.
m

mysterious-teacher-68276

10/14/2022, 9:04 AM
hey @glamorous-cat-93012, could you post your full pipeline yaml/configuration so that I can understand all the steps you’re using
g

glamorous-cat-93012

10/14/2022, 9:22 AM
--- steps: - bash: | ssh-agent -a $(SSH_AUTH_SOCK) mkdir -p ~/.ssh echo "$(echo $GIT_SSH_KEY_BASE_64 | base64 -d)" | tr -d '\r' | ssh-add - ssh-keyscan ssh.dev.azure.com >> ~/.ssh/known_hosts displayName: Add GIT_SSH_KEY # condition: and(succeeded(), eq(variables['Build.Reason'], 'IndividualCI')) condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) env: GIT_SSH_KEY_BASE_64: $(gitSshKeyBase64) # Install the Infracost CLI, see https://github.com/infracost/infracost-azure-devops#infracostsetup - task: InfracostSetup@1 displayName: Setup Infracost # condition: and(succeeded(), eq(variables['Build.Reason'], 'IndividualCI')) condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) inputs: apiKey: $(infracostApiKey) # Clone the base branch of the pull request (e.g. main/master) into a temp directory. - bash: | branch=$(System.PullRequest.TargetBranch) branch=${branch#refs/heads/} echo $branch mkdir $(Pipeline.Workspace)/s/$(Build.Repository.Name)/base git clone git@ssh.dev.azure.com:v3/MetroBank/Metro%20Cloud%20Platform/$(Build.Repository.Name) \ --branch=${branch} --single-branch "$(Pipeline.Workspace)/s/$(Build.Repository.Name)/base" displayName: Checkout base branch condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), or( eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'Manual')) # Generate an Infracost cost estimate baseline from the comparison branch, so that Infracost can compare the cost difference. - bash: | pwd ls -ltr mkdir $(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports infracost breakdown --path=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/base/ \ --format=json \ --out-file=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost-base.json displayName: Generate Infracost cost estimate baseline workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name)/base condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), eq(variables['Build.Reason'], 'IndividualCI')) # Generate an Infracost diff and save it to a JSON file. - bash: | pwd ls -ltr infracost diff --path=$(Pipeline.Workspace)/s/$(Build.Repository.Name) \ --format=json \ --compare-to=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost-base.json \ --out-file=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost.json displayName: Generate Infracost diff workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name) condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), or( eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'Manual')) # - publish: '$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports' # displayName: 'Publish reports' # artifact: drop # Posts a comment to the PR using the 'update' behavior. # This creates a single comment and updates it. The "quietest" option. # The other valid behaviors are: # delete-and-new - Delete previous comments and create a new one. # new - Create a new cost estimate comment on every push. # See https://www.infracost.io/docs/features/cli_commands/#comment-on-pull-requests for other options. - bash: | infracost comment azure-repos --path=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost.json \ --azure-access-token=$(System.AccessToken) \ --pull-request=$(System.PullRequest.PullRequestId) \ --repo-url=$(Build.Repository.Uri) \ --behavior=new displayName: Post Infracost comment condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), or( eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'Manual'))
m

mysterious-teacher-68276

10/14/2022, 9:26 AM
Ok two follow-up questions: 1. Do you have this
infracost.yml
file checked into the main branch yet (or is it being introduced in this PR) 2. What is the output you’re getting on the PR, what projects is the
table
showing?
g

glamorous-cat-93012

10/14/2022, 9:33 AM
We did an ls -ltra on the directory to prove this ourselves and also attached the Pr comments.
m

mysterious-teacher-68276

10/14/2022, 9:34 AM
so the problem is this:
in your pipeline you specify
diff
&
breakdown
with
--path
rather than
--config-file
g

glamorous-cat-93012

10/14/2022, 9:35 AM
Sorry I posted the wring branch
2 secs
--- steps: - bash: | ssh-agent -a $(SSH_AUTH_SOCK) mkdir -p ~/.ssh echo "$(echo $GIT_SSH_KEY_BASE_64 | base64 -d)" | tr -d '\r' | ssh-add - ssh-keyscan ssh.dev.azure.com >> ~/.ssh/known_hosts displayName: Add GIT_SSH_KEY # condition: and(succeeded(), eq(variables['Build.Reason'], 'IndividualCI')) condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) env: GIT_SSH_KEY_BASE_64: $(gitSshKeyBase64) # Install the Infracost CLI, see https://github.com/infracost/infracost-azure-devops#infracostsetup - task: InfracostSetup@1 displayName: Setup Infracost # condition: and(succeeded(), eq(variables['Build.Reason'], 'IndividualCI')) condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) inputs: apiKey: $(infracostApiKey) # Clone the base branch of the pull request (e.g. main/master) into a temp directory. - bash: | branch=$(System.PullRequest.TargetBranch) branch=${branch#refs/heads/} echo $branch mkdir $(Pipeline.Workspace)/s/$(Build.Repository.Name)/base git clone git@ssh.dev.azure.com:v3/MetroBank/Metro%20Cloud%20Platform/$(Build.Repository.Name) \ --branch=${branch} --single-branch "$(Pipeline.Workspace)/s/$(Build.Repository.Name)/base" displayName: Checkout base branch condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), or( eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'Manual')) # Generate an Infracost cost estimate baseline from the comparison branch, so that Infracost can compare the cost difference. - bash: | pwd ls -ltr mkdir $(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports infracost breakdown --config-file $(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost.yml \ --format=json \ --out-file=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost-base.json displayName: Generate Infracost cost estimate baseline workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name)/base condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), eq(variables['Build.Reason'], 'IndividualCI')) # Generate an Infracost diff and save it to a JSON file. - bash: | pwd ls -ltr infracost diff --config-file $(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost.yml \ --format=json \ --compare-to=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost-base.json \ --out-file=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost.json displayName: Generate Infracost diff workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name) condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), or( eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'Manual')) - publish: '$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports' displayName: 'Publish reports' artifact: drop # Posts a comment to the PR using the 'update' behavior. # This creates a single comment and updates it. The "quietest" option. # The other valid behaviors are: # delete-and-new - Delete previous comments and create a new one. # new - Create a new cost estimate comment on every push. # See https://www.infracost.io/docs/features/cli_commands/#comment-on-pull-requests for other options. - bash: | infracost comment azure-repos --path=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost.json \ --azure-access-token=$(System.AccessToken) \ --pull-request=$(System.PullRequest.PullRequestId) \ --repo-url=$(Build.Repository.Uri) \ --behavior=update displayName: Post Infracost comment condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) # condition: and(succeeded(), or( eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'Manual'))
m

mysterious-teacher-68276

10/14/2022, 9:42 AM
hmm this does look strange
i’m wondering if it’s getting an old artefact
what happens when you remove
- publish: '$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports'
        displayName: 'Publish reports'
        artifact: drop
step
g

glamorous-cat-93012

10/14/2022, 9:52 AM
The publish step was just added at the end for us to be able to download the results, all it does is take the json files and publish them to a drop location so we can see them in the azdo dashboard.
Same results with or without this task
Another thing we noticed with InfraCost --path setting that does not make sense with config-file
m

mysterious-teacher-68276

10/14/2022, 9:53 AM
could you send me the
$(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost-reports/infracost.json
artefact in a private message
g

glamorous-cat-93012

10/14/2022, 9:55 AM
So when we use --path in the command like: infracost breakdown --path=$(Pipeline.Workspace)/s/$(Build.Repository.Name)/base/ \ infracost diff --path=$(Pipeline.Workspace)/s/$(Build.Repository.Name) \ You notice on break down we point to /base which in the temp main branch, then on diff we point to the current branch to we can compare both, right??
m

mysterious-teacher-68276

10/14/2022, 9:56 AM
yup
g

glamorous-cat-93012

10/14/2022, 9:56 AM
When you change to --config-file, there is no why to have two different paths for each branch so basically scans the same branch twice? --config-file $(Pipeline.Workspace)/s/$(Build.Repository.Name)/infracost.yml \
as the config file just has a field for --path and can't be changed between calls
m

mysterious-teacher-68276

10/14/2022, 9:57 AM
ah I think I see what’s going wrong with your yaml
g

glamorous-cat-93012

10/14/2022, 9:58 AM
The way we got around this was to switch working directories on the bash task, so we can use the config-file. workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name)/base workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name)
m

mysterious-teacher-68276

10/14/2022, 9:58 AM
yeah, that’s the correct way to do it
g

glamorous-cat-93012

10/14/2022, 9:59 AM
What have you spotted ?
m

mysterious-teacher-68276

10/14/2022, 10:00 AM
I thought I had but no joy sadly
can you send me that infracost.json
g

glamorous-cat-93012

10/14/2022, 10:00 AM
sure
m

mysterious-teacher-68276

10/14/2022, 10:16 AM
hmm @glamorous-cat-93012 this does look like a bug our side. It seems that we’re not populating the
past_resources
properly in for this case. I’ve managed to replicate your pr output locally - give me some time to figure out whats going on
g

glamorous-cat-93012

10/14/2022, 10:19 AM
Cool, no worries..
Also a great idea to update your doc to use workingDirectory when using --config-file as this caught us out for a while. workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name)/base workingDirectory: $(Pipeline.Workspace)/s/$(Build.Repository.Name) 🙂
m

mysterious-teacher-68276

10/14/2022, 10:28 AM
hey @glamorous-cat-93012 so I think i’ve found the issue (which turns out not to be a bug but more a UX issue). With that PR are you introducing any cost changes?
g

glamorous-cat-93012

10/14/2022, 10:37 AM
In the usage file we have different settings for each env and we also updated the modules to use different ec2 instances so these should show the break down of nonp and prod?
and we do not see the different envs breakdown in the dashboard
Strange, it's like it is not picking up the main resources at all from main.tf
only finding resources in the usage-file.yml
m

mysterious-teacher-68276

10/14/2022, 10:44 AM
It looks like it's not being able I pick up the usage file path
This fails silently so you wouldn't see this
Can I check that the path structure you've got is correct - where are you usage files in relation to the project
g

glamorous-cat-93012

10/14/2022, 10:45 AM
this is the ls from the root, they are stored in config/env/...
m

mysterious-teacher-68276

10/14/2022, 10:47 AM
Ok I've got to go offline for about an hour and ill pick this up when I'm back.
g

glamorous-cat-93012

10/14/2022, 10:47 AM
No worries, thanks for looking into this
coffee time
m

mysterious-teacher-68276

10/14/2022, 12:11 PM
hey @glamorous-cat-93012 I’m back
give me a shout when your about and if you’re free to jump on a zoom to help debug this and get you to success
g

glamorous-cat-93012

10/14/2022, 1:00 PM
Free in an hour, sneaked out for a cheaky Friday lunch beer with the laptop.... shhhh 🙂
m

mysterious-teacher-68276

10/14/2022, 1:01 PM
ha
I’ve got meetings from 2:30pm (BST) to 4pm. So ill be free after that
g

glamorous-cat-93012

10/14/2022, 2:55 PM
How's you cal for Monday morning ?
m

mysterious-teacher-68276

10/14/2022, 2:56 PM
very free, ping me an invite to hugo@infracost.io and let’s get to the bottom of it
first meeting is at 2:30pm (BST)
g

glamorous-cat-93012

10/14/2022, 3:11 PM
Hows 10:30 GMT for you will.pereira@metrobank.plc.uk gerard.kavanagh@metrobank.plc.uk Could you set it up as we don't have a corp zoom for this? Building a nice new templated terraform model and see InrfaCost fitting in.
m

mysterious-teacher-68276

10/14/2022, 3:12 PM
works fine with me - I’ll ping you now
Sent
g

glamorous-cat-93012

10/14/2022, 3:35 PM
see you Monday. Thanks `Hugo... great supprt.