Hey All So I ve been testing the new addition of sync usage

Question

Hey All So I ve been testing the new addition of sync usage file and run into a small issue when testing it with atlantis diff Namely if I add the option for it to the diff script will send a PR after I get the test working then it creates the usage file from scratch fine but it seems that it breaks the summary json file by not grepping out the below line before outputting to json ``` Synced 0 of 6 resources``` I guess it could be worked around with grep v in the script but that seems like a hack so wanted to check did somebody else run into this or is this something you re aware off

Andrzej Piorkowski · Answer

Debug ```time= 2021 10 25T13 09 38Z level=info msg= starting Syncing usage data from cloud time= 2021 10 25T13 09 38Z level=info msg= starting Calculating monthly cost estimate Running infracost output using $ infracost output no color format diff path infracost breakdown json Error Error parsing JSON file invalid character â looking for beginning of value```

Alistair (Infracost) · Answer

Thanks < Andrzej Piorkowski> yeah this looks like a bug I ve managed to reproduce it The fix will be to change this line to `cmd PrintErrln`

Alistair (Infracost) · Answer

Let me know if you want to send a PR for this Otherwise I can do it We ll get this fix out in the next release Wednesday

Andrzej Piorkowski · Answer

Ah stdout vs stderr Ok

Andrzej Piorkowski · Answer

I can add it to my branch since it already has the addition of new var for the sync usage file to atlantis

Alistair (Infracost) · Answer

raised hands thanks

Ali (Infracost) · Answer

< Andrzej Piorkowski> out of interest why are you syncing the usage file in atlantis because that has the creds to call cloudwatch

Andrzej Piorkowski · Answer

So I think the answer is twofold 1 Cause I didn t actually think about syncing it on my local machine before messing with atlantis already wink 2 Still figuring out the process testing everything but my original idea was that if atlantis could do it automatically this would cut down on the manual step

Andrzej Piorkowski · Answer

But longterm I think it would be good if this was automatic out of sight out of mind type of thing where this will just work automatically to supplement the already great estimates

Ali (Infracost) · Answer

That makes sense Recently I ve also been thinking whether sync from cloudwatch should happen automatically as it provides a more accurate estimate so it could be tried and we can fall back to just showing prices if it fails to connect with the cloud there could be an option to disable this for ppl who do not want it

Andrzej Piorkowski · Answer

Yep so for now my current testing was focused on adding a new env var switch in atlantis diff which when set to true will try to do the sync

Andrzej Piorkowski · Answer

I guess looking at the synced 0 out of 6 it already fails open so if it doesn t work no big deal

Ali (Infracost) · Answer

I summarized the above learning in <https github com infracost infracost discussions 985> feel free to comment further this was very helpful slightly smiling face

Andrzej Piorkowski · Answer

Sent a PR with my hacky addition of the new env var and the fix mentioned by < Alistair Infracost >

Duncan · Answer

Re auto sync from CloudWatch I am being very paranoid but wondered how aggressive are CloudWatch rate limits Could Infracost exhaust them in a sufficiently large deployment which could have negative impact across wider orgs obviously But again a bit paranoid here

Ali (Infracost) · Answer

Good point the PR only added an option so users can enable it if they want it s off by default but if we move towards having that on by default then some sort of caching is probably needed specially in CI CD