https://infracost.io logo
Title
d

dazzling-journalist-91580

02/17/2023, 8:58 PM
@mysterious-teacher-68276 is there a channel you could pull me into for talking about the api or would you prefer a private message?
m

mysterious-teacher-68276

02/17/2023, 8:59 PM
which API are you referring to?
d

dazzling-journalist-91580

02/17/2023, 8:59 PM
cloud-pricing-api
m

mysterious-teacher-68276

02/17/2023, 9:00 PM
we don’t have a dedicated channel for that atm, are you having problems with it?
d

dazzling-journalist-91580

02/17/2023, 9:00 PM
it's relating to github issue 2091 and you saying you're welcoming contributions... I've done the changes to my local, but it's still for some reason pulling/sourcing from your api's output csv which obviously does not have the necessary machine types
m

mysterious-teacher-68276

02/17/2023, 9:01 PM
ah right, gotcha, just your GitHub handle is different, so didn’t piece it together
do you have a instance of the cloud pricing api running locally?
d

dazzling-journalist-91580

02/17/2023, 9:02 PM
yes
add_missing_machinetypes.diff
the problem is, even if I run
docker-compose run update_job
it just downloads your csv instead of actually scraping google itself.
m

mysterious-teacher-68276

02/17/2023, 9:04 PM
let me have a look at the src, I haven’t looked at it in a bit
d

dazzling-journalist-91580

02/17/2023, 9:05 PM
either way, I would highly recommend that you stop hard-coding machine types... especially since there are super convenient rest endpoints that let you query what machine types are available worldwide
m

mysterious-teacher-68276

02/17/2023, 9:05 PM
yeah I’m not sure why we do that tbh, I’m sure there was a reason at some point
d

dazzling-journalist-91580

02/17/2023, 9:06 PM
I think I need to run something like
data:scrape:dev
as an npm script
😅 except
ts-node
doesn't exist in the docker container
m

mysterious-teacher-68276

02/17/2023, 9:07 PM
yeah just run it on your machine
pointing to the exposed db ports
d

dazzling-journalist-91580

02/17/2023, 9:09 PM
it really shows that you're mostly focused on aws
m

mysterious-teacher-68276

02/17/2023, 9:09 PM
yeah most of our customers are AWS or Azure
d

dazzling-journalist-91580

02/17/2023, 9:10 PM
that's probably because your google side isn't very well supported
m

mysterious-teacher-68276

02/17/2023, 9:11 PM
yeah it’s mostly because of cloud provider market share tbh
you managed to get the
data:scrapce:dev
working?
d

dazzling-journalist-91580

02/17/2023, 9:12 PM
yeah, it's running now, though I'm not sure if it's going to do the right thing or not, it's just downloading a bunch of aws jsons right now
m

mysterious-teacher-68276

02/17/2023, 9:13 PM
nice
d

dazzling-journalist-91580

02/17/2023, 9:14 PM
ah, it's not pleased, 1s
m

mysterious-teacher-68276

02/17/2023, 9:18 PM
FYI I’m gonna log off in 5, (it’s late here in the UK), but I’ll leave you in the capable hands of @crooked-daybreak-55253 / @white-airport-8778 who are in US timezones
d

dazzling-journalist-91580

02/17/2023, 9:21 PM
(I had to add a
ports:
stanza to postgres in docker-compose.yml in order to expose 5432)
have a nice weekend, Hugo
@crooked-daybreak-55253 @white-airport-8778 could either of you advise on how it is exactly that you're determining what the active region is? I believe the fallback to us-central1 is happening in
infracost
itself, instead of in the api
confirmed that it works with t2d
Screen Shot 2023-02-17 at 4.56.38 PM.png
w

white-airport-8778

02/17/2023, 9:59 PM
Thanks @dazzling-journalist-91580! you’re right that the fallback is happening in the CLI, also thanks for including that api that returns the instance types, I can’t remember why we didn’t use that API, it might have been because when we wrote that code originally ~2yrs ago we didn’t want to force users to enter GCP creds to just run that service (but things have changed a lot and users can download the pricing DB dump from us so we should switch to using that api at some point to avoid this confusion again) 🙂
d

dazzling-journalist-91580

02/17/2023, 10:01 PM
well, you're stuck using credentials anyway, one other point in that regard, google has a LOT of helpers available, so requiring the use of
GCP_KEY_FILE
is kinda an antipattern now since application default credentials exist: https://cloud.google.com/nodejs/docs/reference/google-auth-library/latest#choosing-the-correct-credential-type-automatically
the funny thing? I'm a systems architect that works primarily with python, but I've had to 🔨 my devs about using application defaults myself in the past
w

white-airport-8778

02/17/2023, 10:03 PM
😄 defaults and overrides, users have 🔨 us about those
d

dazzling-journalist-91580

02/17/2023, 10:05 PM
either way, I'm now unblocked. I'm not familiar enough with golang to fix what I presume to be a parse error in the region selector, but I can at least get a ballpark estimate
on a personal note, it's really sketchy that the repository url is being picked up (and presumably submitted) even in the open source version and some of the errors are potentially misleading.... specifically
time="2023-02-17T12:13:37-05:00" level=debug result="{CloudEnabled:false ActualCostsEnabled:false UsageAPIEnabled:false}" msg="Successfully loaded settings from Infracost Cloud"
w

white-airport-8778

02/17/2023, 10:07 PM
I’ll take a look at that line as I’m pretty sure that’s misleading, when you self-host there is no Infracloud Cloud
d

dazzling-journalist-91580

02/17/2023, 10:08 PM
I wasn't self-hosted at the beginning of the day: https://github.com/infracost/infracost/issues/2091
I only went self-hosted so I could fix the missing machineTypes
w

white-airport-8778

02/17/2023, 10:10 PM
right yeah I saw your GH comment, sorry it’s been a long week of debugging user issues
d

dazzling-journalist-91580

02/17/2023, 10:10 PM
yeah, it happens, I had a dev try to send a 1.1GB json response today... so, I feel you
the user was like "why does it take 30 minutes to load this project?"
@mysterious-teacher-68276 I changed the comment to a TODO with a reference to the function name used later. I'm not a js/typescript developer, so that level of refactor is beyond me
m

mysterious-teacher-68276

02/20/2023, 2:13 PM
sounds good, I’m just running the build then I’ll merge and release to the hosted API
d

dazzling-journalist-91580

02/20/2023, 2:13 PM
awesome, thanks
thanks for the merge ❤️
m

mysterious-teacher-68276

02/20/2023, 2:41 PM
ok that’s deployed, we should pick up the new machine types when we run a scrape tonight ~4am UTC. I’ll test tomorrow.