https://infracost.io logo
Powered by
Title
m

mammoth-painting-38799

10/18/2022, 7:25 AM
I have self hosted cloud-pricing-api in my server which doesn’t have internet/blocked by firewall...is there any way to update manually and where i can see/change the actual pricing URLs
b

busy-agent-35515

10/18/2022, 7:47 AM
This is a default URL for data download - 
<https://pricing.api.infracost.io>
, it is defined here: https://github.com/infracost/cloud-pricing-api/blob/c19f79972788a88dffa60d47abf13ac96048524a/src/config.ts#L82 Where it is used for data download: https://github.com/infracost/cloud-pricing-api/blob/636115f365e6fc442fc1194a1da18cd42ba5a94c/src/cmd/dataDownload.ts#L28
Downloading pricing updates is optional, it's convenient as we collect all prices in one file. It is possible to download prices from cloud vendors directly without using Infracost data. This is a dedicated command: 
npm run data:scrape
data:scrape
There are no default firewall rules, this allows you to decide which way you'd like to protect the self-hosted CPAPI endpoints.
Does this answer your question?
m

mammoth-painting-38799

10/18/2022, 8:45 AM
Thanks for details. my firewall allows only limited domains like *aws etc....i will try to npm run data:scrape and see.
b

busy-agent-35515

10/18/2022, 8:50 AM
It's a long process and it may require API keys for cloud vendors (for example, for Google) to use their APIs. The ideal solution would be whitelisting the Infracost domain
m

mammoth-painting-38799

10/18/2022, 9:53 AM
Thank you for all the details
b

busy-agent-35515

10/18/2022, 9:55 AM
Sure thing! 🙂
m

mammoth-painting-38799

10/18/2022, 10:15 AM
@busy-agent-35515 for the first run of cloud-pricing-api, need internet egress call for which is either init_job or data:scape right?
b

busy-agent-35515

10/18/2022, 10:21 AM
Yes, to work properly the CPAPI needs data. It either downloads the data dump from Infracost via 
npm run job:init
or you run the scraper. With the scrape approach you can also choose the cloud vendor, these are the available options for the flag 
--only=[aws:bulk,aws:spot,azure:retail,gcp:catalog,gcp:machineTypes]
BTW, how do you deploy the CPAPI? Docker, k8s or as a node app?
m

mammoth-painting-38799

10/18/2022, 10:35 AM
As podman, within firewall... currently we are doing POC
b

busy-agent-35515

10/18/2022, 11:00 AM
Thank you. I assume that you're using the docker-compose.yml as an example then mentioning 
init_job
m

mammoth-painting-38799

10/18/2022, 11:03 AM
Yes, correct. all clear so far
b

busy-agent-35515

10/18/2022, 11:03 AM
Awesome!
BTW, we've recently released a GitHub App integration in Infracost Cloud. It allows a super easy setup without need to use CI/CD pipeline. Ali wrote about this in a blog post https://www.infracost.io/blog/oct-2022-update/#github-app
Maybe it can work as a PoC for you
m

mammoth-painting-38799

10/18/2022, 11:07 AM
Thank you and really helpful
@busy-agent-35515 1st i have created network as 
docker network create cloud_pricing_api
which created bridge network 2nd run postgres:13.4 as container, started successfully 
starting PostgreSQL 13.4 (Debian 13.4-4.pgdg110+1) on x86 _64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
listening on IPv4 address"0.0.0.0".port 5432
listening on IPv6 address"::", port 5432
listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
database system was shut down at 2022-10-19 07:09:16 UTC
database system is ready to accept connections
3rd when i run init job, getting below error after 10 retry 
{"level":50,"time":1666164483057,'pid :29, " hostname": "545af87929a2"msg': "Waiting for Postgres@L
to become available: connect ECONNREFUSED 127.0.0.1:5432 }
("level":50, "time":1666164493066, "pid":29, "hostnalme" •"545af8792922"erp"Error."message":""Failed to connect to PostereSOL". "stack": "Error: Failed to connect to PostereSOLIn
at /usr/src/app/dist/cmd/db
Setup.js:80:31\n
at step (/usr/src/app/dist/cmd/dbSetup.is:33:23)\n
Object.next (/usr/src/app/dist/cmd/dbSetup.is:14:53n
at fulfilled (/usr/src/app/dist/cmd/dbSetup.JS:5:58)h,msg:Falledtoconnect
to PostgreSOL"}
nom ERRI Code ELIFECYCLE
nom ERR. errno 1
pm ERR! Cloud-pricing-api@0.3.7 db: setup:
*node dist/cmd/dbSetup.js
nom ERR.
pm ERR! Failed at the cloud-pricing-apige.3.7 db:setup script.
nom ERRI This is probablv not a problem with nom.
mare
likely
additional lossing
outnut above.
pm ERR! A complete log of this run can be found in:
/root/.npm/_logs/2022-10-19T07_28_13_0872-debug.log
nom ERR! Code ELIFECYCLE
errno
pm ERR! cloud-pricing-api@@.3.7 job:init: 'npm run db:setup && npm run job:update
npm ERR! Exit status
pm ERR! Failed at the cloud-pricing-api@o.3.7 job: init script.
npm ERR. In1s 15 probably not a problem witn nom.
There is likely additional logging output above.
pm ERR! A complete l0g of this run can be found in:
nomER
/root/.npm/_logs/2022-10-19T07_28_13_118-debug.10g
pls help how to fix this. Highly appriciated
b

busy-agent-35515

10/19/2022, 8:23 AM
Hey! On the first glance 
connect ECONNREFUSED 127.0.0.1:5432
looks suspicious, it tries to connect to localhost, not your network. Does the init_job container use the network?
It relies on 
POSTGRES_HOST=postgres
and 
networks: - cloud_pricing_api
in the docker-compose.yml
https://github.com/infracost/cloud-pricing-api/blob/master/docker-compose.yml#L14-L26
m

mammoth-painting-38799

10/19/2022, 8:32 AM
@busy-agent-35515 Yes i run the below podman commands, anything i am missing here? 
docker network create cloud _pricing_api

#db
docker run -d\
--network cloud pricing api \
-v postgres-data:/var/lib/postgresql/data/ \
-u posteres \
- POSTGRES_PASSWORD=postgres \
--health-cmd="CMD-SHELL" \
--health-cmd="pg_isready" \
--health-interval=60s \
--health-timeout=10s \
--health-retries=3 \
--health-start-period=10s \
-restart on-failure \
postgres:13.4

#init job
docker run
--network cloud pricing_api \
-e INFRACOST_API KEY \
-e POSTGRES_HOST \
-e POSTGRES_DB \
-e POSTGRES_USER \
-e POSTGRES_PASSWORD \
cloud-pricing-api:latest npm run job:init
i tried with POSTGRES_HOST as localhost and postgress… both the ways its not working.
b

busy-agent-35515

10/19/2022, 8:45 AM
Can you try adding 
-h postgres
to the postgres container? This way you'll set its hostname, so you can use 
POSTGRES_HOST=postgres
with the init job (and other containers that depend on the db)
m

mammoth-painting-38799

10/19/2022, 8:51 AM
i did this.. now different error during init_job 
Waiting for PostgreSQL to become available: getaddrinfo ENOTFOUND postgres
@busy-agent-35515 i suspect something todo with network? i see default bridge nework is created with 
docker network create cloud _pricing_api
b

busy-agent-35515

10/19/2022, 8:57 AM
Hm, weird. Okay. Maybe you can get the IP of the postgres container for now and use it as a env var host?
Yes, this is not related to CPAPI functionality, just the network setup
m

mammoth-painting-38799

10/19/2022, 8:58 AM
sure will try with its ip
b

busy-agent-35515

10/19/2022, 8:58 AM
With our docker-compose config they see each other no problem, however something's tricky happens here
I also apologize as I'm not an expert with docker networking 😅
Doing some googling, seems like podman sets hostnames using containers' names. Another option to try - set a name to the postgres container 
--name postgres
instead of 
-h
one
And it seems running 
podman pod inspect <container>
should tell you what the hostname is
m

mammoth-painting-38799

10/19/2022, 10:50 AM
@busy-agent-35515 after changing 
--network host
, 
init_job
it, 
data_scare
working with download errors, now i am getting new error while dataDownload & dataScpae both 
There was an error downloading data: connect EINVAL 0.0.1.187:80 - Local (0.0.0.0:0)
I can confirm that no issue with infracost.io, amazonaws.com,*googleapis.com,*azure.com domain access
b

busy-agent-35515

10/19/2022, 11:13 AM
Hm, the dataDownload tries to reach 
<https://pricing.api.infracost.io>
. Quick googling showed that the reason for this error might be missing protocol (https://), but I find it strange as the default endpoint certainly has it. Here's an SO where I found it https://stackoverflow.com/questions/48939454/error-connect-einval-0-0-xx-xx80-local-0-0-0-00
m

mammoth-painting-38799

10/19/2022, 11:21 AM
strange
@busy-agent-35515 i can confirm that am able to download the dump.tgz usung curl command inside the CPAPI container (using docker exec). However when i run the init job getting error 
There was an error downloading data: connect EINVAL 0.0.1.187:80 - Local (0.0.0.0:0)
any workaround or fix please? Appreciated your help
b

busy-agent-35515

10/20/2022, 8:39 AM
Just to double check, you don't have 
INFRACOST_PRICING_API_ENDPOINT
env var set, right?
m

mammoth-painting-38799

10/20/2022, 8:44 AM
Correct I haven't set INFRACOST_PRICING_API_ENDPOINT
b

busy-agent-35515

10/20/2022, 9:05 AM
Okay. Plan B. You have the dump file downloaded, that's good. You can use another command 
npm data:load --path=path/to/file
to populate the database. I'm sorry, it's not the solution for the issue, but it should help unblock you with the CPAPI
Oops, hold on, I might be mistaken with the command Nope, all's good. The task expects 
csv.gz
file
m

mammoth-painting-38799

10/20/2022, 10:21 AM
@busy-agent-35515 CPAPI using Node version v14.15.4 & npm version 6.14.10 should be okey?
b

busy-agent-35515

10/20/2022, 10:22 AM
Hm, we use 16.13
FYI: https://github.com/infracost/cloud-pricing-api/blob/master/Dockerfile#L11
I guess using 16.14 should be alright too
m

mammoth-painting-38799

10/20/2022, 10:27 AM
Will update to 16.14 and test. will see how it goes
@busy-agent-35515 after updating to 16.14, during init_job, getting EACCES permission denied error... I need to set permissions?
b

busy-agent-35515

10/21/2022, 10:16 AM
Seems like your user doesn't have permissions to delete files. I guess you can either run as root, or use a custom user and use a directory different from 
/root
to save/delete these files
m

mammoth-painting-38799

10/25/2022, 3:08 AM
@busy-agent-35515 @white-airport-8778 I have hosted CPAPI locally running http://localhost:4000, genereted $SELF_HOSTED_INFRACOST_API_KEY After auth login api key stored in ~/.config/infracost/credentials.yml export INFRACOST_PRICING_API_ENDPOINT= http://localhost:4000 export INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY export INFRACOST_API_KEY=“<<MY API KEY>>” infracost configure set pricing_api_endpoint http://localhost:4000 infracost configure set api_key $SELF_HOSTED_INFRACOST_API_KEY infracost configure set tls_insecure_skip_verify true i can confirm that 
curl -i $INFRACOST_PRICING_API_ENDPOINT/health
is 200 okey, 
curl -i $INFRACOST_PRICING_API_ENDPOINT/graphql -H "X-Api-Key: $SELF_HOSTED_INFRACOST_API_KEY
is 400 Get query missing when i run infracost breakdown --path . 
Error while Retrieving cloud prices to calculate costs
Error: Invalid API key
Please check your /home/user1/.config/infracost/credentials.yml or INFRACOST_API_KEY env variable.
i am 200% sure that /home/user1/.config/infracost/credentials.yml key is correct. What wrong here ? thanks in Advance
b

busy-agent-35515

10/25/2022, 8:14 AM
Okay, it seems like there is some confusion about the API keys. Let me reiterate a bit: 1. First, there are 2 apps in use: the CPAPI and CLI 2. CPAPI needs to have 
SELF_HOSTED_INFRACOST_API_KEY
generated and exported - this is the key that CLI will use to authenticate with it (for example, it will be 
cpapi-key
). For CPAPI you can also set up 
INFRACOST_API_KEY
with the one you got from Infracost - this is needed to update the database. For example, it will be 
infra-key
. 3. On CLI side you need to configure two env vars: 
INFRACOST_PRICING_API_ENDPOINT
to point to your CPAPI (you mentioned 
<http://localhost:4000>
). And CLI should also have 
INFRACOST_API_KEY
generated by CPAPI, from example it should be 
cpapi-key
This is important, because CLI tries to connect to your instance of CPAPI, it should use its key 
cpapi-key
, not the one Infracost generated for you.
You can either set the CLI's env vars, or set the credentials yaml with the following values: 
api_key: cpapi-key
pricing_api_endpoint: <http://localhost:4000>
The env vars take precedence over the config file
If I read your message correctly the confusion comes from these lines: 
export INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY
export INFRACOST_API_KEY="<<MY API KEY>>"
m

mammoth-painting-38799

10/25/2022, 8:28 AM
Thanks you for detailed response, seems typo in the question...what i set is export SELF_HOSTED_INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY export INFRACOST_API_KEY="<<INFRACOST KEY>>" CLI side i set endpoint, generated cpapi-key Same error
Also i noticed is when i set infracost configue set api_key=generated cpapi-key, its overriding credentials.yml file
b

busy-agent-35515

10/25/2022, 8:31 AM
This is correct, that's the purpose of the 
configure set
command - it stores values in the credentials yaml
m

mammoth-painting-38799

10/25/2022, 8:33 AM
Tried 3/4 times same error
b

busy-agent-35515

10/25/2022, 8:34 AM
This should be done for CPAPI: 
export SELF_HOSTED_INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY
export INFRACOST_API_KEY=infra-key
And this should be done for CLI: 
export INFRACOST_PRICING_API_ENDPOINT=<http://localhost:4000>
export INFRACOST_API_KEY=cpapi-key
It's important that the export is done in different sessions
Can you try to run this command? 
INFRACOST_API_KEY=cpapi-key infracost breakdown --path .
replacing the cpapi-key with your actual key?
This way it would apply specifically for the command ignore anything set in your environment
m

mammoth-painting-38799

10/25/2022, 8:37 AM
I set all env vars in one session and trying to run 
infracost breakdown --path .
Is this could be issue?
b

busy-agent-35515

10/25/2022, 8:38 AM
echo $INFRACOST_API_KEY
Which key does it show?
and 
echo $SELF_HOSTED_INFRACOST_API_KEY
?
m

mammoth-painting-38799

10/25/2022, 9:16 AM
I set $INFRACOST_API_KEY with infracost given key and $SELF_HOSTED_INFRACOST_API_KEY with generated key in the CLI breakdown session Seems these are clashing. Do i need to set only one ?
b

busy-agent-35515

10/25/2022, 9:17 AM
CPAPI needs both, CLI needs only INFRACOST_API_KEY with the one you generated in CPAPI
If you're running both apps in the same session, CLI will use INFRACOST_API_KEY that Infracost provided. Connecting to your localhost will result in error, because your CPAPI app doesn't recognize this key
If you run the CLI command using this notation 
INFRACOST_API_KEY=cpapi-key infracost breakdown --path .
it should not clash. Can you try that?
m

mammoth-painting-38799

10/25/2022, 9:53 AM
Got it...yes my CPAPI & CLI running in same server.. so will try with given suggetion and let you know. Thank you for all details
b

busy-agent-35515

10/25/2022, 9:55 AM
Good luck!
m

mammoth-painting-38799

10/25/2022, 2:40 PM
@busy-agent-35515 awesome `INFRACOST_API_KEY=cpapi-key infracost breakdown --path .`worked. My suggestion is add a special note for localhost users regarding for CPAPI and CLI ENV vars usage
b

busy-agent-35515

10/25/2022, 2:42 PM
Great to hear! And thanks for the suggestion. Usually the localhost setup uses 
.env
file where the env vars are defined. And we recommend to use the docker compose approach as it's fast and clean. Unfortunately, we can't account for all the corner cases. Sorry, that your case was one of those 🙂
2 Views
#helpJoin Slack
Powered by