Title
#help
g

GV R

10/18/2022, 7:25 AM
I have self hosted cloud-pricing-api in my server which doesn’t have internet/blocked by firewall...is there any way to update manually and where i can see/change the actual pricing URLs
Vadim (Infracost)

Vadim (Infracost)

10/18/2022, 7:47 AM
7:49 AM
Downloading pricing updates is optional, it's convenient as we collect all prices in one file. It is possible to download prices from cloud vendors directly without using Infracost data. This is a dedicated command:
npm run data:scrape
7:53 AM
There are no default firewall rules, this allows you to decide which way you'd like to protect the self-hosted CPAPI endpoints.
7:56 AM
Does this answer your question?
g

GV R

10/18/2022, 8:45 AM
Thanks for details. my firewall allows only limited domains like *aws etc....i will try to npm run data😒crape and see.
Vadim (Infracost)

Vadim (Infracost)

10/18/2022, 8:50 AM
It's a long process and it may require API keys for cloud vendors (for example, for Google) to use their APIs. The ideal solution would be whitelisting the Infracost domain
g

GV R

10/18/2022, 9:53 AM
Thank you for all the details
Vadim (Infracost)

Vadim (Infracost)

10/18/2022, 9:55 AM
Sure thing! 🙂
g

GV R

10/18/2022, 10:15 AM
@Vadim (Infracost) for the first run of cloud-pricing-api, need internet egress call for which is either init_job or data😒cape right?
Vadim (Infracost)

Vadim (Infracost)

10/18/2022, 10:21 AM
Yes, to work properly the CPAPI needs data. It either downloads the data dump from Infracost via
npm run job:init
or you run the scraper. With the scrape approach you can also choose the cloud vendor, these are the available options for the flag
--only=[aws:bulk,aws:spot,azure:retail,gcp:catalog,gcp:machineTypes]
10:22 AM
BTW, how do you deploy the CPAPI? Docker, k8s or as a node app?
g

GV R

10/18/2022, 10:35 AM
As podman, within firewall... currently we are doing POC
Vadim (Infracost)

Vadim (Infracost)

10/18/2022, 11:00 AM
Thank you. I assume that you're using the docker-compose.yml as an example then mentioning
init_job
g

GV R

10/18/2022, 11:03 AM
Yes, correct. all clear so far
Vadim (Infracost)

Vadim (Infracost)

10/18/2022, 11:03 AM
Awesome!
11:04 AM
BTW, we've recently released a GitHub App integration in Infracost Cloud. It allows a super easy setup without need to use CI/CD pipeline. Ali wrote about this in a blog post https://www.infracost.io/blog/oct-2022-update/#github-app
11:05 AM
Maybe it can work as a PoC for you
g

GV R

10/18/2022, 11:07 AM
Thank you and really helpful
8:14 AM
@Vadim (Infracost) 1st i have created network as
docker network create cloud_pricing_api
which created bridge network 2nd run postgres:13.4 as container, started successfully
starting PostgreSQL 13.4 (Debian 13.4-4.pgdg110+1) on x86 _64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
listening on IPv4 address"0.0.0.0".port 5432
listening on IPv6 address"::", port 5432
listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
database system was shut down at 2022-10-19 07:09:16 UTC
database system is ready to accept connections
3rd when i run init job, getting below error after 10 retry
{"level":50,"time":1666164483057,'pid :29, " hostname": "545af87929a2"msg': "Waiting for Postgres@L
to become available: connect ECONNREFUSED 127.0.0.1:5432 }
("level":50, "time":1666164493066, "pid":29, "hostnalme" •"545af8792922"erp"Error."message":""Failed to connect to PostereSOL". "stack": "Error: Failed to connect to PostereSOLIn
at /usr/src/app/dist/cmd/db
Setup.js:80:31\n
at step (/usr/src/app/dist/cmd/dbSetup.is:33:23)\n
Object.next (/usr/src/app/dist/cmd/dbSetup.is:14:53n
at fulfilled (/usr/src/app/dist/cmd/dbSetup.JS:5:58)h,msg:Falledtoconnect
to PostgreSOL"}
nom ERRI Code ELIFECYCLE
nom ERR. errno 1
pm ERR! Cloud-pricing-api@0.3.7 db: setup:
*node dist/cmd/dbSetup.js
nom ERR.
pm ERR! Failed at the cloud-pricing-apige.3.7 db:setup script.
nom ERRI This is probablv not a problem with nom.
mare
likely
additional lossing
outnut above.
pm ERR! A complete log of this run can be found in:
/root/.npm/_logs/2022-10-19T07_28_13_0872-debug.log
nom ERR! Code ELIFECYCLE
errno
pm ERR! cloud-pricing-api@@.3.7 job:init: 'npm run db:setup && npm run job:update
npm ERR! Exit status
pm ERR! Failed at the cloud-pricing-api@o.3.7 job: init script.
npm ERR. In1s 15 probably not a problem witn nom.
There is likely additional logging output above.
pm ERR! A complete l0g of this run can be found in:
nomER
/root/.npm/_logs/2022-10-19T07_28_13_118-debug.10g
pls help how to fix this. Highly appriciated
Vadim (Infracost)

Vadim (Infracost)

10/19/2022, 8:23 AM
Hey! On the first glance
connect ECONNREFUSED 127.0.0.1:5432
looks suspicious, it tries to connect to localhost, not your network. Does the init_job container use the network?
8:24 AM
It relies on
POSTGRES_HOST=postgres
and
networks: - cloud_pricing_api
in the docker-compose.yml
g

GV R

10/19/2022, 8:32 AM
@Vadim (Infracost) Yes i run the below podman commands, anything i am missing here?
docker network create cloud _pricing_api

#db
docker run -d\
--network cloud pricing api \
-v postgres-data:/var/lib/postgresql/data/ \
-u posteres \
- POSTGRES_PASSWORD=postgres \
--health-cmd="CMD-SHELL" \
--health-cmd="pg_isready" \
--health-interval=60s \
--health-timeout=10s \
--health-retries=3 \
--health-start-period=10s \
-restart on-failure \
postgres:13.4

#init job
docker run
--network cloud pricing_api \
-e INFRACOST_API KEY \
-e POSTGRES_HOST \
-e POSTGRES_DB \
-e POSTGRES_USER \
-e POSTGRES_PASSWORD \
cloud-pricing-api:latest npm run job:init
8:33 AM
i tried with POSTGRES_HOST as localhost and postgress… both the ways its not working.
Vadim (Infracost)

Vadim (Infracost)

10/19/2022, 8:45 AM
Can you try adding
-h postgres
to the postgres container? This way you'll set its hostname, so you can use
POSTGRES_HOST=postgres
with the init job (and other containers that depend on the db)
g

GV R

10/19/2022, 8:51 AM
i did this.. now different error during init_job
Waiting for PostgreSQL to become available: getaddrinfo ENOTFOUND postgres
@Vadim (Infracost) i suspect something todo with network? i see default bridge nework is created with
docker network create cloud _pricing_api
Vadim (Infracost)

Vadim (Infracost)

10/19/2022, 8:57 AM
Hm, weird. Okay. Maybe you can get the IP of the postgres container for now and use it as a env var host?
8:58 AM
Yes, this is not related to CPAPI functionality, just the network setup
g

GV R

10/19/2022, 8:58 AM
sure will try with its ip
Vadim (Infracost)

Vadim (Infracost)

10/19/2022, 8:58 AM
With our docker-compose config they see each other no problem, however something's tricky happens here
8:58 AM
I also apologize as I'm not an expert with docker networking 😅
9:01 AM
Doing some googling, seems like podman sets hostnames using containers' names. Another option to try - set a name to the postgres container
--name postgres
instead of
-h
one
9:02 AM
And it seems running
podman pod inspect <container>
should tell you what the hostname is
g

GV R

10/19/2022, 10:50 AM
@Vadim (Infracost) after changing
--network host
,
init_job
it,
data_scare
working with download errors, now i am getting new error while dataDownload & dataScpae both
There was an error downloading data: connect EINVAL 0.0.1.187:80 - Local (0.0.0.0:0)
I can confirm that no issue with infracost.io, amazonaws.com,*googleapis.com,*azure.com domain access
Vadim (Infracost)

Vadim (Infracost)

10/19/2022, 11:13 AM
Hm, the dataDownload tries to reach
<https://pricing.api.infracost.io>
. Quick googling showed that the reason for this error might be missing protocol (https://), but I find it strange as the default endpoint certainly has it. Here's an SO where I found it https://stackoverflow.com/questions/48939454/error-connect-einval-0-0-xx-xx80-local-0-0-0-00
g

GV R

10/19/2022, 11:21 AM
strange
3:43 AM
@Vadim (Infracost) i can confirm that am able to download the dump.tgz usung curl command inside the CPAPI container (using docker exec). However when i run the init job getting error
There was an error downloading data: connect EINVAL 0.0.1.187:80 - Local (0.0.0.0:0)
any workaround or fix please? Appreciated your help
Vadim (Infracost)

Vadim (Infracost)

10/20/2022, 8:39 AM
Just to double check, you don't have
INFRACOST_PRICING_API_ENDPOINT
env var set, right?
g

GV R

10/20/2022, 8:44 AM
Correct I haven't set INFRACOST_PRICING_API_ENDPOINT
Vadim (Infracost)

Vadim (Infracost)

10/20/2022, 9:05 AM
Okay. Plan B. You have the dump file downloaded, that's good. You can use another command
npm data:load --path=path/to/file
to populate the database. I'm sorry, it's not the solution for the issue, but it should help unblock you with the CPAPI
9:06 AM
Oops, hold on, I might be mistaken with the command Nope, all's good. The task expects
csv.gz
file
g

GV R

10/20/2022, 10:21 AM
@Vadim (Infracost) CPAPI using Node version v14.15.4 & npm version 6.14.10 should be okey?
Vadim (Infracost)

Vadim (Infracost)

10/20/2022, 10:22 AM
Hm, we use 16.13
10:26 AM
I guess using 16.14 should be alright too
g

GV R

10/20/2022, 10:27 AM
Will update to 16.14 and test. will see how it goes
10:14 AM
@Vadim (Infracost) after updating to 16.14, during init_job, getting EACCES permission denied error... I need to set permissions?
Vadim (Infracost)

Vadim (Infracost)

10/21/2022, 10:16 AM
Seems like your user doesn't have permissions to delete files. I guess you can either run as root, or use a custom user and use a directory different from
/root
to save/delete these files
g

GV R

10/25/2022, 3:08 AM
@Vadim (Infracost) @Ali (Infracost) I have hosted CPAPI locally running http://localhost:4000, genereted $SELF_HOSTED_INFRACOST_API_KEY After auth login api key stored in ~/.config/infracost/credentials.yml export INFRACOST_PRICING_API_ENDPOINT= http://localhost:4000 export INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY export INFRACOST_API_KEY=“<<MY API KEY>>” infracost configure set pricing_api_endpoint http://localhost:4000 infracost configure set api_key $SELF_HOSTED_INFRACOST_API_KEY infracost configure set tls_insecure_skip_verify true i can confirm that
curl -i $INFRACOST_PRICING_API_ENDPOINT/health
is 200 okey,
curl -i $INFRACOST_PRICING_API_ENDPOINT/graphql -H "X-Api-Key: $SELF_HOSTED_INFRACOST_API_KEY
is 400 Get query missing when i run infracost breakdown --path .
Error while Retrieving cloud prices to calculate costs
Error: Invalid API key
Please check your /home/user1/.config/infracost/credentials.yml or INFRACOST_API_KEY env variable.
i am 200% sure that /home/user1/.config/infracost/credentials.yml key is correct. What wrong here ? thanks in Advance
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 8:14 AM
Okay, it seems like there is some confusion about the API keys. Let me reiterate a bit:1. First, there are 2 apps in use: the CPAPI and CLI 2. CPAPI needs to have
SELF_HOSTED_INFRACOST_API_KEY
generated and exported - this is the key that CLI will use to authenticate with it (for example, it will be
cpapi-key
). For CPAPI you can also set up
INFRACOST_API_KEY
with the one you got from Infracost - this is needed to update the database. For example, it will be
infra-key
. 3. On CLI side you need to configure two env vars:
INFRACOST_PRICING_API_ENDPOINT
to point to your CPAPI (you mentioned
<http://localhost:4000>
). And CLI should also have
INFRACOST_API_KEY
generated by CPAPI, from example it should be
cpapi-key
8:15 AM
This is important, because CLI tries to connect to your instance of CPAPI, it should use its key
cpapi-key
, not the one Infracost generated for you.
8:16 AM
You can either set the CLI's env vars, or set the credentials yaml with the following values:
api_key: cpapi-key
pricing_api_endpoint: <http://localhost:4000>
8:17 AM
The env vars take precedence over the config file
8:17 AM
If I read your message correctly the confusion comes from these lines:
export INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY
export INFRACOST_API_KEY="<<MY API KEY>>"
g

GV R

10/25/2022, 8:28 AM
Thanks you for detailed response, seems typo in the question...what i set is export SELF_HOSTED_INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY export INFRACOST_API_KEY="<<INFRACOST KEY>>" CLI side i set endpoint, generated cpapi-key Same error
8:31 AM
Also i noticed is when i set infracost configue set api_key=generated cpapi-key, its overriding credentials.yml file
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 8:31 AM
This is correct, that's the purpose of the
configure set
command - it stores values in the credentials yaml
g

GV R

10/25/2022, 8:33 AM
Tried 3/4 times same error
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 8:34 AM
This should be done for CPAPI:
export SELF_HOSTED_INFRACOST_API_KEY=$SELF_HOSTED_INFRACOST_API_KEY
export INFRACOST_API_KEY=infra-key
And this should be done for CLI:
export INFRACOST_PRICING_API_ENDPOINT=<http://localhost:4000>
export INFRACOST_API_KEY=cpapi-key
8:35 AM
It's important that the export is done in different sessions
8:36 AM
Can you try to run this command?
INFRACOST_API_KEY=cpapi-key infracost breakdown --path .
replacing the cpapi-key with your actual key?
8:37 AM
This way it would apply specifically for the command ignore anything set in your environment
g

GV R

10/25/2022, 8:37 AM
I set all env vars in one session and trying to run
infracost breakdown --path .
Is this could be issue?
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 8:38 AM
echo $INFRACOST_API_KEY
8:38 AM
Which key does it show?
8:39 AM
and
echo $SELF_HOSTED_INFRACOST_API_KEY
?
g

GV R

10/25/2022, 9:16 AM
I set $INFRACOST_API_KEY with infracost given key and $SELF_HOSTED_INFRACOST_API_KEY with generated key in the CLI breakdown session Seems these are clashing. Do i need to set only one ?
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 9:17 AM
CPAPI needs both, CLI needs only INFRACOST_API_KEY with the one you generated in CPAPI
9:18 AM
If you're running both apps in the same session, CLI will use INFRACOST_API_KEY that Infracost provided. Connecting to your localhost will result in error, because your CPAPI app doesn't recognize this key
9:19 AM
If you run the CLI command using this notation
INFRACOST_API_KEY=cpapi-key infracost breakdown --path .
it should not clash. Can you try that?
g

GV R

10/25/2022, 9:53 AM
Got it...yes my CPAPI & CLI running in same server.. so will try with given suggetion and let you know. Thank you for all details
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 9:55 AM
Good luck!
g

GV R

10/25/2022, 2:40 PM
@Vadim (Infracost) awesome INFRACOST_API_KEY=cpapi-key infracost breakdown --path .worked. My suggestion is add a special note for localhost users regarding for CPAPI and CLI ENV vars usage
Vadim (Infracost)

Vadim (Infracost)

10/25/2022, 2:42 PM
Great to hear! And thanks for the suggestion. Usually the localhost setup uses
.env
file where the env vars are defined. And we recommend to use the docker compose approach as it's fast and clean. Unfortunately, we can't account for all the corner cases. Sorry, that your case was one of those 🙂