https://infracost.io logo
Join Slack
Powered by
This message was deleted.
# help
b
This message was deleted.
b
Hello! Infracost CLI can show the cost of traffic between regions for specific resources. The cloud pricing API only provides the prices
In the nutshell, CLI fetches the prices and performs the calculation
p
Oh, great, that's exactly what I need, thanks for the help
b
Sure thing 🙂
p
Vadim, I have seen your infracost CLI it's good, but maybe do you have Rest API or GraphQL which can be called for the cost of traffic between regions for specific resources?
b
May I ask what is your use-case to understand what exactly can help you?
p
Yeah, so I have seen for example AWS EC2 those prices https://aws.amazon.com/ru/ec2/pricing/on-demand/#Data_Transfer we can not focus on the type of each instance, but what I need is that I want to retrieve a regional $cost for transferring outgoing data from Amazon EC2 to any regions within AWS per GB, I would also like to keep the cost up-to-date and similarly for other cloud providers and for this purpose I looking like Rest API's, if you have something like that, it would be great
w
@polite-animal-10151 one workaround you might be able to use just now is to find the Cloud Pricing API filters from the resource files, e.g. this one for data transfer, then pass those filters to the Cloud Pricing API’s GraphQL API, e.g. this example request. This curl call shows an example request being made. The Cloud Pricing API returns uptodate prices so that API call could be repeated when AWS/Azure/Google announce prices changes. I know this is not as easy as having an API that just takes filters and returns the prices as you suggest - hopefully we’ll get to that but we don’t have an ETA. (I’m one of the co-founders, looking after the product side of things, feel free to DM if you want to jump on a call to discuss further)
p
Thanks Ali, I'll take a look.
Hello, I tried to do like this
Copy code
curl <https://pricing.api.infracost.io/graphql> \ 
  -X POST \
  -H 'X-Api-Key: MY_KEY' \
  -H 'Content-Type: application/json' \
  --data 'query { 
  products(
    filter: {
      vendorName: "aws",
      service: "AWSDataTransfer",
      productFamily: "Data Transfer",
      region: "us-east-1",
      attributeFilters: [
        { key: "transferType", value: "InterRegion Outbound" },
        { key: "fromLocation", value: "US East (N. Virginia)" }
        { key: "toLocation", value: "US West (Oregon)" }
      ]
    },
  ) {
    attributes { key, value }
   prices { USD }
  }
}'
and I got {"error":"Bad request"} Maybe I'm doing something wrong and another question I found an example only for aws other providers you do not support?
b
Hey! We support AWS, Azure and Google. Let me try this request and see what happens 🙂
Alrighty, sorry, got a bit distracted for a moment. Here are couple issues with your query: 1. Our examples show GraphQL queries in GraphQL format. When we want to use curl, we have to reformat it to be a string. 2. There is a typo in 
fromLocation
line - there should be comma Here's how data should look like for your request:
Copy code
--data '{"query": "{ products( filter: { vendorName: \"aws\", service: \"AWSDataTransfer\", productFamily: \"Data Transfer\", region: \"us-east-1\", attributeFilters: [ { key: \"transferType\", value: \"InterRegion Outbound\" }, { key: \"fromLocation\", value: \"US East (N. Virginia)\" }, { key: \"toLocation\", value: \"US West (Oregon)\" } ] },) { attributes { key, value } prices { USD } } }"}'
BTW, here are examples for all 3 cloud providers, however you'll have to format them for curl too https://github.com/infracost/cloud-pricing-api/tree/master/examples/queries
Curl can be tedious for GraphQL, you can try using this instruction to use the playground https://github.com/infracost/infracost/blob/master/CONTRIBUTING.md#querying-the-graphql-api
p
ok, thanks it works
🙌 1
b
Another way might be installing the CPAPI locally. This way you'll have access to the database and can query it using SQL
p
Thanks, I will take a look. Vadim do you know if you have plan to connect other cloud providers, for example, OCI (Oracle)? if so, maybe you know the approximate dates?
b
Thanks for asking! This is the sort of thing we'd love to support longer term but our team is currently focused on supporting AWS, Azure and GCP so we prefer not to work on adding new vendors at this time. We would like to move towards a provider model where community-supported cloud vendors could also be added (similar to Terraform providers). There are couple open issues where you can subscribe for updates, but we can't provide any dates right now. https://github.com/infracost/infracost/issues/911
👀 1
p
Hey Vadim, a quick question: I see in the request body for AWS we have two fields: service: \"AWSDataTransfer\", productFamily: \"Data Transfer\" but correct me if I'm wrong, for Azure, the name of the service and the product family will be something like this or the name of the service will be different, the same for GCP, for example: service: \"AzureDataTransfer\", productFamily: \"Data Transfer\" service: \"GCPDataTransfer\", productFamily: \"Data Transfer\" and where can I see all the names of services and productFamily name's?
b
You are correct, the names will differ. We get these names from cloud vendors' price APIs directly and store them as is in the CPAPI database. As the db has millions of prices, it might be very hard to guess them. There are couple ways how to research the proper names: 1. Reading our CLI source code: you can find the resource you're interested in and look at its source code and see what filters its cost components use 2. Install CPAPI yourself and use its database to search through. In the guide I've mentioned before there is an instruction how you can narrow down the search
BTW, it's not just service/product family names, the filter key names will differ as well.
p
ok, thanks I will check
👍 1
Vadim, all it works thanks for the help, most likely we will use CPAPI for our company purposes and correct me if I'm wrong the all data in the database and the API doesn't have any tricky logic? I mean, if something happens to the API, we will be able to extract data from the DB directly just SQL right?
🙌 1
b
Yes
May I ask you if you considered our Infracost Cloud product? Can it bring you any value?
If no current value, what features would you like to see in there to consider purchasing?
p
So I'm mostly a technical part and I just touched CPAPI all the organizational issues and the subscription is probably to our mgmt, I'll write them a letter today with my thoughts and I hope that they will contact with you at least they have email hassan.hosseini@infracost.io As for other products, I don't know yet how they will work for us, maybe we will find value
b
Awesome, thank you! And thank you for your feedback! 🙂
p
Hey Vadim, quick question I tried to run this request 
curl <https://pricing.api.infracost.io/graphql> \
-X POST \
-H 'X-Api-Key: MY_KEY' \
-H 'Content-Type: application/json' \
--data '{"query": "{ products( filter: { vendorName: \"azure\", service: \"Bandwidth\", productFamily: \"Networking\", region: \"eastus\", attributeFilters: [ { key: \"fromLocation\", value: \"US East 2(Virginia)\" }, { key: \"toLocation\", value: \"West US 2 (Washington)\" } ] },) { attributes { key, value } prices { USD } } }"}'
I got {"error":"Invalid API key"}curl: (3) URL using bad/illegal format or missing URL I checked API key it looks ok
b
Hey! Just checked the same command with my key and it worked 🙂
I wonder what the missing URL error means
Can you try to make it a one-liner and retry?
Looks like curl can't detect the URL which makes me think that the api key header is also not provided correctly
p
yep something with formatting , one-line it works
b
But the error is returned, so this is strange: curl makes a request to API, gets the result, but reports an invalid URL format 🙂
Are you on mac or linux?
p
mac
b
Same 🤔
Iterm - zsh, copy-paste and replace MY_KEY - worked
p
you know this query
Copy code
curl <https://pricing.api.infracost.io/graphql> -X POST -H 'X-Api-Key: MY_KEY' -H 'Content-Type: application/json' --data '{"query": "{ products( filter: { vendorName: \"azure\", service: \"Bandwidth\", productFamily: \"Networking\", attributeFilters: [ { key: \"fromLocation\", value: \"US East 2(Virginia)\" }, { key: \"toLocation\", value: \"West US 2 (Washington)\" } ] },) { attributes { key, value } prices { USD } } }"}'
returned empty value {"data":{"products":[]}} maybe I need to add more attributeFilters ?
b
No products means that your current filters don't find anything
p
sorry, I'm a little confused about what I saw in azure https://azure.microsoft.com/en-us/pricing/details/bandwidth/ for bandwidth pricing
b
Try using 
value_regex:
instead of `value`: 
value
is the exact match, the 
value_regex
takes regular expression so you can do partial matches https://github.com/infracost/cloud-pricing-api/blob/master/src/typeDefs.ts#L40
The way the website shows the regions might not be identical to the values cloud vendors provide in their pricing APIs. This is even true between different services within same vendor 🙂
Have you found the product records in the CPAPI database with such values?
p
yep Bandwidth | Networking
b
Looking at the raw data is the only way to know what the actual values are
attributes
column holds the specifics
The values 
fromLocation
and 
toLocation
might be different that you provide
You can start without filters, just ensure that the Bandwith records are returned for your region. Then gradually narrow down the search introducing filters
p
Vadim for this request:
Copy code
curl <https://pricing.api.infracost.io/graphql> -X POST -H 'X-Api-Key: MY-KEY' -H 'Content-Type: application/json' --data '{"query": "{ products( filter: { vendorName: \"azure\", service: \"Bandwidth\", productFamily: \"Networking\", attributeFilters: [{key: \"meterName\", value: \"Standard Inter-Region Data Transfer\"}, {key: \"skuName\", value: \"Standard\"}] }) { attributes { key, value} prices { USD } } }"}'
I got this response
Copy code
{"data":{"products":[{"attributes":[{"key":"meterId","value":"e92d3650-2471-55d2-8433-d8cacd330f70"},{"key":"skuName","value":"Standard"},{"key":"meterName","value":"Standard Inter-Region Data Transfer"},{"key":"productId","value":"DZH318Z0BNVX"},{"key":"serviceId","value":"DZH318FHKQ5W"},{"key":"armSkuName","value":""},{"key":"productName","value":"Rtn Preference: MGN"},{"key":"serviceFamily","value":"Networking"},{"key":"effectiveStartDate","value":"2022-11-01T00:00:00Z"}],"prices":[{"USD":"0.012"}]}
what is this data and what region does it belong to?
b
This shows the product attributes and prices matched to your request. You would also need to ask GraphQL to return 
region
https://github.com/infracost/cloud-pricing-api/blob/master/src/typeDefs.ts#L26
This file specifies the GraphQL schema
Each item is basically a product in cloud vendor (SKU)
p
ah, ok thanks for the help
👍 1
Vadim, correct me if I'm wrong, there will be a charge after the free trial period, right?
if I use only CPAPI
b
No. CPAPI is currently free to use
👍 1
p
Hey Vadim, our DevOps prepared PR, could someone from your team check it out https://github.com/infracost/helm-charts/pull/30
b
Yep, thank you for the heads up!
@polite-animal-10151 Alistair merged the PR, however we're currently unable to release a new version due to some issues with dependencies. We'll get to it a bit later. In the meantime is it possible for you to use the master branch of the chart?
p
@abundant-terabyte-72655 Can we use master branch?
a
Hi, @busy-agent-35515 We are not cloning or using git approach, we add this as repository link to our helmfile. And this is using a semantic version (i suppose based on git tags).
b
Hey! I see. Are you able to use helm-git for that? Sorry, I've never used helmfile myself, so not much of a help here 😞 I'm afraid we won't get to the fix until next week, hopefully it's not a blocker for you
a
It is not a blocker (at least on development side), we can wait. Thank you.
👍 1
p
@busy-agent-35515 Maybe do you know approximately when can we expect a new version this week?
b
Hey, unfortunately, no estimate yet 😞
😞 1
@polite-animal-10151 @abundant-terabyte-72655 Hey! Sorry for waiting! We've released a new helm version! https://github.com/infracost/helm-charts/releases/tag/cloud-pricing-api-0.6.0
🙌 1
a
awesome, thanks.
🙌 1
p
Great thanks
🙌 1
a
Looks like CI/CD is happy with the new vision :))
✅ 1
🦜 2
p
@busy-agent-35515 Hey, maybe you know better who can help us with this issue? We got with deploy of the pod next error maybe we are doing something wrong: Warning Failed 5s (x6 over 63s) kubelet Error: couldn't find key postgresql-password in Secret manager/cloud-pricing-api-postgresql and however we have: kubectl get secret | grep postgres cloud-pricing-api-*postgres*ql Opaque 1 3m34s
b
@little-author-61621 Have you seen this issue before?
l
@polite-animal-10151 are you using an external postgresql or the postgresql that’s deployed with the chart?
p
with the chart
a
@polite-animal-10151 can you list the content of the env file here (no sensitive data)?
p
sure in our env file we have:
Copy code
postgresql:
  existingSecret: cloud-pricing-api-postgresql
  postgresPassword: xxxxxxx
  secretName: cloud-pricing-api-postgresql
l
What’s in the 
cloud-pricing-api-postgresql
secret?
a
Is the same value of 
postgresPassword
Copy code
{{- if and (not .Values.postgresql.enabled) (not .Values.postgresql.existingSecret) }}
  postgresql-password: {{ .Values.postgresql.external.password | b64enc | quote }}
  {{- end }}
Is this the right one? In this way it’s suppose to be?
and in the default file you have:
Copy code
postgresql:
  # -- Deploy PostgreSQL servers. See [below](#postgresql) for more details
  enabled: true
and in the chart, password for postgres always referring to 
key: postgresql-password
l
So when you deploy does the 
cloud-pricing-api-postgresql
secret have the correct value for the password?
I’ve wondering if you want to remove the 
(not .Values.postgresql.existingSecret)
part from the secret contents
a
Yes, we did last week and it was the right pass. Let’s us to do more tests. Thank you.
I can confirm, does not work as doc says. Also, I i edited 
postgresql-password
key to 
postgres-password
on secret side as it is creating and the pods are up, but is going to do 
restart
over and over. The error is 
Can not connect to the db...
Question, is your service going to create a user and pass for db? I saw you are using postgresql 11.x, and in doc I saw 
postgresql.postgresqlPassword
but in the postgresql chart, you need to specify 
postgresql.auth.postgresPassword
.
l
@abundant-terabyte-72655 hmm odd… we don’t create a password for the DB, so that’s a value you have to pass in. Have you tried passing in 
postgresql.auth.postgresPassword
when installing the helm chart?
Happy to Zoom chat with you to pair to get it working sometime if you’d like. You can use https://calendly.com/aiscott/30min to find a time.
@abundant-terabyte-72655 @polite-animal-10151 thanks for pairing the other day 🙏. I’ve updated the helm-chart and you should now just need to set 
postgresql.auth.password
. Let me know if that works for you.
p
@little-author-61621 Thank you very much for your help, we will check
a
@little-author-61621 i can see all pods are up and running, but on the logs:
Copy code
{"level":30,"time":1676291116738,"pid":18,"hostname":"cloud-pricing-api-77d59d858f-qjw2p","msg":":rocket:  Server ready at <http://0.0.0.0:4000/>"}
{"level":50,"time":1676291124437,"pid":18,"hostname":"cloud-pricing-api-77d59d858f-qjw2p","msg":"Could not connect to database: Error: connect ECONNREFUSED 10.7.39.139:5432"}
{"level":50,"time":1676291124442,"pid":18,"hostname":"cloud-pricing-api-77d59d858f-qjw2p","msg":"Could not connect to database: Error: connect ECONNREFUSED 10.7.39.139:5432"}
Is it a normal behaviour?
l
@abundant-terabyte-72655 is the DB pod up and running ok?
a
Copy code
cloud-pricing-api-77d59d858f-qjw2p   1/1     Running   0          4m42s
cloud-pricing-api-init-job-ktclb     1/1     Running   0          4m42s
cloud-pricing-api-postgresql-0       1/1     Running   0          4m42s
On Postgres i can see:
Copy code
2023-02-13 12:25:15.730 GMT [1] LOG:  pgaudit extension initialized
2023-02-13 12:25:15.740 GMT [1] LOG:  starting PostgreSQL 14.5 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-02-13 12:25:15.740 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-02-13 12:25:15.740 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2023-02-13 12:25:15.746 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2023-02-13 12:25:15.754 GMT [154] LOG:  database system was shut down at 2023-02-13 12:25:15 GMT
2023-02-13 12:25:15.765 GMT [1] LOG:  database system is ready to accept connections
2023-02-13 12:26:34.945 GMT [155] LOG:  checkpoints are occurring too frequently (17 seconds apart)
2023-02-13 12:26:34.945 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:26:51.101 GMT [155] LOG:  checkpoints are occurring too frequently (17 seconds apart)
2023-02-13 12:26:51.101 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:27:06.994 GMT [155] LOG:  checkpoints are occurring too frequently (15 seconds apart)
2023-02-13 12:27:06.994 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:27:23.373 GMT [155] LOG:  checkpoints are occurring too frequently (17 seconds apart)
2023-02-13 12:27:23.373 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:27:39.559 GMT [155] LOG:  checkpoints are occurring too frequently (16 seconds apart)
2023-02-13 12:27:39.559 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:27:55.455 GMT [155] LOG:  checkpoints are occurring too frequently (16 seconds apart)
2023-02-13 12:27:55.455 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:28:12.392 GMT [155] LOG:  checkpoints are occurring too frequently (17 seconds apart)
2023-02-13 12:28:12.392 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:28:29.696 GMT [155] LOG:  checkpoints are occurring too frequently (17 seconds apart)
2023-02-13 12:28:29.696 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:28:48.088 GMT [155] LOG:  checkpoints are occurring too frequently (19 seconds apart)
2023-02-13 12:28:48.088 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:29:09.263 GMT [155] LOG:  checkpoints are occurring too frequently (21 seconds apart)
2023-02-13 12:29:09.263 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:29:29.364 GMT [155] LOG:  checkpoints are occurring too frequently (20 seconds apart)
2023-02-13 12:29:29.364 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:29:48.452 GMT [155] LOG:  checkpoints are occurring too frequently (19 seconds apart)
2023-02-13 12:29:48.452 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:30:09.027 GMT [155] LOG:  checkpoints are occurring too frequently (21 seconds apart)
2023-02-13 12:30:09.027 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
2023-02-13 12:30:29.605 GMT [155] LOG:  checkpoints are occurring too frequently (20 seconds apart)
2023-02-13 12:30:29.605 GMT [155] HINT:  Consider increasing the configuration parameter "max_wal_size".
l
Is this after a fresh uninstall, removing the PVC and installing?
a
Actually i recereate the whole cluster. Let me do it again.
jus tot be sure.
Actually it is my fault. i used a different values file. Now looks like it works.
l
Awesome, thanks for trying it!
a
Is it possible to use a tag (versioning) on github for the helm chart?
l
Yeah. This is in the tag v0.6.1 now.
🙌 1
a
awesome (Now i saw it). Thank you a lot for the help.
🙌 1
p
Hey @busy-agent-35515 I need a little bit of your help, I have this request: curl https://pricing.api.infracost.io/graphql -X POST -H 'X-Api-Key: MY-KEY' -H 'Content-Type: application/json' --data '{"query": "{ products( filter: { vendorName: \"aws\", service: \"AmazonEC2\", region: \"us-east-1\",productFamily: \"Storage\", attributeFilters: [{key:\"regionCode\",value:\"us-east-1\"}, {key:\"servicename\", value:\"Amazon Elastic Compute Cloud\"}] }) { attributes { key, value} prices { USD } } }"}' and I got this response: {"data":{"products":[{"attributes":[{"key":"location","value":"US East (N. Virginia)"},{"key":"operation","value":""},{"key":"usagetype","value":"EBS:VolumeUsage"},{"key":"regionCode","value":"us-east-1"},{"key":"volumeType","value":"Magnetic"},{"key":"servicecode","value":"AmazonEC2"},{"key":"servicename","value":"Amazon Elastic Compute Cloud"},{"key":"locationType","value":"AWS Region"},{"key":"storageMedia","value":"HDD-backed"},{"key":"maxIopsvolume","value":"40 - 200"},{"key":"maxVolumeSize","value":"1 TiB"},{"key":"volumeApiName","value":"standard"},{"key":"maxThroughputvolume","value":"40 - 90 MB/sec"},{"key":"maxIopsBurstPerformance","value":"Hundreds"}],"prices":[{"USD":"0.0500000000"}]} but I can't see the instance type of EC2 because not specified in the response for example "t3a.xlarge" and I can't see it in DB as well, how can I find out what type of instance?maybe I'm doing something wrong
l
@polite-animal-10151 does it work if you set 
productFamily
to 
Compute Instance
?
✅ 1
p
@little-author-61621 Thanks for your help, I didn't notice a little that there are more productFamily for this type. Thank you
b
For the reference, you can search 
attributes
column as a string: 
select * from products where "vendorName" = 'aws' and region = 'us-east-1' and attributes::text ilike '%t3a.xlarge%' limit 1
will give you one record where you can see the details
p
ok, super thanks
👍 1
a
Hello, in case i want to 
rolling update or upgrade
for the cloud pricing api helm chart, how is the behaviour, will work? Right now, we have one issue when we do helm upgrade for our stack and we get this error.
Some invalid label for the job.
m
It seems like we might be missing these job fields 
controller-uid
and 
job-name
from the init job to support a rolling upgrade. Probably something we can add here: https://github.com/infracost/helm-charts/blob/master/charts/cloud-pricing-api/templates/job-init.yaml#L6-L7 . I’ll raise an issue in the helm chart repo to track and fix. That being said I don’t think we’ll have a fix for this today as the team is heads down on shipping some important features/goals. So we’ll most likely get to this some point next week.
a
Awesome. No problem. Is not an block for now (we are still in development), is just when we do upgrade.
m
ok great, as always if you find a fix or want an resolution sooner, feel free to open a PR with contributions infraheart
a
Sure. Thanks.
40 Views
Open in Slack
PreviousNext
Powered by