@plain-potato-53219 thanks for taking on the GCP custom instances issue. I’m going to post in github for better visibility, but feel free to ping me here if that works better.

Thanks, Tim! I’m looking forward to understand what I should code in order to turn such feature available. Do we already have means of querying for custom instances on cloud pricing API?

Ok, I just outline the steps in the GH issue so let me know if it makes sense.

Do we already have means of querying for custom instances on cloud pricing API?

Yes it looks like prices for custom instances are already in the cloud pricing DB. This guide has a good example of how to do the search and map them to a cost component

I managed to query something like:

query {
  products(
    filter: {
      vendorName: "gcp",
      service: "Compute Engine",
      productFamily: "Compute",
      region: "europe-central2",
      attributeFilters: [
        { key: "resourceGroup", value: "RAM" }
      ]
    },
  ) {
    attributes { key, value }
    prices { USD }
  }
}

But I'm not sure how I could improve the result accuracy in this query 🤔 I found in the results some descriptions like

Custom Instance Ram running in Warsaw

and I think these should be our case for an object like:

resource "google_container_node_pool" "dev-1" {
  cluster  = "dev"
  name     = "dev-1"
  location = "europe-central2-c"

  autoscaling {
    max_node_count = 20
    min_node_count = 1
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }

  upgrade_settings {
    max_surge       = 1
    max_unavailable = 0
  }

  node_config {
    preemptible = true
    oauth_scopes = [
      "<https://www.googleapis.com/auth/cloud-platform>"
    ]

    machine_type = "custom-22-20480"

    image_type   = "COS_CONTAINERD"
    disk_type    = "pd-standard"
    disk_size_gb = "100"

    labels = {
      "pool" = "dev-1"
    }

    workload_metadata_config {
      mode = "GKE_METADATA"
    }
  }
}

Cool I’ll check it out 🚀

I separated the cost components for CPU and RAM since they require different queries on cloud-pricing-api. The weird thing is that I am succeeding on retrieving both CPU and RAM through GraphQL

query {
  products(
    filter: {
      vendorName: "gcp",
      service: "Compute Engine",
      productFamily: "Compute",
      region: "europe-central2",
      attributeFilters: [
				{ key: "description", value_regex: "/^Custom Instance Core.*$/" },
      ]
    },
  ) {
    attributes { key, value }
    prices { USD }
  }
}
---
query {
  products(
    filter: {
      vendorName: "gcp",
      service: "Compute Engine",
      productFamily: "Compute",
      region: "europe-central2",
      attributeFilters: [
				{ key: "description", value_regex: "/^Custom Instance Ram.*$/" },
      ]
    },
  ) {
    attributes { key, value }
    prices { USD }
  }
}

But not on the final pricing

google_container_node_pool.dev-1                                                                          
 ├─ Standard provisioned storage (pd-standard)                                    300  GB           $14.40 
 ├─ Custom Instance CPU usage (Linux/UNIX, preemptible, custom-20-20480)        2,190  hours         $0.00 
 └─ Custom Instance RAM usage (Linux/UNIX, preemptible, custom-20-20480)        2,190  hours         $0.00

Another issue that I already spotted in my code is that I didn't multiplied the Core/Ram cost by the unities specified in the instance name (i.e 20 cpu and 20480 ram). And I'm not sure how I'll make this right now. I'll get back to this issue some other time in the near future.

But not on the final pricing

Looks like you have

ProductFamily: strPtr("Compute Instance"),

in the code, not

ProductFamily: strPtr("Compute"),

Hey everyone, it took a while but I finally got the code right for querying costs for GCP custom instances. Whenever possible, I'd appreciate a code review here: https://github.com/infracost/infracost/pull/2004 Looking forward to let that code in the best shape for Infracost standards 😃 @crooked-daybreak-55253 @little-author-61621

Amazing! Thanks @plain-potato-53219! We will have a look this week.

Now I just realized that I only did that for GKE node pools.. I need to extend the custom handling to regular VMs as well. Will do it asap.

awesome, hopefully it won’t be too much extra

I think it would only take a verification like the following here I made a quick test while I was waiting a CI check to run 😬 but weirdly Infracost skipped the google_compute_instance I added. I’ll understand what is missing later.

It probably skipped because of this check: https://github.com/infracost/infracost/blob/master/internal/providers/terraform/google/compute_instance.go#L29-L31

oh thanks for that! I guess we might safely remove this verification as we’ll start handling custom instances, right?

Yep

I noticed that the node pool is 3x times bigger than the actual instance.

google_compute_instance.vm-dev-1
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, custom-20-20480)                 730  hours               $583.90
 └─ Custom Instance RAM usage (Linux/UNIX, on-demand, custom-20-20480)                 730  gibibyte hour       $170.97

 google_compute_instance.vm-dev-2
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, custom-20-20480-ext)             730  hours               $583.90
 ├─ Custom Instance RAM usage (Linux/UNIX, on-demand, custom-20-20480-ext)             730  gibibyte hour       $170.97
 └─ Standard provisioned storage (pd-standard)                                          10  GB                    $0.48

 google_compute_instance.vm-dev-3
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, n2-custom-20-20480)              730  hours               $434.58
 └─ Custom Instance RAM usage (Linux/UNIX, on-demand, n2-custom-20-20480)              730  gibibyte hour        $63.69

 google_compute_instance.vm-dev-4
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, n2-custom-20-20480-ext)          730  hours               $434.58
 ├─ Custom Instance RAM usage (Linux/UNIX, on-demand, n2-custom-20-20480-ext)          730  gibibyte hour       $119.00
 └─ Standard provisioned storage (pd-standard)                                          10  GB                    $0.48

 google_container_node_pool.dev-1
 ├─ Standard provisioned storage (pd-standard)                                         300  GB                   $14.40
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, custom-20-20480)               2,190  hours             $1,751.69
 └─ Custom Instance RAM usage (Linux/UNIX, on-demand, custom-20-20480)               2,190  gibibyte hour       $512.90

 google_container_node_pool.dev-2
 ├─ Standard provisioned storage (pd-standard)                                         300  GB                   $14.40
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, custom-20-20480-ext)           2,190  hours             $1,751.69
 └─ Custom Instance RAM usage (Linux/UNIX, on-demand, custom-20-20480-ext)           2,190  gibibyte hour       $512.90

 google_container_node_pool.dev-3
 ├─ Standard provisioned storage (pd-standard)                                         300  GB                   $14.40
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, n2-custom-20-20480)            2,190  hours             $1,303.73
 └─ Custom Instance RAM usage (Linux/UNIX, on-demand, n2-custom-20-20480)            2,190  gibibyte hour       $191.07

 google_container_node_pool.dev-4
 ├─ Standard provisioned storage (pd-standard)                                         300  GB                   $14.40
 ├─ Custom Instance CPU usage (Linux/UNIX, on-demand, n2-custom-20-20480-ext)        2,190  hours             $1,303.73
 └─ Custom Instance RAM usage (Linux/UNIX, on-demand, n2-custom-20-20480-ext)        2,190  gibibyte hour       $356.99

A quick troubleshoot lead me to this https://github.com/7onn/infracost/blob/master/internal/providers/terraform/google/container_node_pool.go#L64-L74 In fact, there are three possible zones.. but this doesn't mean that three nodes will be provisioned. If the autoscaling.min_count is 1 and the cluster is cool, we'll have a single machine in some of the AZs of the selected region and not three 🤔 I believe this is wrongly multiplying the resource consumption for GKE node pools 🤔 Please, correct me If I'm wrong.

As far as I remember it differs depending on what the

location

is set to. If it’s set to a region (us-central1) then it creates 3 nodes. If it’s set to a AZ (us-cental1-a) then it creates 1.

Makes sense. But still, I am passing the zone

europe-central2-c

. Therefore 1 node would be provisioned, right? I'll see how I can handle this and avoid multiplying this result by 3. But don't worry. I won't end up 🪒🐂 shaving the yak hahaha

what does the terraform look like that you are testing?

All node pools are set to a zone and not a region.

😛artyparrot:

@little-author-61621 I addressed all of your suggestions in the PR 😃 Looking forward for the next steps.

Thanks! Having a look 😒imple_smile: