Billing Reports and Cost Tables

Google Cloud provides several tools to help you understand and manage your cloud spending. The primary interface is the Google Cloud Console Billing page, which offers a graphical overview of your costs. However, for deeper analysis, you need cost tables—specifically, the BigQuery billing export feature that streams detailed usage and cost data into a BigQuery dataset. This allows you to run custom SQL queries, create dashboards, and perform advanced analytics.

Cloud costs can spiral without visibility. Billing reports give you a high-level view, but they are insufficient for granular cost allocation, chargebacks, or trend analysis. Cost tables solve this by providing raw data at the resource level, enabling you to:

When you use Google Cloud services, usage data is collected by the Cloud Billing system. This system aggregates usage metrics (e.g., compute hours, storage GB-months, network egress bytes) and applies pricing rules from the Cloud Billing pricing catalog. The resulting cost data is stored in the Cloud Billing account and made available via: - Cloud Console Billing reports: Pre-built charts and tables refreshed every few hours. - BigQuery billing export: A continuous stream of detailed cost data into a BigQuery table, updated daily.

The BigQuery export includes a table named gcp_billing_export_v1_<billing_account_id> (or a custom name you choose) with over 40 columns, including: - invoice.month: The billing month. - service.description: The GCP service (e.g., Compute Engine). - sku.description: The specific SKU (e.g., N1 Predefined Instance Core). - project.id: The project that used the resource. - labels: A repeated field with key-value pairs. - cost: The cost in USD (or your billing currency). - usage.amount: The amount of usage (e.g., hours). - credits: Any applicable credits (e.g., sustained use discounts).

Billing Account Types: - Self-serve (online): Pay-as-you-go with automatic card charges. No invoice PDF. - Invoiced (offline): Monthly invoicing with purchase order. Invoice PDF available.

Budget Alerts: - You create a budget at the billing account or project level. - You can set amount (a fixed dollar amount) or last month's spend as a reference. - You define alert threshold rules with percentages (e.g., 50%, 90%, 100%). - You can also set forecasted threshold alerts (e.g., 100% of forecast). - Alerts are sent via email to Billing Administrators and Budget Administrators by default, but you can add Pub/Sub topics for programmatic notifications. - Alerts are triggered once when the threshold is crossed, and then again if the spend drops below and re-crosses the threshold.

BigQuery Export: - You must enable the export in the Cloud Console under Billing > Cost Management > Cost Table. - The export creates a BigQuery dataset and a table that is updated daily (usually by 10 AM Pacific Time for the previous day's data). - Data is retained indefinitely (no automatic deletion). - You can query the table using standard SQL. - The table is partitioned by day on the _PARTITIONTIME pseudo-column for efficient queries.

Labels: - Labels are key-value pairs attached to resources (e.g., environment:production, team:engineering). - They are not retroactive; you must apply them before usage to see them in cost tables. - Labels appear in the labels column as a repeated record with key and value fields.

Enable BigQuery Billing Export (via Console): 1. Go to Billing > Cost Management > Cost Table. 2. Click Enable BigQuery Export. 3. Select an existing BigQuery dataset or create a new one. 4. Optionally, set a custom table name. 5. Click Save.

Verify Export with `bq` command:

bq ls --dataset <dataset_name>

You should see a table named gcp_billing_export_v1_<billing_account_id>.

Query Example:

SELECT
  project.id,
  service.description,
  SUM(cost) AS total_cost
FROM
  `project.dataset.gcp_billing_export_v1_*`
WHERE
  _PARTITIONTIME >= TIMESTAMP('2023-01-01')
  AND _PARTITIONTIME < TIMESTAMP('2023-02-01')
GROUP BY
  project.id, service.description
ORDER BY
  total_cost DESC

Create a Budget (via Console): 1. Go to Billing > Budgets & alerts. 2. Click Create budget. 3. Set scope (billing account or a specific project). 4. Set amount (e.g., $1,000). 5. Set alert thresholds (e.g., 50%, 90%, 100%). 6. Add notification channels (email, Pub/Sub). 7. Click Create.

Verify Budget with `gcloud`:

gcloud billing budgets list --billing-account=XXXXXX-XXXXXX-XXXXXX

A large enterprise runs 50 projects across three departments: Engineering, Marketing, and Sales. Each department has its own budget. The finance team needs a monthly report showing costs per department. The solution: apply a label department to every resource (e.g., department:engineering). Enable BigQuery billing export. Write a SQL query that groups by labels.key and labels.value to get costs per department. Set up a scheduled query that exports the results to a Google Sheet via Apps Script. The challenge: ensuring all resources are labeled consistently. Use Organization Policy to enforce label requirements. Common pitfall: resources created without labels before the policy was enforced will not have labels in historical data. The finance team must backfill labels manually or accept incomplete data.

A SaaS company runs its production environment in a single project. The monthly budget is $10,000. They want to be alerted when spending reaches 80% and 100% of the budget. They create a budget at the project level with thresholds at 80% and 100%. They add a Pub/Sub topic that triggers a Cloud Function to send a Slack message. They also set a forecasted alert at 100% to catch potential overruns. The budget alert works well, but one month, due to a DDoS attack, network egress costs spike, causing the 80% threshold to be crossed within a day. The alert fires once. The team misses the alert because it was a single email. They later add a second threshold at 90% to get more granularity. The exam may test that alerts are not real-time and only fire once per threshold.

A media company uses Compute Engine with committed use discounts (CUDs). They want to analyze whether CUDs are actually saving money. They export billing data to BigQuery and query cost with and without credits. They compare the cost column (which includes CUD credits) against the cost before credits. They discover that some CUDs are underutilized. They create a dashboard in Data Studio showing monthly cost trends, CUD utilization, and recommendations. The key performance consideration: querying large tables with many partitions can be expensive in BigQuery. They use clustering on service.description and project.id to reduce query costs.