MARK MIRCHANDANI: Monitoring, analyzing,
and optimizing costs have become an important part of managing
your cloud environment.
But it can be challenging to work with all
your cost and usage data.
On this episode of "Beyond Your Bill,"
we'll look at exporting your billing data to BigQuery
and how to create custom dashboards.
[MUSIC PLAYING]
Our last video covered how to explore your GCP costs
through the billing reports that you can easily access
through the Cloud Console.
Now let's look at exporting it to BigQuery,
and then create a dashboard using Data Studio to further
explore it.
It's important to note that these actions can only
be performed by a billing administrator.
As we mentioned back in the resource organization video,
you'll want to enable BigQuery export for your billing account
as soon as possible.
Ideally, you do this as soon as you create it.
That's because the export will start collecting data
as soon as you enable it, and you
can't get detailed billing data from before it's been enabled.
With that being said, let me show you how to enable it.
In the Cloud Console, click on Billing, and choose the billing
account you want to work with.
You may have to click on Manage Billing Accounts
first, then click on Billing Export on the left.
Here's where you can see if the billing exports for BigQuery,
or file exports, are enabled.
While there is an option to do a CSV or JSON export under file
export, we don't actually recommend doing this,
because it doesn't contain as much information
as the BigQuery export.
The more information you have, the more analysis
you'll be able to do.
So go ahead and click on Edit Settings to choose a project,
and then choose a BigQuery dataset.
If you don't have any projects, you'll
need to create one with BigQuery enabled,
then create a BigQuery dataset.
Creating a dataset is as easy as going into the BigQuery
interface, clicking on your project,
and clicking on the Create Dataset button.
If you already have a bunch of projects,
feel free to make a new one just for this export or all
of the other cost management efforts that you're working on.
Once you know which project and dataset
you want to use, choose them in the export settings
and hit Save.
You should see a screen that looks
like this, which means that you've successfully
enabled the export process.
You can click the dataset name to jump into BigQuery,
but you probably won't see anything yet.
It may take a few hours to start seeing data,
since different services have different update frequencies.
Also worth noting is that you may incur some nominal BigQuery
costs for storing this data.
You can quickly calculate any costs
by looking at the BigQuery pricing page
and seeing how much data you're generating
every day with the export.
You can also set a budget specific to BigQuery
if you want to keep closer track of your spending.
We'll cover this in more detail in a later video.
Once the data starts showing up, you can run queries against it
through the BigQuery UI.
Here's a sample query that shows the amount spent
on each service by month since BigQuery export was enabled.
This is a simple place to start, but there
are lots of columns you can work with, like Location, Project,
and Labels.
From here, you can build and run queries
as you see fit to analyze your data
or even start making predictions.
Now, let's talk about another way
to analyze your billing data, creating a custom dashboard
with Data Studio.
Data Studio is an interactive analytics tool built into GCP.
And it can integrate easily with BigQuery to pull data and then
let you visualize it.
The quickest way to set up a dashboard
is to start with the sample billing report
that we've put together for you as a starting point.
You can do this by visiting the Sample Billing Report
and clicking the How to Copy tab.
By following the steps outlined on that page,
you'll set up some special queries
and connect them to your own copy of the billing
report in Data Studio.
This sample report is ready to go, with charts and filters
to help you visualize your data.
On the Resources page, you can filter by label.
So you'll easily be able to see how your costs are broken down.
For example, if you chose to label your resources
by environment, you can see how much
you're spending on production versus development
environments.
This is another reason it's important to choose
naming conventions that are clear and easily understood
by everyone.
You're also able to build your own reports
by using Data Studio.
So I'll quickly show you how to get
started with a basic report.
First, head to the Data Studio page
and choose a blank report to start from scratch.
Once it's been created, look for the option
to Create New Data Source, and then choose BigQuery.
From here, you just need to select the same project
and dataset as well as the table that BigQuery created.
After choosing it, click Connect, and then click
Add to report.
Once the data source is added to the report,
you'll be able to reference it with different graphs
and filters.
As an example, you can insert a new pie
chart that shows how much you've spent on each service.
Click on Insert, and then Pie Chart to add it to the report,
then click on it to customize it.
By changing the Dimension to service.description and then
the Metric to Cost, you can quickly
see how your costs are split between different services.
Different graphs have different fields.
And you can create interactive elements
to dynamically update them.
Exporting your data to BigQuery lets you analyze your cost data
using BigQuery and SQL, as well as setting you up
to visualize your costs with Data Studio or other data
visualization tools.
In the next videos, we'll look at advanced cost control tools,
like budgets and quotas.
[MUSIC PLAYING]