SPEAKER: When approaching the architecture of your data
warehouse, it's important to understand the constructs
of the services you'll use.
So what are folders, projects, and data sets,
and how do they come together to support
warehousing fundamentals like security and cost management?
On today's episode of "BigQuery Spotlight,"
we're talking about the BigQuery resource model
and giving you our top tips for structuring your own resources.
[MUSIC PLAYING]
Let's get started by talking about how
BigQuery fits into the Google Cloud resource hierarchy.
An organization is the root node of the Google Cloud resource
model.
It represents a company, probably the one you work for,
and it's linked to a domain through a Google Workspace
or Cloud Identity account.
An organization resource isn't required
to get started with BigQuery, but it is recommended.
With an organization, resources like BigQuery data sets
or tables can be centrally controlled
by organization administrators, instead
of solely being owned by the employee that creates them.
Within an organization, you create billing accounts
that specify who actually pays for Google Cloud resources.
You can also create folders, which
offer an additional grouping mechanism for resources.
Folders often represent business units or legal entities
within your company and make it easier to organize and monitor
resource utilization.
Folders also act as a policy inheritance point.
Access rules granted on a folder are automatically
inherited by all projects and folders
included in that folder.
Now, let's go a little bit deeper.
Within an organization or a folder,
you can create projects, which form
the basis for using BigQuery.
Projects are tied to a single billing account, which
means that if different teams need to be billed individually
for their BigQuery usage, they would need their own projects,
and additional billing accounts would
need to be created at the organization level.
Let's take a look at how this resource hierarchy is reflected
in the Cloud Console, where you interact with
and analyze your data in BigQuery.
In BigQuery, projects are kind of like databases
in other systems.
But here, you can have separate projects
for storing and analyzing your data.
When opening up the console, you can see your projects
on the left-hand side, with the data stored nested
beneath each project name.
All storage costs will be billed to the associated projects
here.
Even with data stored across different projects,
you can analyze it together by joining tables
in a single query.
At the top of the console, you can see the project where
your query will actually run.
All compute costs will be billed to this project
and its associated billing account.
Speaking of billing, BigQuery offers two different pricing
models.
Projects with on-demand pricing are
charged for the number of bytes processed by each query.
But with this model, there are per-project limitations
and quotas that are important to consider.
The other option is flat-rate pricing,
where customers purchase compute capacity to handle workloads.
With this model, reservations can
be assigned at the organization, folder, or project levels
to handle jobs within that resource.
On-demand pricing is great for getting started with BigQuery,
but as your consumption grows, flat-rate offers
scalable compute with predictable pricing.
Great-- so now back to the resource model.
So far, we've discussed resources
applicable to all Google Cloud deployments,
but now, we'll focus on resources specific to BigQuery.
Within a project, data sets are used
to house, organize, and control access
to tables, views, routines, and machine-learning models.
They're similar to schemas in other systems.
When you create a data set, you choose a geographic location
where your data will be stored.
BigQuery offers a variety of regions and multi-regions
for your data residency requirements.
However, you can't query across data
that lives in different locations,
so make sure to plan accordingly.
Data sets also store metadata.
You can create descriptions and labels,
which can be viewed in the console
or used as tabs for monitoring.
You can also find details about each one of your BigQuery
resources by querying the information schema.
But data sets aren't the only BigQuery resource
belonging to projects.
Other concepts, like connections and jobs,
can be found here as well.
We'll cover more detail on these in the next few videos.
Now that you understand the top-level containers
of the BigQuery resource model, let's talk
about some common ways users structure their deployments.
First off, some BigQuery architects
choose to create a common storage project that is queried
in other compute projects.
In this case, a centralized data platform team
can also create pipelines in a storage project
to actually ingest data from different sources.
You can think of this storage project as a data lake
or a storage repository that holds a bunch of raw data.
Then, each department can have their own project
for querying the data, saving results, and creating
aggregate views.
You can think of these projects as data marts,
or a repository of summarized data
collected for analysis on a unit within the organization,
for example, the sales department.
This structure is especially useful
when departments want to be individually billed
for their queries, or if queries need
to be run in separate projects to avoid
per project limitations.
Another option is to do the reverse, where
each department creates and manages their own storage
project to house their raw data.
Aggregate tables or transformations of the raw data
are then stored in a centralized data warehouse
project where analysts can read and query the data.
This structure allows for project-level access controls
against raw data and offers a simple data access layer
for connecting business intelligence tools to BigQuery.
Many BigQuery administrators use some combination
of these two structures.
When deciding on your own architecture,
you should be asking questions like,
how often will analysts be running queries
against these data sets?
How should storage and compute be billed
to our different departments?
And who should have access to what data?
We hope this overview gets you well on your way to designing
your BigQuery architecture.
For more guidance on getting started
with BigQuery or the BigQuery resource model,
check out the links below.
Thanks for joining us today for "BigQuery Spotlight."
Watch out for the next episode, and remember, stay curious.
[MUSIC PLAYING]