[MUSIC PLAYING]
WEI HSIA: Hi.
Thanks for joining us today to talk about Open Analytics
Lakehouse on Google Cloud.
So why build a lakehouse?
Well, it's very likely you, like most companies,
have had a growing pool of data and been collecting it
into a pool or bucket of some sort.
The first place you move your data to from this landing zone
is probably your data lake.
You can start to explore the data there,
but as you develop a specific data usage needs,
there are parts of the data that you segment off.
One specific use case has likely led
you to create a data warehouse so
that you can have easy access to your curated well-formed data.
This lets you understand your data at a deeper level.
At this point, you've created a data architecture
to address your business needs, but that architecture
has made data silos.
It's usually difficult to put your unstructured data
into a data warehouse, and at the same time,
analytic capabilities over your structured data
are limited when it stays into your landing zone
or in a data lake.
Your data lake becomes larger, and although you
can find what you need eventually,
it's not nearly as efficient.
Beyond data silos, there are additional challenges
with trying to use this sort of architecture effectively.
Firstly, your data team will need
to have both broad and deep expertise across the varying
technologies.
It can be difficult to find the right talent
and may lose operational efficiencies when
roles are deeply specialized.
Additionally, your tooling set will
need to work across your various data stores,
for example, tools that integrate
the data into analytic workloads or your governance tools.
So which services enable building
your lakehouse architecture on Google Cloud?
BigQuery and Google Cloud Storage
provide a flexible layer to store
structured, semi-structured, and unstructured data.
BigLake provides a consolidated API interface
to interact with your data, no matter where it's stored,
and also integrates with solutions
across Google Cloud and open source
to handle a diverse set of analytic workloads
with the tools you prefer.
In addition, it provides a uniform governance
and management you need in the architecture.
At the heart of all of this is BigLake, a storage engine
enabling BigQuery and your other processing engines
to use the data in an efficient way.
For example, you can use Apache Iceberg format
and integrate it with BigLake and use BigQuery, Dataproc,
or Dataflow to interact and work with the data.
SQL is powerful, but there are times
where you still need to use additional languages
or libraries.
In this case, you still have many options available.
You have your external user-defined functions,
or UDFs for short, that you can link between a Cloud Function
or Cloud Run and BigQuery.
Using the interface of BigQuery, you and your fellow analysts
can use the functions without needing
to move data around or look for expertise to run the code.
There's also tooling to help with data governance.
Dataplex helps you govern and manage
your lakehouse uniformly.
It has automatic data quality checks
and, whenever possible, provides options
to remediate the problems.
As an open platform, Google Cloud
supports your choice of tooling for data processing.
You can use a flavor of Dataproc for a fully managed Apache
Spark experience.
You can also use Dataflow for a fully managed Beam experience.
You have access to open source connectors
to use tools such as Trino and Hive.
You can also use Google Cloud's Vertex AI
service to accelerate access to ML training and prediction
tools.
Ready to try it out now?
Try out the interactive tutorials
with the resources linked here and in the description.
Thank you for watching.
[MUSIC PLAYING]