LEAH JARRETT: Hi, my name is Leah Jarrett,
and today I'm going to be talking to you about how
a retail company might create an end-to-end data
architecture with Google Cloud.
Meet Cymbal Direct, our fictitious apparel retailer,
where different brands sell clothing items
both in their stores and on their website.
Cymbal is looking to leverage Google
Cloud in three key areas.
First, they're exploring new solutions that can
power highly personalized and dynamic product
recommendations.
Next, they're building a portal to surface analytics back
to sellers on their site as a brand new revenue stream.
And finally, they're also looking
to use data to improve digital marketing efforts so
that they're increasing their return on investment
for search campaigns.
So let's dive in and see how they're
using Google Cloud to support these initiatives.
First off, they've built their e-commerce site on App Engine,
and they use Spanner as their database
for storing information related to customers and transactions
on their website.
They also have unstructured and semi-structured data,
like product reviews and images, stored
in Cloud Storage buckets.
They're using BigQuery as their analytical processing Engine,
and they're able to easily access the information stored
in both Spanner and Cloud Storage
through federated queries.
Now, to track interactions on their site,
they use Tag Manager to push events
to Pub/Sub, which are then processed and streamed
into BigQuery in real time, using Dataflow and BigQuery's
streaming API.
Outside of Google Cloud, they have a few other data sources
that are imperative to include in these analytics initiatives.
First off, they have inventory and distribution management
information in an Amazon S3 bucket.
Lucky for them, BigQuery supports multi-cloud workflows,
either using BigQuery Omni to work with data stored
outside of GCP or a data transfer service
to replicate data into BigQuery's native storage.
And finally, Cymbal also uses BigQuery data transfer
to integrate data from SearchAds360,
and they use Dataprep to wrangle and merge this advertising data
with relevant sales information.
With processed and prepared data centralized in their BigQuery
data warehouse, they make use of built-in machine learning
and intelligent caching capabilities
to deliver insightful reports using Looker and model
endpoints to recommend products using Vertex AI.
This information is then surfaced back to key players
through App Engine.
Now with data stored and accessed in different places,
Cymbal leverages Dataplex for unified data and access
management plus integrations with Data Catalog
for asset discovery and metadata tracking.
Each part of their data ecosystem
helps them deliver a reliable and scalable platform that
delivers real business value.
So now let's actually see all of these components in action.
As we mentioned, Spanner is their transactional database,
where they can manage order items and users.
They chose Spanner because it's the only database of its kind
to offer reliability at a global scale.
The separation of storage and compute
allows it to scale up to meet transactional workloads
and customer demands.
They're also using BigQuery for all analytical workloads.
Lucky for us, we're able to create a direct connection
from BigQuery back to Spanner.
Then we can run federated queries
to fetch data from Spanner in real time.
Here, we're creating a Federated query
and then joining the results back
on to an existing BigQuery table, the DMA table.
Then we're using BigQuery's built-in geospatial
functionality to map each one of our orders back to a DMA
and finally join it onto Google's proprietary Google
Trends dataset, where we can calculate the top three Google
Search terms for each area where we had the highest total sales.
Aside from creating a query to just analyze our data,
we can also use external queries to create ETL pipelines using
nothing but SQL.
Here, I'm fetching the data from Spanner
and then transforming that data into a denormalized table
that's optimized for query processing.
Finally, we're taking advantage of BigQuery's ability
to handle semi-structured data in record format or even
repeated data.
Like we showed in the architecture diagram,
Spanner data isn't the only asset the e-Commerce department
needs to analyze.
They also have Cloud Storage buckets containing
product reviews and images.
In order to centrally manage each of these data entities,
the team has created a single e-commerce data lake
inside of Dataplex.
Here they can control access to the entire lake
from a single pane, for example, granting view access
on all e-commerce data to our e-commerce Google group.
And this data lake goes beyond just managing permissions.
When new objects get added to the storage buckets,
for example, a new file containing the latest product
reviews, it becomes available as an external table
inside of BigQuery.
These assets will also be available in Dataproc Metastore
for Spark workflows and Data Catalog.
As soon as technical assets are created,
they're automatically discoverable in Data Catalog.
Here, I'm able to see this table and drill
into all the related technical metadata
and even create schematized tags that
can be attached to certain columns, for example
the email column we've marked as containing PII.
With this information in place, we
can create policy tags that guard column level
access to sensitive data.
We can also use DLP to scan data for more sensitive information
or even de-identify our data, replacing PII with an asterisk
or pound sign.
Next up on Cymbal's architecture is bringing click events
on their website into BigQuery.
With pre-built Pub/Sub to Dataflow templates,
they're able to rapidly develop a pipeline that transforms
and appends new clicks into a table,
taking full advantage of BigQuery's novel ability
to insert new rows in real time.
At this point, we've covered a lot of data integration
across Google, but what about the data that's
stored outside of Google Cloud, like in an Amazon S3 bucket?
We can choose to keep the data stored here
by leveraging BigQuery Omni to run queries directly in AWS
or Azure for that matter.
Alternatively, we can leverage BigQuery's unique data transfer
capability to actually bring data in from Amazon S3
into a native BigQuery storage table.
And that's not the only data source this service supports.
Cymbal can also use it to integrate data from other SaaS
applications, like SearchAds360 for their marketing analytics
use case.
Pushing information into BigQuery
is an important task for Cymbal's data team.
However, they still need to do some additional transformations
and preparations for advanced analytics,
specifically to clean, aggregate, and join
their search data with sales to effectively measure
campaign impact.
This is where Dataprep comes into play.
With Dataprep, the team is able to build processes
to wrangle datasets, all in a simple interface that
allows them to explore the data at each step.
And with that, all of Cymbal's data
is centralized into BigQuery, and the team
is ready to unlock new insights.
With all this information stored in one place,
we're able to create a BigQuery machine learning model
to cluster our customers into specific groups.
We can then use the ML predict function
to cluster incoming customers and figure out
what group they belong in so we can better recommend them
new products.
Finally, we can look at the training and evaluation
heuristics of our model or even export it directly
into Google Cloud Storage.
Once it's in Google Cloud Storage,
we can import it directly into Vertex AI,
deploy it as a model endpoint, and we can then
make real-time predictions directly from our application.
Finally, we've created some embedded dashboards, where
queries are running from Looker back to BigQuery
to fetch data for sellers, so they
can see how their products are performing
on the e-commerce site.
Just to wrap things up, Cymbal chose Google Cloud
for a few different reasons.
First off, scalable governed data management,
where they're able to get fine-grained access
control plus robust metadata management and deidentification
services.
They also have access to data where and when they need it,
with real-time access to information
from a lot of different sources, including external systems
and data stored in other clouds.
Finally, BigQuery allows you to handle all different types
of data, from unstructured information to geospatial data,
and you can join first-party datasets
with proprietary Google owned datasets
and train machine learning models without ever
leaving the console.