[MUSIC PLAYING]
LUCIA SUBATIN: Hi there.
I'm Lucia Subatin, technical lead for Google Cloud Cortex
framework, and today, I'd like to deep dive
into the technical details of our Cortex Data Foundation
for SAP.
From previous videos, you know that Google Cloud Cortex
framework delivers not just the reference architecture
to solve business problems, but we also
deliver solution content, templates,
and deployment accelerators.
We provide not just the recipe, but we also
include the ingredients and some pro tips
to make your meal as tasty as possible.
Let's first take a look at the ingredients for the Cortex Data
Foundation for SAP.
We start with raw table data from an SAP, CC, or S/4HANA
system.
We want to learn the data into BigQuery,
and we want to make sure the data is
both consistent and updated as frequently
as our business requires.
The records keep changing as updates, deletes,
and inserts flow into BigQuery.
This means we need an automated way
to process these records that are constantly changing,
such as scripts to process Change Data Capture, or CDC
for short.
Finally, we need to make sure our users can
make sense of the current data, answer business questions,
and improve decisions with insights
powered by machine learning.
We also want to give users the flexibility to consume insights
with their preferred reporting tool of choice, an application
or even a spreadsheet.
So how does Google Cloud Cortex framework
help you put these ingredients together?
Let's take a deeper look.
We want to keep the integrations options open,
so any tool that gets the data into the same structure
with the same field names as in the original SAP tables work.
If you follow an ELT pattern, as is preferable with BigQuery,
you need a way to distinguish which
is the latest replicated record for reporting purposes.
This is accomplished by loading each record with a timestamp
and an operation flag.
Once the raw data records are in BigQuery,
you have two options to process the data,
leveraging the provided CDC process in scripts.
[MUSIC PLAYING]
The first one-- you can generate views to scan the raw table
and pick the latest record in real time.
This is useful when you're replicating
data in near real time from a tool like our BigQuery
connector for SAP SLT and the business
wants updates reflected as they happen.
The second option is to use scheduled jobs in Cloud
Composer to merge or upsert the new or updated record
in a table in a specific frequency in batch.
[MUSIC PLAYING]
This is a good option to keep costs under control
when the business does not plan on querying
the data in real time.
[MUSIC PLAYING]
Some replication tools have embedded change data
capture processing capability and merge the records directly
when inserting them.
In this case, you can skip the CDC processing in BigQuery,
but we supply the scripts just in case.
Now that we have processed our replicated data
and have the latest changes in BigQuery,
it's time to create some models.
Our Cortex Data Foundation has more than 60 pre-built views
that anyone can deploy and adjust to their needs,
and the list of modeling objects available
will continue to grow as we release new content.
You will find the current list of use
on the tables they're referencing
in the entity relationship diagram in the GitHub
repository link below.
These views not only incorporate SAP specific business modeling,
but they also tackle common but technically hard
challenges to solve, such as unit of measure and currency
conversions, and the ability to deploy SCC and S/4 models
separately are consolidated into a single view.
You will also find scripts to process data in batches,
flatten hierarchies, and consume and store
data from other sources, such as weather or search trends,
through directed acyclic graphs.
[MUSIC PLAYING]
These scripts are ready to be executed by Cloud Composer
within a schedule you can customize according
to your business needs.
Everything you need to know about the deployment
of our Cortex Data Foundation is documented in our open source
GitHub repository.
You will find the key starting point
is the Cloud Build file that executes all of the steps,
from optionally providing test data through generating change
data capture scripts to updates tables with the latest results,
to creating the views that provide the business logics.
After you have cloned your repository,
you can customize parameters to match your business needs
and available data sources.
And once your Cloud Build deployment is complete,
you will see the data sets you used as parameters
have been populated.
And you will find scripts to process changes
and retrieve additional data in the bucket
you have provisioned.
These models are foundation to more advanced solutions.
For example, on the Google Cloud marketplace,
you will find our Cortex Application layer content
with sample applications for synchronously exposing data
through REST APIs or integrating with other applications
through pops up.
Now that you know what Cortex Data Foundation is about,
check out the GitHub repository link here
to deploy the data foundation solution content.
You will find all the step by step instructions there.
And this is just the beginning.
Subscribe to this channel and stay
tuned to learn more about advanced business solutions
and other exciting options like our Microservices-based Cortex
Application layer.
[MUSIC PLAYING]