[MUSIC PLAYING]
PRIYANKA VERGADIA: When you have a data-intensive applications,
such as health insurance, and your on-premise traditional
data warehouse does not quite fit the pace and the wallet
with which your data science team wants to innovate,
you know it's time to migrate your data warehouse to Cloud.
So today, I invited Gary Morreale, director
of Data Services team, to share with us
how they migrated their data warehouse to BigQuery
to innovate faster.
Hi, Gary.
Welcome to the show.
GARY MORREALE: Hi.
Thanks for having me.
PRIYANKA VERGADIA: Well, let's start with what is your role.
GARY MORREALE: Sure.
What my team does, the Data Services team,
is we support all of the databases at the organization.
PRIYANKA VERGADIA: So I imagine it is large amount of data.
As a data team, what's your team's biggest challenge?
GARY MORREALE: So our biggest challenge
is getting that data into our databases
and then getting it back out in a timely fashion
so we can meet the businesses' needs.
PRIYANKA VERGADIA: So how much amount of data
are we talking about here?
GARY MORREALE: We're talking about terabytes of data.
And it's continuously growing.
As everybody in the industry probably knows,
it's got an exponential curve to it.
And it's getting bigger all the time.
PRIYANKA VERGADIA: So it sounds like the biggest goal was
to really create a forward-looking system
to store this ever-growing data in a secure, scalable, and also
cost-effective manner.
GARY MORREALE: So getting that data in and out
and making that data available to all of our users--
we have thousands of users and terabytes of data.
And meeting those challenges was getting harder and harder
in our existing environment.
PRIYANKA VERGADIA: So migration from on-premise data
warehouse to BigQuery is quite an undertaking,
definitely not flip of a switch kind of migration.
You need to make sure your downstream systems--
in this case, thousands of downstream systems--
don't break due to inconsistent results in your final tables
after your migration.
So how did you approach this migration?
GARY MORREALE: So we engaged Datametica,
one of Google's business partners,
to help us complete an assessment of everything
we have today, which included not just our data warehouse,
but all the applications that use it and all of the processes
that pull data in and take data out.
And they were able to come back and give us
a complete picture of what it was we
were going to be dealing with.
PRIYANKA VERGADIA: Well, after hearing that solid migration
plan, I'm very excited to dive into the migration
architecture.
It sounds like this is an ongoing migration right now.
Walk me through.
GARY MORREALE: So the first thing we had to do
was put the foundation in place.
So that involved getting our networking set up, getting
the environment in Google set up,
all the connections between the different environments here
and there between our data center and Google--
so we got all of that in place.
And then once that was there, then we
were able to start working on moving the data over.
So we have all the data in our existing system.
And we used a transfer agent to transfer to BigQuery moves.
And that allowed us to pull the data into Cloud Storage.
And then from there, the data went into data quality checks
via Dataflow, which allowed us to move
the data into the environment.
We were able to set that scheduling
up using Cloud Composer to do scheduling and orchestration as
well.
Then we got it into the staging environment
in BigQuery, which allowed us to push it into our core
environment and then our information
layer so that our end users would have access
to everything.
And then we have to worry about keeping everything
in sync because while we're in the migration process, which
will go on for some time, we need to make sure
that what was happening in the old system
is what's happening in the new system.
And Pelican is the tool with a GKE cluster
that allows us to do that comparison between the two
and make sure things stay in sync, which has
been a really good feature--
gives us a lot of confidence in what we're doing.
And then the end result of all of this
is that our consumers on the final end of this journey--
people using SQL Server Reporting Services, Tableau,
other applications we've written,
or third-party applications--
are able to now just point at BigQuery instead of pointing
at our existing system and do the same things that they
were doing previously.
PRIYANKA VERGADIA: Well, awesome.
That was a lot.
Now, given your consumers of the data, which
is the data scientists and the business analytics team,
you have this modern data warehouse in BigQuery
at this point.
How do you ingest and process the data?
And then how do data science and business analytics team
do their job, which is analyze this data and make predictions?
GARY MORREALE: Sure.
Well, we have a lot of different data sources.
Being a health insurer, we get data from various providers,
like hospitals and doctors' offices,
testing centers, things like that.
We also get data from our members.
And then we have a lot of processing business partners
that we work as well as government entities.
So we needed a way to set up secure file transfers, which
we can do.
And we can load that into Cloud Storage
and then use API-based services, as well, to do the same thing.
And then we can move that using Dataproc
with our Hadoop-type environments
that we're used to using to load the data up into BigQuery.
So now we've got everything in place.
And the data scientists can then use the tools
that they're used to using, which, in this case,
would be the Vertex artificial intelligence pipelines that
are provided in the Google environment.
They use Jupyter Notebooks to work with all of that.
They're able to take advantage of the fact
that it's not a static environment, like we
had before.
So they can add resources.
For example, GPUs become a huge benefit in this environment
to data scientists.
So they can dynamically add them as needed.
And then what they end up doing in the end
is they build models that help us assess and do
preventative care, understand what's
going on with our pharmacies, what risks people may have,
what kind of fraud might be going on.
It just allows them to do the magic
that it is-- the data scientists do
and understanding how all this data
can be put to use to make our members' health better.
PRIYANKA VERGADIA: That is amazing.
So what is next in this architecture?
And what's your team working on now?
GARY MORREALE: So the nice thing is now
that we have this foundation in place
with the security, the networking, all of that,
we can start to look at what other types of opportunities
there might be.
So we've been looking at voice recognition software.
We've been looking at different types of reporting tools
that are provided up there.
There's just a whole host of different applications
that are provided on the Google platform
that we can now easily put into place compared to before,
having to buy different hardware in our own data center.
So it just presents a lot of opportunities.
And we're looking at a lot of them.
PRIYANKA VERGADIA: Well, now that everything is in one place
and your infrastructure layers are all
set up, you can just innovate on top of that, which is amazing.
Thank you so much, Gary.
And we congratulate you for what you have accomplished, you
and the team have accomplished.
And we look forward to seeing more updates
on this architecture.
GARY MORREALE: Great.
Thank you.
It's been great talking to you today.
PRIYANKA VERGADIA: If you have any specific topics or industry
architectures you would like me to cover,
put them in the comments, and I will bring them to you.