AMY KRISHNAMOHAN: My name is Amy Krishnamohan.
I'm from a product marketing team for databases.
So many of you maybe wonder, like,
why person from marketing or databases are here.
Like, whoa.
Oh, it's not working?
Yeah.
But actually building a right data foundation
for machine learning and AI is actually extremely important.
We are seeing more and more of those use
cases from many of our customer and also from the Google
itself.
We'll start when we actually start the slides.
[LAUGHS] But I can actually talk through
without seeing the slide.
Actually, that's fine.
Yeah.
So many of you guys are data scientists, right?
And then I'm sure you have seen many of the survey that
says that, like, 60% of time you are spending on cleansing
and organizing data.
And then some of the survey data also said that 60% of you guys
just hate those jobs.
Really putting the right data in the right place
is actually very, very important,
and especially something like a machine
learning AI, where you're handling
massive amount of data.
It's not just gigabytes or megabyte,
where you can run things on your laptop.
You are not just downloading CSV file.
You are really loading petabytes of data
into really great databases, where you can actually
build a platform for data, so that you can leverage
those data over and over again.
So that's why I'm here to talk about those building
a right data platform, and really share experience
that Google had.
So Google had-- years ago, we've been using MySQL, CSV file,
Postgres, or enterprise database that all of you knows,
which is extremely expensive.
We tried all of them.
And once we wanted to build not only to serve our users,
but also, with development of ML and AI,
we want to really try to provide a different services.
And what we are seeing, the data just growing.
Not just growing linearly, it's growing exponentially.
It's just that our database couldn't keep up.
We've been using mySQL.
We sharded the database, sharded again,
sharded again, and sharded again.
And one day, we actually had to shut down
operations for a week for us to update the database.
Can you imagine that Google Search going down for a week?
What will you do with that?
That's not a solution that we can--
yeah.
That's not something that we can handle.
So these are all the things that we talked about.
So a lot of you are data scientists.
Like, you cannot just, maybe, download something from Google
Sheet, put something in the, like,
MySQL and download it from there.
And now there is some database called NoSQL
where maybe SQL may not work.
Maybe you have to use some program language
to pull the data.
And then your enterprise team says
that, hey, we have a data warehouse where
you have all the data.
And when you actually looked at it all,
oh, sorry, you're missing some data.
So you have to pull all those data, massage it,
and put it all together in your really great,
8-core Intel laptop, which you cannot get anything.
So we are here, really, to provide
a full range of services that you guys will
need to process all of your data needs,
starting from, like, operational data,
getting data from your website--
maybe getting the customer data, transactional data.
We have a simple, plan MySQL Postgres
databases, two really great databases like Spanner,
where you actually--
I'll really recommend you to learn a bit more about it.
And also, we have a streaming, and batch processing tools,
and data warehousing.
I'm sure many of you know about the BigQuery and BigQueryML.
A lot of customers are loving it.
And the digitalization tool, we just
made announcement with the Looker.
Also, you could actually visualize your old insights
really well, and also, the AI platform
that you guys all love.
So what does Google know about database?
Like I said, we tried it all, and we had to be--
we are now reached to a point where
we have eight products that have more than a billion users.
And as you know, many of our services
already have ML and AI deeply integrated in our services.
In Gmail, we recommend how you can reply,
and in Google search, I feel like I can just type A
and that they know what I want to type.
There are a lot of really interesting recommendation
engines that you see--
and Google Drive as well.
So Google really rethought how to build this infrastructure,
like, what we should do, how we should build a database.
We cannot just rely on the database that exists
in the market.
So they really thought through about building a new file
system, building a new processing like a MapReduce,
and building a new database like BigTable and Spanner,
and really building a new engine for the AI, like TensorFlow.
So then a lot of papers that published that actually
triggered a lot of interesting open source projects
that you've seen here.
And then, at Google Cloud, we love
to actually prove we are actually
providing all the services as fully managed services
for-- you can just leverage very easily.
So I'll give two example of our databases
and how we can leverage this foundation together
with a lot of ML and AI use cases.
So first one is our BigTable.
It is a NoSQL database.
It's a very scalable, wide-column database that
are used together with TPU.
So it is-- actually, there is a great tutorial.
I'm not going to go through this tutorial today,
but I would highly recommend you guys to go through it.
It actually has a great use case on deep residual learning
for image recognition.
And it had a training data set, ImageNet.
And it also had a TPU estimator to make sure
that you are using the TPU in the right way.
So it actually has a very great, step-by-step process,
so I highly recommend you to go through it.
So why BigTable?
I'm sure many of you already know about TensorFlow
from other sections, so let me go a little bit more
deeply into the BigTable.
So BigTable is actually really great high-throughput.
So when we were doing the Ads business,
we realized that there is a really, really fast bidding
process in the Google Ads where we really required fast
read and write coming in.
And we are seeing the same use case
in the AI and ML process where you
want to train your data with batch data coming in streaming.
Maybe there's something happening on social media.
You are seeing so many texts coming in, it's in the burst.
Maybe you launch a new game and you see explosion of users,
and you are getting so much of data.
And those petabytes of data needs really high throughput.
That is where the BigTable comes in.
So it has actually had a really, really high throughput
and had a low single latency.
So it is widely used in a lot of our Google services right now.
And the biggest strength is the scalability.
Is scales linearly.
The platform scales linearly as we add more nodes.
And the reason why it works is because of the separation
of compute and storage.
Because we have a routing layer that actually separates
the compute and storage, so whenever
there is more queries that are coming in
and one node gets much, much more traffics,
we actually route those traffic to another node
where it has less traffic, so it evens out.
And we felt that if you feel that you
may need to add more nodes, and those databases automatically
rebalance those workloads.
So as a data scientist or a developer,
you don't really want to think about how you can manage
or perform databases, right?
How do you want to load balance your databases-- that should
be least of your concern.
You should focus more about getting the insights,
putting the data up fast, and getting out the data fast.
That's what BigTable really allows-- to do that.
So this is a quick summary.
If you use BigTable and TPU, like I mentioned,
it has a great tutorial that you can leverage.
It has a really low latency--
very scalable.
You can scale your data to petabytes of scale.
And you can really train your data set-- really,
really ultra high-speed.
And next example is actually a little bit more boring use
cases.
How many of you guys are using MySQL or Postgres?
It's quite a lot, yes.
So I think that's like-- for an operational database,
you cannot avoid that.
But for, maybe, data science side, it's a little bit more,
like, half and half.
And how many of you are using, like, data warehousing?
Oh, OK, and how many of you are using both of them together?
OK, that's great.
So you would think that you--
the use case that I mentioned, sometimes you
have a large amount of data in the BigQuery
or in some of the data warehousing platforms
that you have, and then you realize,
hey, I just want to add some transactional data to analyze
this.
And, can I get it?
And your enterprise team might have
said-- data team say, oh, maybe wait for, like, three weeks.
We'll get back to you.
And let's see what happens.
Oh, you need to schedule a new ETL job.
You need to cleanse those data.
Are you sure the data format is right?
You get, like, tons of questions,
and you feel like, OK, maybe we'll do it in next quarter.
This use case is very, very common.
So we have seen a lot of customers using classical,
right?
The MySQL and PostgresQL has been around for 30 years,
and everybody just loves it, and they just
want to use hassle-free MySQL and Postgres.
So we introduced the classical, where it's
fully-management databases.
It's all backed by Google SRE, and you
can upgrade your instances to big, 64-gigabyte size,
and also, it ensures 99.95% high availability.
However, you want to bring those data that
are stored in the classical to the data warehousing easily,
at your fingertips, right?
You want to, maybe, query the live data--
hey, what happened to this past 10
minutes of this crazy Black Friday sales?
I want to analyze that.
But I want to just join, using SQL query, with my analytics.
But you cannot just keep the root-level access
to your databases, especially if you're running E-Commerce
Engine, just because data analysts ask you, hey,
I want to get the root password.
No enterprise DBA will give that, right?
So you really need a secure connection to pull those data.
So we introduced a feature Cloud SQL Federation free.
So let me actually give you a quick demo on that.
So let me see if I can pick this one here.
All right, so I put it-- everybody can see the screen?
I think I should make it little bit more bigger.
So there are two databases.
So I made data really, really simple.
I know you guys are dealing with a very, very complex problem.
But for just demo purposes, I made a very simple table.
For customer data, where we usually store
it is a data warehouse, and that goes into BigQuery.
And the same data in this use case of we--
I picked liquor sales, There's the vendor name.
There's an item description-- how many volumes they buy.
Somebody bought, like, 200 bottles of vodka.
I don't know why.
And there's a customer ID.
It's not it.
So what I did is first, classical,
create a simple MySQL database.
And this is the instance.
Can you guys see OK?
Yeah.
And then, I simply load data.
So it is very easy to just--
you can import those data with a CSV file or SQL.
And one of the life-changing tips for you guys
is that you use a CSV to SQL converter.
That is easily available in the Google search.
So if you don't want to write, like, schema for your CSV file,
which I really, really hate, there
are some tools available that actually generate the old SQL
query for you.
So this is an excellent one which I used.
So all you need to do, just browse it, and then load
your CSV file in the cloud storage.
And then, boom, it's there, and it will
be all entered in the database.
And the BigQuery side, the same thing.
You can actually go and add the data.
So there is a different way that you can actually add the data.
I know many of you already know that you can use a public data
set.
It's available.
Also, you can click that.
Or, you can create a connection.
So which is more?
I'm going to show you.
So let's look at the customer data, which is already existing
in the work of BigQuery.
And then you can see this schema here.
There is a small table, and the preview.
And the beauty of the BigQuery is that when--
like a MySQL where you need to define the table first
and then what kind of columns will be in it,
BigQuery actually automatically detects
all those schema for you.
So you don't need to worry about it.
You can just upload your CSV file,
and they will automatically create a table
and create a schema.
It's a very neat function.
So in the customer table, then, you can see this here.
And I want to add--
join-- those customer data together with the sales data
using the customer ID--
very simple join transaction.
But like I mentioned, if you have data warehousing
team in here and you have an e-commerce operation
team in the other way, to making those data come together
is actually a lot of work.
But here, all you need to do is add data and create connection.
So here, it actually provides you a very secure connection
to connect the MySQL instance, and you
can add the connection ID, which is a random text that you
can just make it here.
And Friendly name, Description-- you can make it all optional.
And then you need to add a classical instance ID.
So one tip that I've found is that it can be,
actually, very long to find a project ID, an instance
ID, and things like that.
It's very cumbersome.
But if you go to classical console,
and there, you can actually see the instance connection name,
you can just simply copy here.
And then copy the database, and enter a database name.
It is my table--
store-- and the user name.
User name is important because, like I mentioned,
you want you any root-level access to your databases,
right?
So your DBA needs to create a separate user,
so you guys can pull the data, so that you guys don't actually
access any sensitive information, so that there
is some kind of segregation of duty.
And that can be done.
It's actually very simply.
So when you come to MySQL, click Users,
and let's create a user account.
That is that simple.
And under the hood in the database level,
you can also set up different access for different tables
or different data.
And here, so you enter "amykm," username, and the password.
Yeah.
So I already set up the connection.
So once you've done that, you'll be
able to see that external connection,
something called like this comes out.
And then, the iData you defined will be shown here.
So here in the BigQuery, where you can the SQL query directly,
let's see whether we can pull the data.
So this is my cheat sheet for pulling the data.
So we want to, first, pull the system table,
and let's see-- it works.
OK, it's Running.
See, yay, it's working.
So the information schema for that MySQL table that I created
is all here, right?
And let's see where you can do, actually, more interesting
joins.
So as you've seen, customer data is stored in the BigQuery,
and transaction data is stored in MySQL.
So we want to select the ZIP code and vendor name.
We want to see, in what ZIP code, what kind of brands
are popular.
This is very simple analysis.
And so we select those columns and the description,
and then FROM.
When you do FROM, you just make a connection string--
add a connection string here.
And as RP, just like you're deferring to other tables.
And then, add external query function there.
You can add a connection string here.
Sorry, it was a connection string here, and then comma,
and then add the same SQL query that you use everywhere.
It is not that--
this is really, really simple to use.
So SELECT ALL from mytable, and then you query--
join on the custom ID on those two tables.
And you run this.
And as you can see, this is really, really simple, right?
So you don't need to worry about bringing the data,
downloading the CSV file, put it here, and put it-- load it
again, the database, merge it on the database,
join together-- you don't have to worry about all this things.
Using the connection string, you can just use a single SQL query
and match the data.
And this is very simple use cases,
but if you want to, maybe, apply the BigQueryML
and you don't have data on hand--
all the data is in different databases-- you can totally
leverage this federated query feature to really get
all the data that you want.
So I think that's pretty much all the summary that I have.
I have only 30 minutes, so it seems like I five minutes left.
So I'll be happy to take your questions.
Thank you.
[APPLAUSE]