SPEAKER 1: So they give you the picture
of how this data platform puts together,
we're going to use a scenario of an advertising agency
that puts ads in the back of those screens in New York City
taxis.
If ever if you've been in a cab recently in New York.
There's an ad that plays, and you can interact with it.
To give you the picture first, this
is a simplified architecture of this data solution.
The taxis are sending their position in real time.
Where they are in the city, as well
as their destination, how many passengers they have,
and what sort of interactions the customers are doing.
Pub/Sub is then ingesting that data.
We're using dataflow to process a data pipeline.
To handle and process all that data.
We're also archiving the entire historical data
in BigQuery, which is a really powerful tool for you
to do ad hoc analysis of all of your historical data.
And at the end we want to make a visualization.
So why don't we actually see the pipeline in action.
So this the dataflow visualization tool.
And you see Robert's hovered-- we're
ingesting a little over 15,000 new elements per second
into this database, of all of this information.
But how do you visualize all that?
Google Maps is super powerful, but I'd
hate to point 15,000 updates per second at my Chrome browser.
So dataflow actually allows us to down-sample that events
stream to give us a real time visualization of taxi
positions.
And this is that visualization.
This is showing where all these taxis are in New York
City at any given time.
Now what if I want to look at one particular customer who's
placed ads with us.
A particular agency.
We can filter that dataflow, and now the visualization
will update so that we're only seeing ads that are being
placed for a specific store.
In this case, the Acme store-- it looks like it's
on the Upper East Side.
And sure enough, we see that we're placing ads, roughly,
geographically, near the store.
But right now, we're just using a dumb rule for this.
It's just a very simple rule.
Where is the store?
Where's the taxi?
Place the ads.
Can we do better?
Can we automate this to start to take
into account all the various data that we have about this?
Well, let's start by using BigQuery
to explore this dataset.
The query Robert has here looks at all of the taxis coming
from the airports and tries to see how many of them
are traveling near restaurants.
Now note here he's querying this historical data,
even as we're ingesting new data at 15,000 events per second.
So BigQuery gives him the ability
to have an always up-to-date real time view into his data.
In this case, we see that roughly 17% of our taxis
are from the airport and near a restaurant.
That sounds like an insight that maybe we ought
to do some ad targeting there.
And if you look at his query, you
can see it really is a non-trivial query.
And BigQuery is handling all that for us.
So right now, we are manually configuring these ads.
So I think what we want to do is look at a way
that we can do this in an automatic fashion.
Now we could similarly search for every different customer,
but how do we build a generalizable and flexible
model.
Well that sounds like a job for a machine learning.
Machine learning is tailor-made for taking
large chunks of data and building models
that give you insights out of that.
So we're going to build a model that takes into account all
of the data we have.
Where did the taxi start?
Where did it end?
How many people were in it?
What time of day is it?
Where's it going?
First, we're going to start by using--
as you see in this picture--
we're going to use Dataprep.
To take all that historical data,
clean it up, make sure that it's ready to use,
and then we're going to use it with ML engine.
So this is Dataprep.
The first thing to notice is across the top,
Dataprep has automatically inferred the schema
from my table.
I didn't have to tell it that.
Now if you look at that passenger count column,
you'll notice that it looks a little funny.
The histogram, up at the top, includes some elements that
says there's 10 or 11 people.
Now, I don't know about you, but I've been in a New York City
cab, and I'm pretty sure you can't actually
put 10 or 11 people in a cab.
So that's a common data bug.
Dataprep allows me to fix that in a number of ways.
I could delete the data.
I could replace that 11 with a one, on the theory
that it's a fat-finger mistake.
And Dataprep allows me to build simple recipes that clean up
my entire data at scale.
It also can do things like-- if you look at our location column
there, it's a composite column.
It's longitude and latitude, comma separated.
I'd really like to have a latitude column and a longitude
column.
Dataprep allows me to build a recipe that
splits columns with Composite data into separate columns.
So that I now have a better schema to work with.
Now this just scratches the surface
of how Dataprep can take this really tedious job of preparing
data for analytics, and make it much faster and much more
reliable.
Once we've done this, machine learning,
as you heard from [INAUDIBLE], allows
me to build a model on my laptop, upload it to Google,
and deploy at scale.
Let's see how that actually works.
If you scroll down in the dataflow pipeline,
you'll see we've added an ML engine call.
So now, as our taxi data is coming in,
we're calling to machine learning with the data
and saying, help us choose the best ad
to target for this particular user.
So let's switch back over to the map
with the deployed machine learning model,
and now you'll see that we start seeing
some interesting combinations.
When there's one passenger coming in towards midtown--
the taxi is currently in Chelsea, and it's on its way
to midtown--
we're presenting an offer for Manhattan Bagels.
Because it's a single person, maybe he or she
wants a bite to eat.
If we pick a taxi that has more than one person in it--
There it is.
In this case, that's a one person one,
but they're heading into a different location.
And instead we've chosen a winery.
If you pick a taxi with three people,
the system would automatically recommend discounted tickets
to Broadway, for example--
as multiple people coming to midtown in the evening.
So that's Cloud ML, making suggestions based
on all our available data.
We used Pub/Sub to ingest a huge stream of data at scale.
We use dataflow to build a pipeline to give us
a good analytics system.
We used BigQuery so that we always
had the complete historical data available.
We never had to deal with sample data.
And at the end, we use Cloud ML and Dataprep to build and train
at scale a model.
What's important here is what you didn't see.
I didn't deploy a virtual machine.
I didn't deploy patches.
I spent my time actually in the data.
That's the point of big data is to spend the time in the data
and not in the machines that take care of the data.
And that's what big data on the Google platform does.
So I want to thank you all and I'll turn it back over
to Brian.