ELIZABETH WU: Today we're presenting on real time
streaming applications on GCP.
So I'm going to present with my colleague, Akhilesh Singh.
So we're both from Syndey from the J/APAC Professional
Services team.
And Akhilesh is the lead strategic cloud engineer.
And I'm a cloud consultant in data analytics.
So we work on this stuff every day.
So hopefully we'll share a little bit
of what we've learned with you.
So there's three things we're going to cover.
The first is, why Google Cloud?
Then, I'll talk about what we have
to offer on Google Cloud and the different suites of products--
like the different products that are available to do streaming
real time applications.
And then I'll go through how you would do it.
And then after that we've got a demo
and there's a chance to win-- we have some free prizes so we're
going to get some interactivity going, and at that point
I'm going to ask you to pull out your phones.
So if everyone wants to get your internet connection ready,
that would be good.
So let's start with why.
Big data is in our DNA at Google.
We have over eight products--
so we have eight products with over 1 billion users.
A lot of the products that we're making available through to you
to do these types of problems-- to solve these problems,
are things that we use internally
and that we have made available to the community.
So this was started in 2002, where
we've been innovating on lots of different products.
So you might have heard of Bigtable or Spanner.
And a lot of these innovations were
made available to the open source community.
And so what we're now doing is then bringing those back
to Google Cloud Platform and making
them available as services-- as managed services to you.
Things like BigQuery and Pub/Sub and Bigtable
are all services that we provide.
So our focus for the customer is to focus on insights.
So rather than you having to worry about the infrastructure,
letting you focus on what you want to do with the data
and get the insights out is our key focus.
Another is looking at streaming analytics.
So being able to get real time business insights
and make your business more responsive.
So if you can get insights in real time,
then you can also react in real time.
And then we make that easier to do using machine learning.
So what you can do is, once you've
got your data coming in real time,
apply our machine learning algorithm on top of that
and actually make it actionable automatically
without you having to manually intervene.
So I'll talk about what next.
So what's available on Google Cloud Platform
to do streaming analytics and streaming apps.
So we focus on four stages of a data platform.
So the first is the ability to ingest at any scale.
So what we're going to show you today
is actually not that we can do streaming,
but that we can do it at scale.
So we're going to have the ability
to ingest thousands of millions of messages.
The second is to process and transform the data reliably.
The third is to store seamlessly.
And the fourth is to analyze, learn, visualize, with speed.
So we're going to go through each one of these
and I'm going to talk through all the Google Cloud Platform
products that enable each of these.
So that brings me to the how.
This is a typical data pipeline that I going to walk through.
And we have lots of different ways of doing things,
so this is just, sort of, one way of piecing things together,
but it's quite a common architecture.
So we start with streaming events coming in
from IoT devices, or in this case
we're actually going to have mobile device data that's
coming in and that will be written to a Cloud Pub/Sub
topic.
And we can read from that Cloud Pub/Sub topic as well.
So Cloud Pub/Sub as is a serverless, fully managed
asynchronous messaging platform.
So basically it's pretty straightforward.
You just publish a topic and read from it.
The second is, we can also ingest batch data.
So you can actually do both of those at the same time.
So Dataflow, which is going to be the next step,
is going to be able to handle both those types of streams
of data sources.
And that is going to be read into Cloud Storage, which
is our performant, unified and cost-effective object storage
bucket for binary blob data.
So then it's going to go into Cloud Dataflow.
So Dataflow has connectors to Pub/Sub and to Cloud Storage.
And it's going to be able to process our data
and apply transformations as we need them to batch your stream.
So one interesting thing about Dataflow
is that we can handle real time streaming data.
So we can apply things--
something you do differently to batch data
is you apply things like windowing functions, where
you take an aggregate of the data that's being read in
and you pass that downstream.
But we can also use all the products like Dataproc.
So, if you have already got Spark and Hadoop jobs,
you can migrate them over to Dataproc.
And we can then, from Dataflow, write it out
to be BigQuery or Bigtable.
So it can go to lots of different sources,
this is just an example.
So BigQuery is our Data Warehousing solution
and Bigtable is our high performance NoSQL
database for large workloads.
And then we can apply machine learning on top of that.
So once the data is in the database, we can pull it out
or we can actually use it directly from Dataflow.
And then we have a lot of visualization options
and different analytical tools.
So we can use Cloud Datalab law which
is was a Jupyter notebooks type of environment.
You can use either third party tools or something
like Data Studio, which is a Bi tool that we have.
You can share with other people or you can
produce applications reports.
So we've done the last one today as well,
so we can show you that.
OK, so we're about ready to start the competition.
So can I ask everyone if you want to participate,
get your phones out right now and get a web browser open.
And we're going to give you a URL in a minute,
so you would get that ready.
These are the prizes, just to give you
a little bit of an incentive.
So we have a pair of Google Cloud socks, a beanie,
and a t-shirt.
So what we're going to do is, you're going to vote
and the highest score will win one of these prizes.
Over to you, Akhilesh.
AKHILESH SINGH: So this is the URL we need to go to.
And you will see a very simple voting application
with military logos.
You can't go wrong there.
Just into your username and you start voting.
So, if everything goes well, you'll
see your name happening here.
ELIZABETH WU: Akhilesh--
Akhilesh.
Probably just leave URL a little bit longer.
AKHILESH SINGH: Sure.
ELIZABETH WU: Yeah.
OK, so we're going to start a timer
and we'll bring the URL back up and you'll have one minute.
So go back to the URL page, Akhilesh.
Can you bring up the URL?
Yep.
So it is bit.ly/2WRivbP and the "ivb" is lowercase.
Try and vote as many times as you can
when launching the products.
You might want to pick one that you really
like with the socks or the heart and the beanie
and then in about a minute we'll wind down.
Can you get the timer up Akhilesh?
Cool.
So Akhilesh, while it's going do you
mind explaining what's happening in the back end?
AKHILESH SINGH: Yeah, so in reak time
when you're pressing a button, the request
is going to App Engine instance.
And App Engine scales automatically,
you don't have to scale it.
We were actually not running in instances.
But because of the voting application,
we'll see that there will be many instances which
be recreated on the fly.
On the other side of things, there's
Dataflow which lists out the vote.
And in Dataflow flow what happens
is like, when the messages out in the pops up queue,
it reads from that.
And as you can see it's processing
huge number of messages without any scaling.
Was that the results that actually presented here?
So you can see like the Rangers winning as of now
for Sox and with a total of 15 votes.
And this is updating in real time
as you click on the buttons, right?
ELIZABETH WU: Oh, how much time left?
Are we nearly done?
AKHILESH SINGH: Yeah.
14 seconds away.
ELIZABETH WU: Oh, well 10--
10 seconds 5, 4, 3, 2, 1.
Akhilesh take a screenshot.
AKHILESH SINGH: Yeah.
ELIZABETH WU: So whoever is listed at the top
now after the session, please come over
to the experiment pod 1 and we'll give you your prize.
AKHILESH SINGH: Yes, so we'll take a second to just let it
refresh and we declare the winners.
Good.
So this is not the end of it.
Like, when you build a really cool app
it may actually become very popular across the globe.
And not only 100, but let's say like there
are millions of users.
So I'm using a tool which is, JMeter,
it's a bare bones JMeter which can create millions
of requests on our server.
And we start seeing that we are encountering 0% data
and the number of requests is this.
So there's already 3,000 requests
which has been passed in a short duration of time.
And let's see if it reflects on the--
that's what--
So as you see this is completely a serverless solution.
It relies on components which Liz talked about,
this App Engine, and this Dataflow.
This scale all available without any developer intervention.
Liz, up to you.
ELIZABETH WU: Back to the slides.
Cool, so I'm just going to talk through what we just
did, just to recap.
So can you skip--
sorry, I'll do it.
So what we did was we read real-time events
in from your mobile devices using an App Engine front end.
So that's actually scaling up to meet the demand.
So however many people here who have us
logging on at this time, we're scaling up,
and then Akhilesh just showed you
that last screen, when he was doing
was injecting a lot of extra events from a robot.
And then that's being read in through Cloud Pub/Sub.
So it's been published to a topic
and we're reading that back through from Cloud Dataflow.
So Cloud Dataflow is taking those messages
and processing them in Parallel, in real-time,
with latency of just a couple of seconds,
and publish them to BigQuery.
And from BigQuery, we're representing that dashboard
on the screen, which is showing you the leaderboard.
So we've managed to piece it together end to end.
And then we can also do a few more things
which we didn't during this presentation,
but it's just to demonstrate as well that you can actually
get batch data and at the same time.
And you can actually process both batches of streaming data
together with Cloud Dataflow, or separately if you wish,
and then output it to lots of different options
of destinations.
So just to recap, there's a couple of big customers
that are using solutions like this.
We have Twitter, who's using it to help them scale.
So they've got a hybrid Hadoop and Dataproc cluster
and they're working towards a fully serverless environment
on Google Cloud.
We have AirAsia with 500 million passengers.
They're using BigQuery, Pub/Sub, Data Studio.
And they've been able to reduce their costs using those tools.
And we have Spotify who are Cloud Native
and have implemented these solutions to help them.
So the key takeaway today is that the focus
is on real-time insights.
So we have these massive amounts of data coming in
and we want to be able to make use of them in real time,
make decisions in real time, and add value to your business
in real time.
So thank you very much.
[MUSIC PLAYING]