FELIPE HOFFA: Hello.
I'm Felipe Hoffa, and today I am in Melbourne, Australia
to meet my good friends at Shine Solutions.
With [INAUDIBLE],, we've done many challenges
like processing 30 terabytes of data with BigQuery,
or running Hadoop jobs while in the train, thanks to Dataflow.
But today, we want to see how much data
we can process with Dataflow before reaching the city.
Follow me.
I'm here with Graham Polley from Shine Solutions.
I'm so glad to see you again.
GRAHAM POLLEY: It's been a while.
Two years.
FELIPE HOFFA: It's been a while.
Two years.
I remember we did the 30 terabyte
challenge with BigQuery.
Then you recorded all this data while with Dataproc while
on the train.
GRAHAM POLLEY: Yeah, in 17 minutes,
we spun up with Dataproc cluster, Hadoop Spark,
and did it in the 17 minutes on the train.
FELIPE HOFFA: Nice.
And today, we're going to jump from Hadoop to Dataflow.
GRAHAM POLLEY: Dataflow, right.
FELIPE HOFFA: Yes.
GRAHAM POLLEY: We're going to read lots of data
and try and get the cluster up and running
and the pipeline finished by the time we hit the city.
FELIPE HOFFA: That is pretty cool.
But by the way, you've been doing a lot of great stuff
other than these challenges.
GRAHAM POLLEY: Yeah, a lot's happened in the last two years
since we've seen each other.
So I'm an official Google Developer Expert now, GDE.
FELIPE HOFFA: Congratulations.
GRAHAM POLLEY: Thank you very much,
for Google Cloud Platform.
And the only one in Australia.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: It's been cool.
It's an honor.
And we've got the new Sydney region there,
for Google Cloud Platform So we're-- that's super news
for us.
FELIPE HOFFA: Yeah.
The latency has gone down by a lot.
GRAHAM POLLEY: It's like 10 milliseconds the last time
I checked the DCP [INAUDIBLE].
And also, Shine have a new office up
in Sydney, which is cool.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: Yeah, so we're in Melbourne and Sydney.
FELIPE HOFFA: I should check that one out.
GRAHAM POLLEY: Yeah.
FELIPE HOFFA: So let's go toward [INAUDIBLE]
and then we will discuss all what you're
doing with your customers.
GRAHAM POLLEY: OK.
So we're going to run a simple pipeline while on the tram,
before we hit the city, and hopefully it
will be completed by that time.
FELIPE HOFFA: We are pretty close to the city.
GRAHAM POLLEY: Right.
So I better kick it off.
I'm going to kick it off with my development environment
so we can have a look at the code as well, how it's working.
But I'm going to get it started.
FELIPE HOFFA: Perfect.
GRAHAM POLLEY: So this pipeline is
going to read 10.6 billion records from Google Cloud
Storage.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: It's going to run them
through a transformation, a ParDo,
and then it's going to write the results, the BigQuery,
for further analysis.
So it somewhat simulates what we do in production
for our customers.
FELIPE HOFFA: Nice.
How many files is this?
GRAHAM POLLEY: It's a few thousand.
FELIPE HOFFA: A few thousand files containing 10 million--
10.6 million [INAUDIBLE]?
GRAHAM POLLEY: It's about half the terabyte.
Yeah, so the beauty of this is I only
have to write about 20 lines of code.
And using the Java SDK-- there are other SDKs for Dataflow
as well.
You have the Python one.
FELIPE HOFFA: Yeah?
GRAHAM POLLEY: I chose data.
I wrote 20 lines of code, but all the infrastructure
spun up for me.
I don't care about any of the infrastructure.
You guys worry about that stuff.
I don't have to worry that.
FELIPE HOFFA: Yeah, you have 20 lines of codes.
You are telling Dataflow to first read the files.
GRAHAM POLLEY: Correct.
FELIPE HOFFA: Then--
GRAHAM POLLEY: Transform it.
So I'm looking for a certain word in each title.
So the data that we're using is a sample from Wikipedia,
and so it's got all the page views for Wikipedia.
FELIPE HOFFA: This is whatever logic you want to run.
GRAHAM POLLEY: Yeah.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: And I'm looking for Melbourne at a time,
because we're in beautiful Melbouren.
FELIPE HOFFA: Nice.
And then we'll, at the end--
GRAHAM POLLEY: We do a write to BigQuery.
FELIPE HOFFA: Write to BigQuery.
GRAHAM POLLEY: Yeah.
FELIPE HOFFA: A strange write?
GRAHAM POLLEY: No, not a strange write.
So while I was running my tests, I
found out doing a strange write to BigQuery
was taking a little bit longer than I had hoped.
And the reason was Dataflow was producing a lot of files
that BigQuery then subsequently has to do a load
job in each of those files in.
So what I do is I do a little trick.
I specify without charting, and I write back
to GCS a single file, and then load that
in using the BigQuery API.
A little pro tip.
FELIPE HOFFA: And how is your deployment going?
GRAHAM POLLEY: OK.
Let's have a look.
So we can see from the Java logs that the workers
have started successfully.
So probably the next thing we want to do
is flick over to Google Cloud Platform console
and have a look at the Dataflow platform.
FELIPE HOFFA: So who is monitoring all of your--
GRAHAM POLLEY: Everything.
The beauty of this is that it allows me to focus on the code.
I don't have to worry about any of the infrastructure.
So it's like hands off, great, it
lets us do what we want to do.
FELIPE HOFFA: That's good.
GRAHAM POLLEY: So if we go to the Google Cloud Platform
and we jump over to Dataflow.
Fingers crossed, we're tethering off my phone, remember.
FELIPE HOFFA: Yes.
GRAHAM POLLEY: We're not using any type of problem.
FELIPE HOFFA: We're doing this well on the tram.
GRAHAM POLLEY: Yeah.
So I'm tethering my phone, which is in my bag,
and we're spitting up a massive cluster of Google Cloud
Platform.
FELIPE HOFFA: So now we are running Dataflow here.
GRAHAM POLLEY: Yeah.
So the pipeline is spun up.
It's Actually spun up--
I think it's spun up to 50 VMs or 35 VMs, which
Dataflow did itself.
It looked at the amount of data that it had to read
and went, oh, I need more VMs in the stack, started
bringing them in.
And you can see it's autoscaling now.
FELIPE HOFFA: Able to [INAUDIBLE]----
how many records are we processing?
GRAHAM POLLEY: This is crazy.
On 4G network, it's spinning all this up.
We're processing 27 million records a second.
FELIPE HOFFA: A second.
GRAHAM POLLEY: A second.
FELIPE HOFFA: We'd better hurry up.
We're getting close to the city.
GRAHAM POLLEY: Where are we now?
Shrine of Remembrance.
So we're pretty close.
We've better get cracking.
FELIPE HOFFA: It is some city highlights, too.
GRAHAM POLLEY: So of course, when
we deploy this for our customers,
we don't run it from our development environments,
obviously, so we have a proper deployment process for now.
And we use Cloud functions for all that stuff.
We're doing good.
FELIPE HOFFA: OK, so what kind of stuff
are you doing for your customers?
GRAHAM POLLEY: So with our customers,
this is a very basic pipeline that I'm showing today.
It's just the basic read, transform, and write.
But for our customers, we apply a lot more complex business
logic and rules to the pipeline, so we have multiple side inputs
coming in for our Dataflow pipeline
from disparate data sources, transforming it
and then multiple writes out to BigQuery for further analysis
by the product teams.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: And I can show you some examples.
FELIPE HOFFA: I would love to see one.
GRAHAM POLLEY: Yeah, I've got some graph examples.
So this is a really good example of one
of our public in production.
This is a massive read from GCS.
I think this is maybe 100 files, reading
about 100 million records.
FELIPE HOFFA: These are real life files.
GRAHAM POLLEY: Yeah, right, sitting in GCS.
But the beauty of this is look at all these side inputs
coming in and enriching the data,
applying business rules, logic, transforming that data,
and then writing to multiple destinations in BigQuery.
FELIPE HOFFA: Nice.
And you're reading this from a huge file in Google Cloud
Storage, enriching it with BigQuery tables?
GRAHAM POLLEY: Actually, this one
is reading from multiple files in Google Cloud Storage
and then writing to multiple tables.
But I have an example of another one which is one big file read.
You can see here.
So this is a massive read from GCS, a transform,
and then, I don't know, 20 outputs to BigQuery.
FELIPE HOFFA: Nice.
How did you launch this processes?
GRAHAM POLLEY: Not from my development environment.
FELIPE HOFFA: It's not you clicking can run every day.
GRAHAM POLLEY: So we've got a pretty nice pattern
that we use in production and our customers like it as well.
So what we do is we set up a Google Cloud
Function, listening to a [INAUDIBLE],,
and when a new file arrives, that kicks the Cloud Function
off which cause a Dataflow template which then
starts processing the data.
And that's the trick, to run the Dataflow pipeline
as a template.
FELIPE HOFFA: Yes.
You set up all your code as a Dataflow template--
GRAHAM POLLEY: Correct, yeah.
FELIPE HOFFA: --that then, you can trigger from a Google Cloud
Function with whatever is in your file to be read.
GRAHAM POLLEY: Correct.
So it's all hands off.
We don't maintain any of that stuff.
It's servlets.
It's all the way through.
FELIPE HOFFA: Servlets.
GRAHAM POLLEY: Yeah.
I had to get the buzz word in, right?
FELIPE HOFFA: Yes.
How many records have we processed so far?
GRAHAM POLLEY: 8.6 billion.
FELIPE HOFFA: 8.6 billion so far?
OK, so let's go back to the customers,
because it's really interesting to see
how everything is autoscaling.
GRAHAM POLLEY: Bit of sun catching your eye, there.
FELIPE HOFFA: Yeah.
GRAHAM POLLEY: So if you can jump over here,
we can actually see the autoscaling history
on the pipelines, right?
So that looks quite nice, so clusters starting up,
more VMs coming into the stack to process the data.
The flows is detecting that there's not
so much data to process anymore and starts
dropping VMs in the stack, which saves
our clients a lot of money.
FELIPE HOFFA: Is that right?
Because they're paying by server.
GRAHAM POLLEY: Yeah, correct.
FELIPE HOFFA: And this one jumped all the way back
to 20 servers at the beginning?
GRAHAM POLLEY: Zero, 20, and then went back down
to 10, and back up to 20 again, and then, eventually,
the whole cluster was torn up.
FELIPE HOFFA: That's great.
So less servers are needed in some steps--
GRAHAM POLLEY: Correct.
FELIPE HOFFA: --as autoscaling goes down.
GRAHAM POLLEY: But we don't determine those steps.
Dataflow does all that for us with autoscaling.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: So even if we go back here,
we can see all the pipelines that have run in production.
FELIPE HOFFA: Oh, so these are running permanently,
like many times a day, all thanks to Cloud Functions?
GRAHAM POLLEY: Yeah, all triggering
from Cloud Functions.
FELIPE HOFFA: And then you have the templates.
GRAHAM POLLEY: That's right.
FELIPE HOFFA: Nice.
GRAHAM POLLEY: How close are we to the city?
We'd better do a check--
oh, we're at Fed Square, OK.
So we'd better check our pipeline.
So how are we doing?
OK.
Almost there.
FELIPE HOFFA: Almost there?
Running--
GRAHAM POLLEY: We're up to 9 billion records now.
FELIPE HOFFA: Excellent.
GRAHAM POLLEY: 567 gig.
FELIPE HOFFA: Excellent.
GRAHAM POLLEY: So we've got to just wait one more minute,
I would say, and then we'll hit the city,
and hopefully the pipeline will be finished by then.
FELIPE HOFFA: And hopefully-- perfect.
Any other tips that you have for people
doing stuff with Dataflow?
GRAHAM POLLEY: The great thing about Dataflow
is it allows you to concentrate on the code itself,
rather than worry about any of the infrastructure.
The beauty of Dataflow, as well, it
runs at both batch and streaming mode.
FELIPE HOFFA: Yes.
GRAHAM POLLEY: So you write once, but run it
in batch and streaming.
It's really helpful.
So if you've got streaming realtime data,
you can also use Dataflow, which is great.
FELIPE HOFFA: Yeah, we did this with batch,
but you're also using it for--
GRAHAM POLLEY: For streaming as well, yeah.
And we do a lot of BigQuery streaming on top of that,
and we mix the tools together to solve the problems
that we need to solve.
FELIPE HOFFA: Nice.
And you also need to decide sometimes,
do you use BigQuery or do you use Dataflow?
GRAHAM POLLEY: Yeah, so it's picking
the best tool for the job.
Right?
So BigQuery's great at smashing true data and aggregating it,
and Dataflow is really good for pulling all that data in,
transforming it, and sending it to BigQuery
for further analysis.
FELIPE HOFFA: Nice, and how are we doing here?
GRAHAM POLLEY: We're done.
FELIPE HOFFA: We're done.
Excellent.
Time for a pint.
GRAHAM POLLEY: Definitely.
FELIPE HOFFA: Awesome.
So we made it back to the Shine offices on time.
Challenge completed.
GRAHAM POLLEY: Challenge completed.
FELIPE HOFFA: That was pretty nice.
GRAHAM POLLEY: That was a great challenge.
FELIPE HOFFA: Yes.
How much data you process, then?
GRAHAM POLLEY: We processed over 10 billion records that
were in GCS, about 500 gigabytes, almost half
a terabyte, all in under 10 minutes using Cloud Dataflow.
FELIPE HOFFA: In under 10 minutes, 10 billion records?
That's like--
GRAHAM POLLEY: 29 million records a second
being read and transformed in parallel.
FELIPE HOFFA: While we were tethering?
GRAHAM POLLEY: While we were tethering off my mobile phone.
FELIPE HOFFA: That's pretty cool.
Who handled all the complexity?
GRAHAM POLLEY: Well, the beauty of it
is Dataflow handled all the complexity of it, right?
So it spun up the worker pool, autoscaled
VMs, we went up to 50 VMs, and then when it was finished,
tore the whole cluster down.
FELIPE HOFFA: That was pretty nice.
GRAHAM POLLEY: Very nice.
FELIPE HOFFA: Yes.
So what do we do now?
GRAHAM POLLEY: That's a tough question.
Drink a beer?
FELIPE HOFFA: OK.
Salud.
GRAHAM POLLEY: Salud.
Prost.