Alexis Moussine-Pouchkine: Hi, everyone.
We're back here at Devoxx 2014, and I'm here with Felipe.
Hello.
FELIPE HOFFA: Hello, [INAUDIBLE].
MALE SPEAKER: We're here to talk about big data.
You're our big data guy, and maybe more
specifically, BigQuery, and that is
one of the things you were talking
about yesterday at the conference.
FELIPE HOFFA: Yes, exactly.
We did a presentation with Thomas Park
from the BigQuery team.
He's our software engineer.
And we've presented one of the new features of BigQuery,
the user defined functions.
MALE SPEAKER: Before we get to that,
I think that is absolutely a very nice feature that
is coming along really well.
What is BigQuery?
FELIPE HOFFA: BigQuery is a Google product
on the cloud that's useful to analyze data really,
really fast.
It's zero configuration.
You just need to load your data to be queried,
and then you can query it in seconds using just SQL.
MALE SPEAKER: So interactive time
is what we call it, so don't wait for a batch
to run all night and get a result.
So you can iterate.
That's the addictive part of BigQuery is you can tweak it.
How do you query it?
FELIPE HOFFA: It's just using SQL,
and the great feature of BigQuery
is that you don't need to care about indexing your tables.
It just runs fast.
Whatever arbitrary query you write,
it will do full table scans in seconds.
MALE SPEAKER: Right.
So this is big data.
I don't do big data, so why should I care about this?
I don't have big enough data sets, right?
FELIPE HOFFA: If you have been using tables and databases,
if you have been writing SQL queries,
you've had frustrations, probably.
MALE SPEAKER: Yes.
I think that is fair to say.
FELIPE HOFFA: Yeah?
So if you have ever seen a query taking minutes or hours to run,
with BigQuery, you can take the same time to seconds.
MALE SPEAKER: So you're saying if I have long running SQL
queries, I'm doing big data.
I just don't know it.
FELIPE HOFFA: Exactly.
Exactly.
MALE SPEAKER: OK, great.
FELIPE HOFFA: Also, BigQuery is awesome
if you want to share data.
If you have data that you want to share with more people,
BigQuery has strong access controls
that let you share the data immediately with other people
without them needing to set it up.
MALE SPEAKER: That's very good.
So everybody knows SQL.
Upload your data, start querying results
in a matter of a few seconds.
That's great.
To introduce that user defined function feature that you
mentioned to start with, that means
there is some limitation, probably
in SQL as a querying language.
What are those limitations?
FELIPE HOFFA: I love SQL.
It allows me to write 90%, 95% of my questions.
But sometimes it's not enough.
Sometimes you want something more of a procedural language
to transform the data, to analyze it,
but at the same time, you want the power of BigQuery
to go over the full table in seconds.
So the question is, how do we get a procedural language
to work in the same massive cluster
that BigQuery has to analyze data?
That's why the BigQuery team, Thomas,
is adding this new feature, the user
defined functions, where you can write in JavaScript
any arbitrary function you would like
that runs inside the cluster within your SQL query.
MALE SPEAKER: So one of the things
you were able to do with BigQuery
is run regular expressions, I mean embed regular.
So now I could, instead of a regular expression,
have a full JavaScript function that's
actually computing stuff?
FELIPE HOFFA: Exactly.
One of the examples we ran yesterday, we were processing
logs.
Sometimes logs have URL encoded information
or sometimes they have [INAUDIBLE] information.
MALE SPEAKER: Data is never clean, right?
FELIPE HOFFA: And regular expressions
are limited in what they can do, but if you
can have that JavaScript filtering inside your SQL query
that cleans up your data however you may want to do it,
it's there now.
MALE SPEAKER: So how is this implemented?
Do you get a performance hit?
Are you still in interactive time?
Where is this executed?
Tell us a little bit about how this is implemented.
FELIPE HOFFA: So one of the questions
that people ask us is, why JavaScript?
Why did we choose that language?
MALE SPEAKER: Yeah, that's also a valid question.
FELIPE HOFFA: Well, it turns out at Google,
we also developed a very fast JavaScript environment.
MALE SPEAKER: I think I've heard of something called V8.
FELIPE HOFFA: V8, exactly.
So we took the exact V8 from Chrome
and we put it inside the BigQuery cluster,
so its node is running V8 and your JavaScript functions
run inside its node.
MALE SPEAKER: And probably, it's worth
mentioning here that the reason BigQuery is maybe
more than a product but a service
is that Google does all the heavy lifting.
We run the cluster, we set everything up.
Obviously, it's distributed.
This is the only way you can get.
Give us maybe an idea of the amounts of data
that can be processed in a matter of a few seconds,
the data you used yesterday, for example.
FELIPE HOFFA: One of my favorite examples
are the Wikipedia logs.
They have billions of rows each month,
but I am able to query it during the presentation several times
in seconds, and just keep changing the queries to try
different stuff and learn different.
MALE SPEAKER: And so when you do add those user defined
functions, is the performance getting a whole lot worse?
Are we back to--
FELIPE HOFFA: There is a performance hit, of course.
There is communication between processes,
but only inside one node instead of going outside,
so it's much faster, but we could still
run all our queries--
MALE SPEAKER: In interactive time still.
FELIPE HOFFA: Interactive time, yeah.
MALE SPEAKER: That's very exciting.
When can we get our hands on this?
Is this coming?
It's not just a prototype.
It's actually--
FELIPE HOFFA: Right now, it's not available publicly yet,
but several people are testing it.
If you have any use case that would
be very nice to test with us, just call us
and we will enable the user defined function for you.
MALE SPEAKER: Your name is Felipe Hoffa,
and this is the guy you want to ping
if you want to have early access to this.
Certainly keep an eye on this new feature in the cloud
platform from Google specifically in BigQuery,
user defined functions.
Thanks for your time,
[INTERPOSING VOICES]
MALE SPEAKER: And we'll see you online.
Cheers.