FRANCESC CAMPOY FLORES: Hi.
I am Francesc Campoy, and I'm a developer programs engineer
for the Go team here at Google in Mountain View.
And today I'm with Kurt.
KURT SCHWEHR: Hi.
I'm Kurt Schwehr.
I'm on the Google Maps Ocean's team.
And I work on ocean data with all sorts of different things.
But here today I'd like to talk to you about ships.
And we like to look at all the ships in the world.
There's a lot of them.
And what we have is a visualization
here that shows you a wide range of what's
out there, about all these boats moving around.
They're pretty cool.
Lots of fun stuff going on.
But what this really is, is a nice visualization
of the economic activity of our entire world.
So this is really a global scale problem to look at,
and what we've got are each individual ship making up
just a small fraction of this visualization.
The ship starts off with a red at the newest point where
it is, and then over time it fades out to a blue color,
and it gives you this nice, streaky pattern that shows you
where the shipping lanes are, where the people run around
in circles fishing, where people are doing all of that business
that they do on ships.
It really is an indicator of what's
going on in each particular environment.
And what we're getting here is the messages
are actually being broadcast by the ships themselves.
And we're picking this up with a satellite going over the top.
And a company called SpaceQuest has been kind enough
to lend us the data to do this example with,
and they're actually collecting these ship positions
as they fly over, and they're reporting them back to us.
And what we'd like to do is, get millions of points
together and look at the overall patterns and trends
to see what's happening in our world here.
And it's a lot of data, and if you just
work on a small laptop in your own little environment,
it gets very challenging, and your laptop
melts under the load of all this,
and you don't get very far.
So now, we're looking at this great visualization
here, Francesc, that we've built.
How did we get here to the point where
we have this amazing ship traffic going on?
How do we build something to get from that raw data
to this visualization?
FRANCESC CAMPOY FLORES: Yeah.
So that's actually a pretty complex problem to solve,
and we ended up creating this diagram here
where we can see all of the different companies
that we ended up using.
So we have the ships sending data through their GPS, the GPS
data, and then we get that huge amount
of data, lots of points for lots of ships.
And we want to extract information off that.
We decided to use BigQuery because you can actually
extract information really easily, and iterate very fast,
and just have fun, and find cool data about it
from huge amount of data.
But we had a little problem-- that we
had to upload something which is made of columns.
And actually the data we get from the ships
is not at all made of columns or at least doesn't look like it.
It's pretty cryptic.
So we had to decode it first.
So how did we do that?
You actually did it.
KURT SCHWEHR: Yeah.
So definitely, it's not just encoded, it's armored.
So this data comes out in a way that
is completely unreadable to a normal human being.
But I had a C++ library that I wrote over the last couple
years that actually just takes that data and converts it
into a CSV column format that's pretty manageable.
And then I wrote a little Python wraparound
that to make it a little bit easier to call.
That's great, but you can't run that
on an App Engine, which is where we're trying to get to.
So what we did instead was we brought up a Compute Engine
instance with a little Linux virtual machine.
And right there, we can run the C++ code that I wrote,
and it then takes this sort of cron job style,
takes data in from cloud storage,
turns into a nice column formatted text format,
and then loads it right back into cloud storage
into a different bucket where [INAUDIBLE].
FRANCESC CAMPOY FLORES: So it ends up a CSV file pretty much
on--
KURT SCHWEHR: Exactly.
FRANCESC CAMPOY FLORES: --cloud storage.
And then once we have that, loading that data into BigQuery
it's really simple.
Just have to do a REST code and just say, OK.
There's this new file.
I want you to load the data set.
And now we can actually start having fun and iterating really
fast and finding cool data.
So we could find faulty GPS signals or ships that
were fishing and all of that really-- not really
simply, because this is actually a pretty complicated problem--
but very fast.
You can actually just write your query, run it,
and in a matter of seconds you can get that response.
Then to show them the map, what we had to do
is getting that through App Engine app
and encode it somehow.
So we decided to encode it in binary.
But that means that every single time that there was query
coming from the browser the App Engine app
could query BigQuery, get that information,
and encode it and send it to a browser.
KURT SCHWEHR: So that was pretty fast, right?
FRANCESC CAMPOY FLORES: Well, that's
around in the tens of seconds every single time.
BigQuery is really fast actually,
when you think about it.
Considering the huge amount of data that we have,
tens of seconds is actually pretty good.
But it's definitely not fast enough for a web application,
definitely.
So that's why we added this intermediary storage, which
is Datastore.
So regularly the App Engine app could go to BigQuery,
up turn the latest data, and store it in Datastore.
And now every time that you get a query, instead
of going to BigQuery, we could just
get the data from Datastore.
So that makes it faster.
KURT SCHWEHR: But that's not fast enough?
I can see that.
Really?
You don't think that's fast enough.
FRANCESC CAMPOY FLORES: No.
It's fast, but not fast enough.
It's going to take less than 1 second,
but still around 1 second.
And it's taking long because actually when
we get the data from Datastore, we still
have to encode it into binary format,
and actually that's slightly complicated to do.
So what we decided to do on top of the Datastore
is we get data from Datastore, we encode it into binary,
and then we store it in memcache.
And now every time that we have a query, what happens--
KURT SCHWEHR: Is that fast?
FRANCESC CAMPOY FLORES: --is we get directly,
we get just a blob of binary data from memcache,
and we send it back through the HTTP connection.
KURT SCHWEHR: Now that's fast enough, right?
FRANCESC CAMPOY FLORES: Now, that's fast enough.
Yes.
KURT SCHWEHR: So that's great, but why
did you have to go with binary data
when most other web applications you use just an ASCII JSON?
FRANCESC CAMPOY FLORES: Yes.
So that's a good question.
We started JSON, actually.
And we realized that we had this problem where we're
sending a lot of numbers, latitude, longitude,
everything are numbers pretty much.
And we're sending them in JSON, which
means that we have the number.
We encode it in JSON, therefore text.
And then it gets to the browser, and the browser
has to decode it from text back to a number.
And actually that decoding encoding, it takes time.
It takes CPU time.
So what we did was just decide to encode everything in binary
and use binary arrays in JavaScript, which
means that now we can actually get that binary code that we
get directly from the HTTP response, put it into memory,
and just interpret it as a binary array, an array
of floats in our case, and just use that directly as data.
So you skip that encoding and decoding part.
KURT SCHWEHR: And this is why we have millions
of points in the browser so quickly.
FRANCESC CAMPOY FLORES: Yes.
That's why you can actually go through the application
and just pan around or shift and decide,
I want to see the data for this day,
and you won't have to wait for 10 seconds.
This day will just appear automatically instantly.
KURT SCHWEHR: Great.
FRANCESC CAMPOY FLORES: What was the part
that you learned-- what was the most interesting
for you out of this project?
KURT SCHWEHR: For me, this was my first time working
in the cloud kind of architecture,
and I was worried that I would have
to wait for my team members to work
on different parts of the system.
And the neat thing is that I just got to work on my pieces
and all these systems let me decouple and work
in my particular favorite language, which is Python.
I'm most comfortable there.
While you, for instance, were in Go, and I didn't have to one,
worry about the fact you were working in Go,
and I didn't have to wait for you.
So for me that was pretty exciting.
How did we pass data back and forth that made that so easy?
FRANCESC CAMPOY FLORES: Yes.
So since we had different independent components,
so we had these data loggers and the data transformation
and the App Engine app, and all that.
We're completely independent and communicating only
through Datastore and just passing the data around
and using task queues.
And task queues to let the other part of the application
know that there's something to do about it.
And that is actually one of the things
that I learned and I discovered that it was really
powerful that using task queues, it
allows you to have a very robust system pretty
much without doing much.
Because you know that you're going to try something,
and even if your program is faulty
and you have some bugs, things that happen sometimes,
you won't lose that data.
You will actually just fail and send back a message saying,
oops.
Sorry.
And it will be ultimately we'll try later on.
So you know that that data will never be lost.
KURT SCHWEHR: So task queue will pick
back up when your code recovers from--
FRANCESC CAMPOY FLORES: Yes.
You can, of course, change the settings to add more backup
time or whatever.
But the default, actually we're using the default settings,
and it works perfectly for us.
And then the second part, for me, well I'm a gopher,
so of course, I use Go for App Engine.
And I found out that using Go on the App Engine part
was actually a big win because most
of the time on the cloud, what you're doing
is waiting for things to happen or doing a lot of things all
of a sudden very, very fast.
So the fact that you can actually do all of those things
very fast at the same using concurrency,
it's something really important.
And doing it in Go is actually really easy.
You just have to add some Go routines and different channels
and four or five lines of code, you
get something that runs concurrently
and you get really big win in efficiency.
KURT SCHWEHR: This is what let us
scale when we start loading huge blocks of stuff
in multiple [? browsers. ?]
FRANCESC CAMPOY FLORES: Yeah.
Especially when you have to load a lot of different things
from Datastore, the fact that you can load all of them
at the same time and just wait for the slowest one, that's
a big win rather than doing sequentially.
And if you have 1,000, well, you have to wait for 1,000 times n,
so, yeah.
KURT SCHWEHR: So this is pretty exciting.
So now that we have this new architecture,
what do we need to do next to this
or where are we going to be going with it?
FRANCESC CAMPOY FLORES: Yeah.
So I think that the architecture is pretty much ready
to be used.
And so the cool thing now is that we
could consider using this same architecture for very
different applications.
So the same way you are actually showing here ships on a map,
you could also use it to track whales in the ocean,
or you could use it to track cars
on the highways, or anything you can think about.
Or even not related to Maps, you could actually
use the same architecture as some way
to filter huge amounts of data and extract
some information that is something really small,
and it's like very valuable information
out of a huge amount of data.
So yeah.
I think that the next step is just
find out what's the new cool application
to go with this architecture.
KURT SCHWEHR: Great.
FRANCESC CAMPOY FLORES: So also, if you
are interested in knowing more about this architecture
and how we built it, and what we did with it,
we gave a talk in the last edition of Google I/O
here in San Francisco, and you can find the link here
in the description.
So look at it.
Have a look and enjoy it.
KURT SCHWEHR: And just so you know,
we've released the decoding library, LIB IAS for the ship
messages if you are interested in playing with that,
and we hope to be working through the rest
of open sourcing of our little [INAUDIBLE].
FRANCESC CAMPOY FLORES: Yeah.
We we'll definitely open source on the App Engine app
code, and probably the data logger, and all the rest.
So thanks for watching, and keep an eye on Google Developers
Live channel for more videos about our projects
using different Google products.