JOSH LIVNI: Thanks for watching, guys.
My name is Josh Livni, I'm on the Maps
Developer Relations team.
And special guest today, we have--
FELIPE HOFFA: Hello, everyone, I am Felipe Hoffa.
I'm with the Cloud Platform team, with the Big Data team,
specifically.
And specifically BigQuery, the tool that allows you to query
millions, billions of records in very few seconds.
JOSH LIVNI: Yeah, BigQuery is amazing.
And I was watching this video that you and Michael put on
last week about the new correlations, Pearson
correlations, that BigQuery magically
takes care of for you.
And it was a really great video.
Tell me a little bit more about what are correlations
and what you did.
FELIPE HOFFA: That was a very fun video to tape.
To get out if you can, we took 70 million flights across the
United States, across 10 years of story.
We were looking for correlations.
We were looking how to predict the future.
How to know if you're flying tomorrow from any airport in
the United States, how to predict how late your plane
will be depending on what happened today.
JOSH LIVNI: Pretty cool.
Pretty cool.
So I was watching the video.
You had the states that are most correlated with each
other, lots of nice graphs in matplotlib.
And then I was waiting for it, I was waiting for it--
no map.
I was very sad.
But luckily, I was able to track you down in the San
Francisco office, and in a matter of like an hour put
together a map.
So that's what we're talking about today is, let's say you
have a big data analysis you're doing.
You have some output, it fits into a nice spreadsheet.
Is it spatial data?
Why not go ahead and put a map on it?
So this is what we did with this blog post.
Fresh off the presses, Felipe and I, talking a little bit
about how one takes some output from BigQuery,
puts it on a map.
FELIPE HOFFA: We have up here, let's look at it.
What do you do here, Josh?
JOSH LIVNI: What do we have?
So we have a lot of little black circles on this map.
These are the airports that we have lots of data for over
this 10 year history.
And go ahead and click on, I flew out of Seattle last week.
Click on it.
FELIPE HOFFA: Let's see what's happening on Seattle.
So if I click here, I see a lot of circles-- big circles,
small circles, no circles.
What do they mean?
JOSH LIVNI: What do they mean?
Yeah, so basically what we're doing is resizing and
recoloring the circles based on the correlation of airport
delay departure between today and what we predict tomorrow.
So in this case for Seattle, it means that if I'm flying
out of Seattle tomorrow, what airport
should I look at today?
Probably Seattle, which kind of makes sense.
FELIPE HOFFA: There is a very high correlation for having a
[INAUDIBLE]
data point in this time series.
There is a very high correlation on what happened
yesterday in Seattle, predicting what is going to
happen today.
JOSH LIVNI: Well, not that high a correlation-- we're
like 0.47 here.
What if we change the seasons?
I know that there's sometimes more correlations in different
seasons for different airports.
FELIPE HOFFA: Whoa.
JOSH LIVNI: Airports, planes, weather, kind of makes sense.
Yeah, so now at 0.7, that's pretty good correlation.
Again, it just kind of makes sense-- if you want to know
what's in Seattle tomorrow, look at Seattle today.
Probably raining.
FELIPE HOFFA: That's a very good way to predict the
future, whatever happened today would happen tomorrow.
But sometimes there are better ways.
For example, let's fly to a different airport.
JOSH LIVNI: Yeah.
I don't know, where did you fly from recently?
FELIPE HOFFA: Let's call San Diego.
JOSH LIVNI: Okie doke.
FELIPE HOFFA: Let's go to San Diego.
What do we have here?
JOSH LIVNI: A very different pattern.
FELIPE HOFFA: Yes.
Correlation size is smaller.
But this is interesting.
The biggest correlation is not with San Diego
with itself but LA.
What's happening today in LA predicts better what's
happening in San Diego tomorrow.
JOSH LIVNI: Maybe that makes sense.
LA's kind of the hub down there.
So this is kind of interesting.
Small correlations distributed kind of evenly throughout the
country, unlike the West Coast, Seattle.
What happens if you pick a major hub?
FELIPE HOFFA: Let's go to the East Coast.
How about New York, JFK?
JOSH LIVNI: OK, so lots of stuff, bigger circles, bigger
correlations on the East Coast.
It's kind of interesting.
JFK's not even in the top five for predicting what JFK's
going to be like.
FELIPE HOFFA: That's very interesting.
If you want to know what is going to happen in JFK
tomorrow, during spring, look at what happened
in Nashville yesterday.
JOSH LIVNI: Nashville predicts JFK.
I love it.
So yeah, you can click around this map, kind of see some
distributions of how the different airports are
correlated with each other.
It's kind of interesting.
Change the seasons around, have some fun.
And this was super easy to put together.
Why don't we hop back to the blog post?
Just take a quick look at the code, and talk a little bit
about how it works.
FELIPE HOFFA: Please show me how you're doing this.
This is great.
I love it.
JOSH LIVNI: Well, of course, all of the code is
available on GitHub.
If you click through to the map, you can view the source.
But we also have a short gist down here showing the main
functions to recolor things.
It's pretty easy.
I have a method down at the bottom, the get icon one that
passed in the scale of the correlation, and a color to
color [? by, ?] say, red versus black.
Let's get back a circle.
It's pretty easy to draw circles on
a map, that's useful.
And then the other thing is just highlighting the
correlations.
Looping through all of the hard work that Felipe did to
give me the results of the correlations.
And then go ahead and just say for each airport get the
appropriate icon back.
Not super hard.
Not a lot of code to create a map like this.
We started from a template visualizing earthquakes that
we put a really basic article on that you guys can start
with if you have different size circles.
And more importantly, if you're doing some big data
analysis and there's some spatial aspect to the data, go
ahead and make a visualization that has a map with it when
you're finished.
FELIPE HOFFA: I love it.
Thank you very much for doing this, Josh.
JOSH LIVNI: Thank you for doing all
the hard work, Felipe.
My job was easy.
FELIPE HOFFA: Well, your job is awesome.
So anyway, remember, if you have spatial data and you like
it, put a map on it.
JOSH LIVNI: I love that.
Quotes to live by.
Felipe, thank you very much.
I'm Josh.
FELIPE HOFFA: And I'm Felipe Hoffa.
Goodbye.
JOSH LIVNI: Cheers.