[MUSIC PLAYING]
[BUZZER]
STEPHANIE WONG: Get out of here.
JONATHAN CHAM: Oh!
What are you doing?
STEPHANIE WONG: What are you doing?
JONATHAN CHAM: I'm practicing.
I've got an away game to go to.
STEPHANIE WONG: Oh, away game for the Warriors?
JONATHAN CHAM: Yeah!
STEPHANIE WONG: Well, I've got something
that might help you maximize your experience at the airport.
JONATHAN CHAM: Oh, tell me more about that.
STEPHANIE WONG: So today we're talking about Google Cloud's
IoT Core pushing airlines' data into Pub/Sub, Dataflow,
and BigQuery.
JONATHAN CHAM: Interesting.
So what are you collecting, and what are you
telling you about that?
STEPHANIE WONG: So we're talking about customer events data,
like whether they checked bags, how much time you have
in between trips so that you could possibly get a drink
or upgrade your seats.
JONATHAN CHAM: Nice.
I would love that.
I spend a lot of time at the airport.
STEPHANIE WONG: Yeah, well, hopefully, your Dataflow game
is better than your basketball game 'cause
I don't know what that was.
JONATHAN CHAM: Wow.
Whoa, whoa!
Come on now.
All right.
Let's see it.
STEPHANIE WONG: All right.
Five-minute demo starting now.
[MUSIC PLAYING]
So before I jump into the actual demo,
I just wanted to cover sort of the architecture
that we'll be representing through this demo today.
We are generating random data through Compute Engine
and feeding it into a Pub/Sub topic, which we will then
use Cloud Dataflow to stream it into BigQuery in real time.
Let's go ahead and just jump into the actual demo.
So the first thing I want to show
about how I got this set up is I'm
going to head over to Google Cloud Storage, which
I mentioned is how we did the device simulations
and also where we created some folders for our Dataflow
pipeline.
So we created a bucket titled IoT Airlines v1 data.
We created two folders within this bucket.
And so that's going to include some information
about our device simulators here.
And we've also created another bucket
for our Dataflow pipeline.
So for our Dataflow pipeline, we've
also included some files here.
And we created two folders for staging and temp.
It includes spinning up and shutting down,
Google Compute Engine instances, and the access
these GSA buckets for both I/O and temporary file staging.
Now, we've also set up a Pub/Sub subscription.
So the first thing we do here is head over to Pub/Sub
and you see here we've created a topic
and this topic is titled "Airline Messages."
The type of subscription we created under this topic
is a pool subscription.
And we've named it Dataflow here.
So you can see here a publisher application, in this case,
would be our IoT simulators.
They create a topic in Pub/Sub, which we've
titled "Airline Messages."
And that's a service that sends messages to the topic.
Now, going back a little bit, I'm going to show you IoT Core
because this is also the component in which we register
our IoT simulators.
What we've done here is we've registered this right here
called IoT airlines registry.
We've essentially cloned the git repository,
where we have java files.
And we've compiled the application here,
upload a JAR file to the storage bucket that we showed earlier.
This is our registry ID.
It's going to accept in the form of MQTT and HTTP protocols
in the region US central one.
We have Pub/Sub set up for this IoT device registration,
which is the one we created earlier, Airline Messages.
Anytime you register a device to connect to IoT Core,
you just need to register it first with Cloud IoT.
It consists of adding a device to a collection, which
is this registry, and defining some
of the essential properties, like we've done here.
The last thing I'm going to show you
is our Dataflow pipeline that we've created.
What we've done to build this Dataflow pipeline
is we've included SDK client libraries for Java and Python,
plus REST APIs.
And we've essentially cloned the git repository.
We've updated some files there.
And then we deployed it directly to Dataflow here.
I'll show you here what the pipeline we've created
looks like.
So essentially, it's reading messages,
applying window and techniques for streaming information
from our IoT devices, and it's creating a table row and entity
conversion so that we can actually
push it into other sources like BigQuery and Datastore.
OK.
Now that we've had this whole pipeline set up, let's go ahead
into the last step and show you what it looks like in terms
of the output in BigQuery.
We have our data, event_data.
And this is where all the data coming out of Dataflow
gets pushed into this table.
So we've created a data set called IoT Airlines
v2 with our table events_data.
And so you can see here this is what the schema looks like it.
It includes customer ID, the event type,
things like customer name.
We also have how many bags are checked,
how many seats are assigned under their name, how
long the trip was, what the origin is,
what the destination.
Let's just do an example really quickly
of how we can compose a query on this data set.
Certain routes might be more likely than others
for certain upsells, like, for example,
if somebody's flying out at 6:00 in the morning on a Monday,
maybe its a business trip.
You can offer them to buy Wi-Fi.
You can also upsell them if a customer's carrying bags
or has a connecting flight, for example.
Maybe they want to upgrade from coach to first class.
So let's go ahead and just do an example query of just pulling
customer emails, IDs, and see if there's
anybody who has actually done a bag check, for example, so we
can potentially upgrade them.
So let's go ahead and open this.
I have my example query here.
And you just input it directly into this query box
here and run the query.
We were selecting customer email, ID from this data set
and table event_data.
And we're looking for customers who have more than one bag,
and the trip leg is more than one.
And you can see here this is just fake generated data.
But in a real-case scenario, we might have a different list
of customers and their IDs so that we can immediately
email them and offer them another service.
JONATHAN CHAM: Whoa, that was amazing.
Thanks for sharing all that information.
Now I know when's the best time to get upgraded because I'm
tired of sitting in coach.
I need to sit first class.
STEPHANIE WONG: All right.
Well, here's to riding in style.
Thanks again, everyone, for watching Take5 with us.
Please comment, like, and subscribe below.
[MUSIC PLAYING]