JOANNA SMITH: Hi.
I'm Joanna Smith, and today we'll
look at how Stream Analytics from a Google Cloud
helps bring the real-time e-commerce experience to life.
Real-time data is unique in that it
provides an opportunity for businesses
to react without delay.
Valuable events, such as an online shopper clicking
on an item's details, can trigger
dynamic and immediate experiences
on a website or mobile app.
The opportunity to generate value around this data type
is only growing.
In fact, the International Data Corporation
estimates that within five years, more than a quarter
of all data globally will be real-time in nature.
But for an application to react to real-time data,
it's crucial to have data analysis, insights,
and actions that occur without human intervention.
Stream Analytics does just that, offering automated
real-time data analysis.
Let's take a look.
Here, we've got our e-commerce site, Newkicks.
And on the right throughout this demo,
you'll see some context on what's
happening behind the JavaScript on the GCP platform.
Let's take a look at how Stream Analytics
can improve the Newkicks shopping experience
and drive sales.
We've used Pub/Sub, Dataflow, BigQuery and Bigtable
to create the data streams that will allow Newkicks
to make their data actionable within milliseconds
of being generated.
For example, the hot item flag is created in real time,
as Pub/Sub ingests user clicks throughout the site
to be processed by Dataflow, allowing Newkicks to determine
which products are currently being
viewed by at least 50 people.
When our shopper clicks on an item,
they see that there are 62 people looking
at these exact shoes at this moment.
Our shopper adds the shoes to their cart
and immediately sees a low inventory alert
powered by data from the inventory stream,
and based on real-time stock availability information.
To reliably offer this type of shopping experience,
even during busy times, Newkicks relies on Pub/Sub's ability
to ingest millions of events per second,
combined with Dataflow's exactly-once processing
with fault tolerance.
If a user misses their opportunity window
and the shoes become out of stock,
real-time AI can turn a frustrating user moment
into a seamless exchange, with a personalized product
replacement suggestion.
In parallel, this inventory insight
also enables Newkicks to trigger a back-end stock replenish
process.
Our shopper liked the alternative shoes,
and when checking out is offered a chance to pick them up
in a nearby store.
Here Newkicks uses real time inventory and transactional
data from the local stores to offer the user a chance
to get their shoes faster.
Real-time data enabled Newkicks to act, and react,
to create a positive purchase experience.
That's one more happy customer.
Now let's take a look at how this works within the Google
Cloud platform.
Real-time data is generated based on certain events,
like user interactions.
So the first step is to configure an event
to push to Pub/Sub, which will serve
as a trigger for a particular data stream.
To create that data stream, start
by defining a topic in Pub/Sub.
With Stackdriver, we can see high-level summaries
for our Pub/Sub activity.
Here we have the rate of incoming messages
for all three of Newkicks' data streams.
The message volume comes in bursts,
based on the day of the week or the time of day.
Our demo peaks at around 200-300 messages per second,
but could scale to thousands, or even
millions of events per second without anyone
having to change a thing.
No planning necessary, because Pub/Sub
is a fully-managed service that automatically
scales to what's needed.
Another interesting summary is the oldest,
unacknowledged message within any topic.
Here we can see they are being picked up
just a few seconds after being pushed into Pub/Sub.
And with this monitoring system, we
can set up alerts to send out notifications
if values extend beyond our SLOs.
After the data is ingested, it needs to be processed.
And this is where Dataflow takes over.
By establishing Dataflow pipelines,
those initial events can trigger application behaviors, business
logic, or even additional processing steps.
These pipelines run on the flexible, open source,
Apache Beam SDK.
They were built programmatically,
using Dataflow's wide variety of supported programming
languages, including Java, Python, and the recent addition
of SQL, with more set to come.
In Stackdriver we can see graphs of the rate of processing
for Dataflow, but let's take a deeper look
at how we actually process that data.
In the Dataflow monitoring graph,
we can explore each of the three streaming
pipelines we saw for Newkicks-- click, inventory,
and transactional.
On this page, we see various statistics for the pipeline.
CPU, throughput, et cetera.
But for a closer look at what the pipeline is actually doing,
let's change tabs.
Now we can see the processing pipeline.
There are three sources of data, each being processed
independently, before joining together
for cross-stream analytics.
From data validation and correction
to flattening and formatting, all of this
is happening in real time, rather than
through a series of batch-processing steps.
Interestingly, some pipelines pull data from a Pub/Sub sink
and land right back in Pub/Sub for the next stage
of processing.
This is one of the core promises of Stream Analytics,
joining the data analytics processing system directly
to the operational systems.
Once processed, our data moves to either Bigtable,
for low-latency, high-concurrency availability
back to the Newkicks website, or to BigQuery,
for deeper analytics.
We saw how Newkicks applied these data streams
within their website.
So now let's take a look at what happens in BigQuery.
In the BigQuery user interface, the data
sets this user can access are listed on the left.
And we can choose to look at the click-stream table
under the retail data site.
We can see it has about 118 million rows of data,
and that it is auto-partitioned by time
using the timestamp column.
Additionally, right here within the BigQuery web UI,
users have the ability to write and run
SQL queries against both Dataflow, for further ease
and flexibility of pipeline creation, and BigQuery.
We've already seen Dataflow in action.
So let's run a query against the fresh data in BigQuery,
say the last 10 minutes.
We can also run more complex queries
like this one, which merges our click-stream and transactional
data streams to output popular sales items sold
online in the last 10 minutes, versus in-store
in the last hour.
The power of Stream Analytics is that the entire process,
from the first event trigger to your application reacting,
is entirely hands off.
Once you set up your data stream,
the auto-scaling capabilities of Pub/Sub and Dataflow
help ensure that the entire interaction keeps working, even
at the busiest moments.
But when you do want to take a deeper look,
dashboards for real-time data and data
streams are still classic.
Let's take that closer look.
In Looker, we can see visualizations
for data analysts supporting real-time website activity.
For example, here we have our purchase conversion funnel.
This data is updating with an auto-refresh running
every minute.
Even though Stream Analytics occur in real time,
dashboards like this allow you to review performance
in order to glean insights and optimize for the future.
Looker consumes and aggregates real-time data
streams to help Newkicks determine things
like where in the purchasing process
a user is most likely to abandon,
or whether a hot item flags perform best
when set to display for products with greater than 50 viewers,
or greater than 100 viewers.
And that is how Steam Analytics from Google Cloud,
built on the auto-scaling infrastructure of Pub/Sub,
Dataflow and BigQuery, helps bring
to life the real-time e-commerce experience.
Check the documentation to learn more, and thanks for watching.