ANAGHA DIWANJI: In this video, we'll
show you how to detect anomalies using Streaming Analytics
and AI.
My name is Anagha Diwanji.
And my colleague Prerna Bellara and I
are solution engineers for Google Cloud.
Many companies are dealing with the proliferation
of data and difficulty in making sense of that data
at the speed required to make it useful.
This solution aims at sifting through the mass of data
to the events that need additional attention--
those outliers that indicate that something is not
going as expected.
Now, what is anomaly detection?
Spotting the unusual, catching fraud,
and discovering strange activity.
All of this sounds like Sherlock Holmes solving a mystery.
That's what anomaly detection is, except we may not
have a clear suspect to search for,
and you may not even know what the crime is.
When you see this example, do you spot any anomalies?
First thought, it's fairly easy to see the large spike appears
to be an outlier.
But what if you have a large sample of data?
Now the perception changes, and what appeared to be an anomaly
turns out to be part of a regular and familiar pattern.
In this, the anomalous behavior is not the fact
that there are spikes but the observation
that their frequency fluctuates during an episode.
The highlighted region denotes an anomaly
because the same low value exists
for an abnormally long time.
Anomalies are not defined by their own characteristics
but in contrast to what is normal.
The speed with which businesses are able to identify anomalies
impacts their options for response.
For example, not detecting a fraudulent financial
transaction immediately and instead within hours
or even days often results in writing off the financial loss.
On the other hand, finding the anomalous transaction
in seconds allows for the invalidation
of the transaction and corrective actions
to prevent future fraud.
Similarly, by detecting anomalies
in industrial equipment, manufacturers
can predict and prevent catastrophic failures
that could cause capital and human loss by initiating
proactive equipment shutdown and preventative maintenance.
Likewise for retail, we can see this
being used to help manage inventory, detect
fraud, and track user behavior patterns.
Another example of an industry application
is a telco company doing subscriber analytics
and specifically focusing on detecting anonymous wireless
traffic, real-time fraud detection on call and SIM,
and user behavior patterns.
And across all these industries, cybersecurity
is top of mind for most customers because
of economic costs of intrusion and security attacks.
Let's talk about how to design a system for intrusion detection.
Let's deep dive into our story for today.
In our story, we have Stacy, who's a data analyst.
Her main role is to manage and monitor
the network for a large corporation.
She needs to ensure everything is running smoothly.
And if there's an outage or unusual behavior,
she needs to report it and take corrective action.
Stacy's company currently has a hard time
anticipating network traffic changes,
which creates impediments in identifying
cybersecurity attacks.
They want to catch unusual behavior and security attacks
in real time and take action.
Leveraging Google Cloud Streaming service,
Stacy can monitor the network, analyze the traffic,
and now has opportunity to review and take
action such as file a ticket in near real time.
When anomalies are found, the solution automatically
sends an alert to Stacy and updates network health
dashboards.
Now, let's see an overview of how
Stacy implemented anomaly detection in Google Cloud
environment.
There's four major components to this--
ingest, extract and train, predict, and alert.
The first step is ingesting unlabeled streaming data
from all the sources.
Google Cloud Pub/Sub and Google Cloud Storage
make it easy to ingest streaming data.
Next, we transform and store this data in Google BigQuery,
extract features using Dataflow, and train the model
with just some SQL.
After training this data, we use Dataflow
to allow for online real-time reporting and highlighting
outliers.
But it isn't just enough to detect anomalies and monitor
this data.
The key is the action we take.
BI tools, such as Tableau and Looker,
allow you to not only view this information
but take appropriate action such as informing the security team.
This was just a brief overview.
Now let's take a closer look at each of these four components.
My colleague Prerna Bellara will take it forward.
PRERNA BELLARA: Let's assume I'm Stacy, an analyst tasked
with solving this business problem that we discussed
earlier.
First, I'll look into the GitHub repository
to understand all of the four steps mentioned earlier.
The first step, you might recall,
would be to ingest the data.
Google Cloud's Pub/Sub topic can help us
with the streaming ingestion.
In order to do so, we also need to simulate production data
to train our model.
The photo on the right shows the schema
for the model, which resembles a NetFlow schema.
We notice about 250,000 events per second in my production
environment, which comes up to about 20
terabytes of data from heterogeneous sources
coming through a Pub/Sub topic, some of which will be labeled,
some maybe unlabeled.
Let's look at the next step.
Now that we've set up the ingestion,
the next question is, how do we ensure
the solution is scalable?
If you recall, we were generating or anticipating
250,000 events per second.
We need to aggregate this data.
This ties to the second step, which is extract and train.
Using Dataflow [? BIM ?] and its [INAUDIBLE] transform
such as minimum, maximum, and count,
we can ingest and aggregate these logs or events,
thereby reducing the number of elements
or the amount or volume of elements from 11 gigabytes
to 1.03 gigabytes, as you can see.
Additionally, Dataflow [? BIM ?] allows us to autoscale, thereby
generating or helping us build a pipeline that
is production ready.
We will use [? BQMS ?] k-means clustering model
to train this extracted data.
We've decided to use k-means as a--
since it's a common algorithm used for unsupervised machine
learning.
As you can see, I wrote the SQL query to create the model,
ran the command, and in 15 minutes we have a model.
Configuration is easy.
Simply mention the columns I wanted as features
and set the number of clusters.
This is great, because machine learning requires some trial
and error, and the quick turnaround time
helps me to play with the configuration
and determine the most optimal combination for my use case.
Another great feature for an analyst is the scheduled query.
I don't want to just create a model one time and leave it.
I want it to learn continuously and be updated with the latest
patterns.
Using a scheduled query and a [? stored ?] procedure,
I can rerun this query in a timely manner,
thereby keeping my model updated.
Additionally, once the module is created
we will calculate the normalized distance from each centroid
or grouping to determine if the input that we received
is too far from the cluster, thereby deeming it
as an outlier.
Now that the model is ready, it's time to detect outliers.
How do we do this?
The answer is Dataflow.
We pass the normalized data in the pipeline as an input
and reuse this data to calculate how far our input
data is from the center.
If it meets a distance threshold,
we don't throw them as an outlier--
we store them as an outlier in the BigQuery table.
The result, we are now notified of the outlier
and can provide little to no downtime.
Now that we have enabled real-time detection,
it's time to view our results, and most importantly,
take appropriate action.
We will leverage Google's Looker Dashboard to do this.
Let's take a look.
As you can see, in this Looker Dashboard,
I can see the number of events, the source,
whether it was an anomaly, as well as the bytes.
There is another column over here called Actions,
which allows me to take appropriate action.
I can either create a Jira ticket, text a security
consultant, or email the security team, thereby
notifying my team of the right action.
You can also use any other third-party tool
that is supported by Google Cloud for your reporting needs.
In summary, we've built this in a short time, processed
over 250,000 events using Dataflow.
We can consume all of this 20 terabytes of daily data
from data sources.
We trained and evaluated 1 terabyte data
in 15 minutes using SQL.
We found outliers in near real time
and took appropriate action.
We've covered a lot of ground.
For more information on how to implement
this in your environment, please check out the GitHub link.