[MUSIC PLAYING]
POLONG LIN: In this video, we will show you
how to build a serverless, real-time credit card fraud
detection solution using Google Cloud.
My name is Polong Lin, a developer advocate
for Google Cloud.
And together with Quantiphi, a GCP premier partner,
we co-developed a design pattern to help
you accelerate building a fraud detection
solution with Google Cloud.
ESHAN TYAGI: I am Eshan Tyagi.
I am a senior business analyst at Quantiphi.
PAVAN KATTAMURI: I'm Pavan Kattamuri.
And I'm a senior cloud engineer at Quantiphi.
POLONG LIN: Let's first talk about fraud detection
in the industry.
What does Quantiphi see?
ESHAN TYAGI: Fraud detection is a major concern
for many businesses and financial services.
Global losses from payment fraud have
tripled in the last 10 years.
In the UK alone, for every 100 pounds
spent using credit cards, 7.5 pence was lost to fraud.
PAVAN KATTAMURI: So the challenge
is to build and manage a real-time fraud detection
solution at scale.
POLONG LIN: So in this video, we will show you
how to build a real-time fraud detection solution broken down
into the following sections-- how to prepare the training
data in BigQuery; how to build the fraud detection
model directly within BigQuery using BigQuery ML;
how to build a pipeline to do real-time model
predictions using DataFlow; how to set up
alert-based notifications when fraud is detected using
Pub/Sub; and how to create interactive dashboards
for business and technical operations using Data Studio.
ESHAN TYAGI: And the code and instructions
to follow this design pattern are all linked
in the video description below.
POLONG LIN: Let's talk about preparing the training data
in BigQuery.
ESHAN TYAGI: To detect if a transaction is fraud or not,
you want to have information on the transaction
and the customer's information.
So normally, these historical records are in separate tables.
You may have one table for transactions and another
to keep track of your customers' demographic information.
But to make it easier to show you here,
we have combined both of the data sets
into a single table for training.
PAVAN KATTAMURI: This table is going
to be our training data, where each row is a transaction that
occurred in the past, containing details
of the transaction and some customer information.
Each column is a feature such as what times
the transaction occurred, what the merchant was, category,
and so on.
There is also a label feature called is_fraud,
which is a value of 1 or 0.
This is what the model will then try to predict--
a probability of fraud between 0 and 1.
The data here is simulated data, so you
can replace this with your own data for your needs.
POLONG LIN: Could you elaborate on your choice of BigQuery
here?
ESHAN TYAGI: It's an extremely scalable data warehouse.
It's fast and easy to use.
And we find that customers choose BigQuery
because you get the best of enterprise-grade data
storage, data analytics, machine learning, all in one place.
We have customers storing terabytes of data in BigQuery,
and they have the capability to train machine learning models
within BigQuery directly.
POLONG LIN: So speaking of machine learning,
the next step is building the machine learning model
to detect fraud using BigQuery ML.
Perhaps the magic of BigQuery ML is
that, without needing to move your data out of BigQuery,
using BigQuery ML, you can train and deploy machine
learning models directly using SQL.
And you don't have to procure or maintain extra infrastructure
because it uses BigQuery's built-in autoscaling
infrastructure to handle model training
and deployment at scale.
ESHAN TYAGI: And at Quantiphi, we
find our customers really enjoy being
able to use the democratized ML capabilities, which
allow analysts to run machine learning models using SQL.
So with the training data, we can
train a model using just SQL with BigQuery ML.
While there are several classification algorithms
available to choose from with BigQuery
ML, like logistic regression, deep neural networks,
and AutoML tables, we find that XGBoost models worked really
well with minimal hyperparameter tuning required,
making it easy to build good models quickly.
PAVAN KATTAMURI: So in the Create Model segment,
we give the model a name, like simplemodel, set the model
type to BOOSTED_TREE_CLASSIFIER for XGBoost,
set the input level calls to is_fraud as the level of 0
and 1 that we want to predict, and then select the training
data.
You can also adjust some hyperparameters
such as num_parallel_tree and max_iterations for example.
ESHAN TYAGI: We were able to train this model
within just 10 minutes over 1.3 million rows
and 10 features, which is really fast considering
there is no need to set up extra infrastructure
to train the models.
POLONG LIN: Now, I notice that you
called the model "simplemodel."
Could you elaborate on that a little bit?
ESHAN TYAGI: Yes.
In practice, just looking at the transactional data
and the customer's demographics, like what country they are in
and their age, is actually very likely
to be insufficient to produce a strong performing fraud
detection model.
POLONG LIN: So what would you recommend, then,
to build a good fraud detection model?
ESHAN TYAGI: Well, there are many ways.
One way which we can show in this design pattern
is to do more feature engineering
by also looking at the customer's historical activity.
If someone typically spends on average
$100 per month on their credit card,
and this transaction shows you are spending $2,000,
then that might be unusual.
So you would want the model to pick up on this
by adding in relevant features.
PAVAN KATTAMURI: And this is perhaps just a starting point.
In reality, you may want to have a lot more features,
and you may want to improve your model in other ways,
like model selection, hyperparameter tuning,
and model ensembling, which had all possible with BigQuery ML.
Because there are so many approaches depending
on the data available and their exact scenario, in this design
pattern, we just show you how you
can get started with a few extra aggregate features
of the customer activity, such as how long it's
been since their last transaction;
the average spent on the credit card per week or per month;
and also, the number of transactions of average
in the last 24 hours, just as examples of what you could do.
POLONG LIN: With the models built in BigQuery ML,
could you show us how you evaluate the models?
PAVAN KATTAMURI: You can see the evaluation metrics directly
on the BigQuery UI under the Evaluation tab.
You can see the log loss, ROC AUC, precision, recall,
accuracy, F1 score, and other visualizations as well.
To actually compare F1 score across models,
you can use SELECT * from and ML.EVALUATE and then the model
name.
You can do this for both the models
and combine the results using UNION ALL.
ESHAN TYAGI: To compare metrics, you
can use F1 score and ROC AUC, where
the higher the values are, the better the model performs.
For imbalanced data sets like in credit card fraud,
where you have mostly normal transactions,
then we recommend using the F1 score to compare the models.
And indeed, if you look at the model that
uses aggregate behavioral data, the F1 score
is 0.78 versus the simple model's F1 score of 0.56.
So it's improved dramatically by incorporating
the historical data.
Although there is room for model improvement
as discussed earlier, we'll just use the second model
to demonstrate how you can use a BigQuery ML
model like this one in a streaming pipeline
for real-time fraud detection.
POLONG LIN: With the model built,
let's talk about making predictions to detect fraud.
Now, for predictions, there is batch and online predictions.
Could you elaborate on the differences?
ESHAN TYAGI: Sure.
For batch predictions, this is when
you want to make predictions on lots of data in bulk
at a time, which you can do in BigQuery ML with SELECT *
from ML.PREDICT your model and the data you
want to make predictions on.
Batch predictions are useful in making
large amounts of predictions in a single shot,
and you can do this with BigQuery ML.
But they are not optimized for real-time inference,
where you may have lots of predictions
you need to do per second.
PAVAN KATTAMURI: So that's where online prediction comes in,
which is for real-time predictions
in low-latency situations for higher
frequencies of operations.
To do online predictions, you will
need to export and host a BigQuery ML
model on AI Platform.
POLONG LIN: Another way to export the model
for online predictions is, instead of hosting
the model on AI Platform as its own endpoint,
you can similarly export to Google Cloud Storage,
but serve it directly in a DataFlow pipeline.
This may be helpful if you want to optimize for lower
average latency, but it may also make model maintainability
a little bit more difficult.
So in this design pattern, we'll just
show you how the model is hosted as its own endpoint
on AI Platform for online predictions.
So Pavan, how does that work?
PAVAN KATTAMURI: You can easily do it via two steps.
Step one, export the model to a Cloud Storage location
by running bq extract command with the model
name and the destination location as arguments.
Step two, once it's in Cloud Storage,
you will need to create a model on AI Platform which
can hold one or more versions of the model.
So the next step is to create a model version
by running the gcloud ai-platform versions create
command, providing the GCS location of the previously
exported model from BigQuery ML.
ESHAN TYAGI: And then to make predictions,
you can send a request to the deployed model using
the gcloud ai-platform predict command by passing
in the transaction data.
POLONG LIN: So with the model hosted,
the next step is putting it together
into a pipeline that can handle transactions in real time,
make predictions, and then send notification alerts for fraud.
ESHAN TYAGI: To create the pipeline,
you can build a DataFlow pipeline
with the following elements.
First, it ingests new transactions
from a Pub/Sub subscription.
Pub/Sub is an event messaging service.
So the Pub/Sub subscription is waiting
for any new transactions to come in as essentially JSON
messages.
As a new transaction is ingested,
DataFlow then does a lookup on the customer's
historical activity to do some aggregations.
The lookups can be stored on something like Firestore, which
is a serverless NoSQL database.
With the combined data, DataFlow then invokes
the fraud detection model on AI Platform
and gets a response with the probability that it's fraud,
between 0 and 1.
The prediction outputs are then stored in a separate table
in BigQuery.
And in parallel, DataFlow also triggers a notification
to a Pub/Sub topic, which can then trigger
any downstream notifications.
POLONG LIN: When it comes to detecting fraud in real time,
what is the latency that you get with this pipeline
from the incoming transaction to the notification?
And what do you typically see your customers
asking for in terms of their required latency?
ESHAN TYAGI: For some of our customers,
the detection has to typically occur within seconds for it
to be effective.
In this pipeline, the latency between when a transaction
occurs and when the fraud notification is sent
is around two to eight seconds, which
falls in line with what we see institutions asking for.
POLONG LIN: And what if customers need
much lower latency?
ESHAN TYAGI: The first way is, instead
of hosting the model as an endpoint on AI Platform,
you can see decreases in latency by exporting
the model in Google Cloud Storage
and serve the predictions directly from DataFlow,
as mentioned earlier.
However, if you still want to keep the model endpoint
as is on AI Platform, then you can
try to upgrade the machine type, increasing the number of nodes
for model serving, or increasing the quota limit on AI Platform
predictions.
POLONG LIN: And what about DataFlow?
PAVAN KATTAMURI: We find that [INAUDIBLE] is really easy
to use and scalable.
It can autoscale on its own, so you
don't need to worry about the streaming infrastructure needed
to run these patterns.
POLONG LIN: Could you walk us through a demo?
PAVAN KATTAMURI: Certainly.
Let me show you how this works on Google Cloud Platform,
starting with ingesting the transactions.
First, let's verify that we have a live stream
of new messages coming in via Pub/Sub
as real-time transactions.
You can view some of these transactions
by clicking on New Messages here.
As you can see, each message holds transaction details
like the transaction date, amount, merchant information,
and user demographics required for the ML model
to do a prediction.
But where is the aggregate data of customer activity?
Since aggregate data relies on historical activity,
it should be computed on the fly in real time.
Let's check that on Firestore.
We are using Firestore to store the users' recent transactions
and look these up very quickly.
To compute attributes on their recent activity,
the DataFlow pipeline will run these aggregations
for the user, for example, calculating volume
or transaction frequency, and then
add that to the raw transaction data we saw from Pub/Sub.
At the same time, DataFlow will also
update this Firestore record to only maintain
the most recent transactions required
for future computations.
So let's take a look at the DataFlow pipeline.
This is our streaming DataFlow pipeline,
consuming real-time transactions from Pub/Sub,
converting them into JSON format,
doing a lookup to Firestore to fetch the historical activity,
computes aggregations, invokes ML models
deployed on AI Platform to get the probability of fraud.
And once the prediction is done, it
will send alert notifications automatically to Pub/Sub
so then downstream processes can be integrated.
At the same time, it also stores output predictions
from the fraud detection model into a BigQuery table.
So how does the output prediction look like?
The column is_fraud_model_w_aggregates
is the predicted output with value 1
being a fraud transaction and 0 being a genuine one.
POLONG LIN: Notice how the number of rows
here increases from 12,000 to almost 13,000
as new transactions are processed and then
stored here in BigQuery.
So Eshan, what's the next step?
ESHAN TYAGI: Keeping an eye on how everything
is working using dashboards is really useful at this point.
So first, here's a dashboard we created for business users.
POLONG LIN: So as a business user,
I may ask a question of, where are all the fraud
transactions occurring?
PAVAN KATTAMURI: You can see it broken down by merchant type.
You can also see it broken down by geography.
And you can even click on specific parts
on the map and filter for transactions
only in the state of Illinois.
POLONG LIN: And another question may
be, how is the rate of fraud detection changing over time?
ESHAN TYAGI: In these graphs, you
can see the fraudulent transactions
over a period of time and identify the average fraud
percentage on specific days.
Next, we also created this dashboard for a technical team
to monitor the health of the pipeline,
track the incoming transactions, monitor the output
from individual components, and supervise the output
notification channel.
POLONG LIN: And what data is this dashboard built off of?
ESHAN TYAGI: This dashboard uses data
from Cloud Operations, which collects metrics, logs,
and traces across Google Cloud and other applications.
By visualizing this data in a dashboard,
it allows the technical user to monitor model latency
as well as the volume of fraud notifications going
through DataFlow and Pub/Sub.
PAVAN KATTAMURI: If there are unusual peaks,
or any line seems to be dropping to 0 when they shouldn't, then
that would tell us that something perhaps
is wrong that the technical team needs to address.
POLONG LIN: So if you'd like to know more about the solution
or want to see the code, follow the links
in the video description to the public code repository.
And with that, now you know how to build a real-time, credit
card fraud detection solution on Google Cloud.
Thank you, Quantiphi, for your expertise as a GCP partner.
And thank you all for watching.