(electronic music)
- Coming up on this special edition of How We Built It,
we're joined by a leading fraud detection company in Brazil,
Clearsale, helping customers across industries
verify an average of half a million transactions daily.
Now, we're going to look at how they're looking to modernize
their operational analytics data platform
to include Azure Synapse so they can expand
their machine learning analytics
for anomaly detection and operate at greater scale.
So we're joined today by Jelther Goncalves,
who's a Data Engineer at Clearsale, welcome.
- Thanks Jeremy, it's great to be on the show.
- Thanks for joining us today.
And by the way, before we get into this,
just a quick recap, if you're new to Azure Synapse.
This is Microsoft's limitless analytics platform
that really brings together data warehousing capabilities
along with big data processing into a single service.
And you can learn more at aka.ms/mechanicssynapse.
So Jelther, how do you actually do all of this work now?
- So, today we have a well-defined process
that catches 99% of the cases.
It works like this: during an online purchase
Clearsale is already on action on the customer's website.
We collect the information about the purchase
and the behavior of the client.
This last we call a fingerprint.
One example of behavior information
is the number of credit cards that the client tried to use.
When this information arrives at Clearsale,
we run machine learning models, business rules
and historical data from different industries
to check if the purchase is fraudulent or not.
All this runs in a matter of seconds.
For the other 1% of transactions that look suspicious,
we'll have a manual analysis.
This is led by our fraud experts at Clearsale.
They will investigate deeper and if needed
they will contact the person trying to make the purchase
to prevent the fraud.
- Okay, so to date your solution,
has it mainly been on-premises servers that you're running?
- Yes, but we're moving our operations
to Azure's Cloud Effect.
Today we have different SQL Server on-premises instances
that run different jobs.
In the past when the number of transactions increased,
we had to replicate data to new SQL Server instances
to take care of it.
You can see this is very manual and time consuming.
On the analytics side,
since this information was very transactional and spread,
our analysts had to do a lot of engineering work
to transform and index the data for their needs.
- So what have been some of the challenges
then for this approach?
- The main is scale.
Since our data is doubling every two years,
we weren't able to process everything
with the performance we wanted.
To support some of our work we built batch processes
for specific data sets to run outside work hours
to avoid impact or response time.
Also fraud is a very dynamic event.
A leak of data from one industry could affect other,
and being able to correlate data with a good performance
is essential for us.
- And being able to analyze the entirety of the data's
also important I think for analytics.
So, what approach then are you taking
to modernize your data architecture?
- We've taken some temporary steps to help with that.
For our transactional operations, we've deployed
Azure SQL Server Hyperscale databases with replicas
that receive data from our own premises
through custom pipelines.
The data flow works like this:
we read specific data sets from our on-premises databases,
send them to Azure Event Hubs
and Azure Functions ingests this data into Hyperscale.
From this point we run machine learning models
and business rules on a scalable database
to score the fraud risk of the transactions.
Some of them will score low and be successful,
other will trigger the manual analysis
for further investigation.
Also, this data's exposed through APIs
for our services and anti-fraud management products.
- And here I can also see that you have data
that's kind of running throughout Azure Synapse, right?
- Yes, well noted.
At the same time we're sending data to Azure Synapse
but in this case we're not just using specific data sets
but sending all of it.
We want to use the data warehousing capabilities of Synapse.
And instead of using Azure Functions,
now we're using Azure Databricks
to deal with the massive amount of data.
From Synapse we can have the computer powered
and reduce processing time for our analytics work
in a single environment with an infinite scale.
This way we can create, change and enhance
machine learning models features.
Finally, the last step is to send this enhanced data
to Azure SQL Servers Hyperscale databases
to be better at the fraud detection.
- Okay, so you can store and work off a full pipeline.
Now you're able to separate the operational needs then
from ongoing analytics work as well
to kind of improve fraud detection.
But how did this improve then your batch processing?
- We had one scenario that usually took one week to complete
but on Synapse the same execution took only six hours.
Also, the major benefit now
is that we can move into new areas
with our analytics work and process correlations
between data sets with a greater performance.
- These are huge savings of time especially.
So can we take a look at what you've done?
- Yes, sure, the good thing about Synapse
is that we can use it for our favorite tools
and notebooks to connect to it.
In this case, I'm using Azure Data Studio
in our existing Synapse environment.
Here, I have two major data sets
from two different industries that I want to merge.
So I'll click in the first data set and count it
and you'll see that it has around 151 million rows.
Now, in the second data set, I'll count it too
and we can see that it has more than 68 million rows.
Let me cross these two data sets by the Brazilian ID number
and some other purchases by this ID.
Running this query takes around seven minutes
and we can see that we have around 31 meter rows
where the same person has made transactions
across two different industries.
Just to prove that it worked,
we will get the first hundred rows against the aggregation
and you're going to able to see the person ID
and the sum of the transactions.
So this is a dramatic reduction in time
and a much better performance for correlating
these two huge data sets.
We weren't able to do this before
and now our data scientists and analysts
will have faster insights.
- Okay, awesome work.
So we've seen the advantages then
that increased compute power brings.
But does Synapse also help
with different styles of workloads
that you're doing at at Clearsale?
- Yes it does.
A motivator for us is being able
to split the available resources between
our data teams using workload group management.
Here we are on the Azure portal in our existing environment.
And we can monitor the usage
and how many queries are active.
Sometimes, business needs may require different priorities.
We have defined eight workload groups
which allow me to assign and track the consumption
based on specific users and workloads.
We have defined groups for our BI analysts, data scientists,
data engineers and others.
What is great is that if a user has a permission,
he or she can change their own workload groups,
so a critical job can run in a group with more resources.
Here we are assigning a different workload group
to this BI job.
Now with this group assigned,
I'm able to run it with more computer power.
- This is great to see.
So, where else are you going to go then at Clearsale
with your Azure Synapse Evaluation?
- We're expecting to use
the masking capabilities of Synapse.
Being able to do that in the row and column level
is an important feature for us.
Also, we are excited about using Synapse's Studio.
This is going to allow our analytics team
to use not only the DW capabilities,
but also use Spark by Spark
and SQL on demand notebooks and pipelines.
- Thanks so much, Jelther,
for sharing your Azure Synapse Evaluation with us.
Now, by the way, if you liked hearing from Clearsale,
please also check back to the rest
of our How we Built series at aka.ms/AzureSynapseSeries
to learn from early adopters of Azure Synapse.
That's all the time we have for today's show.
Thanks for watching and we'll see you next time.
(electronic music)