(air whooshing)
- Coming up on this special edition of How We Built It,
we're joined by a Singapore-based
mobile technology company, Grab,
who built an everyday, everything super app
to match consumers to common services in South East Asia.
Now, we'll see how they're evaluating Azure Synapse
to improve their data governance
with things like dynamic data masking
and row-level security
to empower business users in different countries
and verticals to compliantly access and analyze their data.
So I'm joined today by solution architect, Adhi Utama,
coming from Singapore today.
Welcome to Microsoft Mechanics.
- Thanks, it's great to be on the show.
- And thanks so much for joining us today
and by the way, before we get into this,
just a quick recap.
If you're new to Azure Synapse,
this is Microsoft's limitless analytics platform
that really brings together data warehousing capabilities,
as well as big data processing into a single service.
And you can learn more at aka.ms/mechanicssynapse.
Also if you're new to Grab, they play an important role
in terms of matchmaking and connecting micro entrepreneurs
and customers, delivered goods and services
and have millions of people using their apps
across Southeast Asia.
Now they currently offer services across eight countries
and over 330 cities.
Everything from ride sharing, delivery,
cashless payments and more.
So Adhi, with so many different services
across so many different regions and countries,
it's a pretty unique challenge to deliver
this kind of service at scale and also in a secure way.
- Right it is, so we have about 300 or so analysts
combing over 30 terabytes of data a day
and we are always looking to find the next opportunity
to better serve our customer and to identify new services
that we can bring into the market.
We've been really focused on building
a self-service analytics platform
that allows the right permissions
for our analysts to access data,
regardless of their skill level.
- So how are you doing this right now?
Can you describe your current setup?
- Sure, let me show you with a visual to explain it.
Our original solution was open-source based.
We decoupled our compute and storage engine.
Our data lake is storage and we used Presto
for our compute and data warehousing.
Our analysts could query from a number of interfaces,
such as our internal ETL tools for SQL-based query,
to get filtered fields of data.
Or they can use data visualization tools and notebooks.
Now as we've grown, we needed a way
that supports a better data governance,
as the current solution doesn't natively support
rich access control features.
So there's no row-level security,
there's no column-level security in the current solution.
We've had to constantly duplicate the data
for each country and filter the views manually.
There are a number of pinpoints here.
For example, because the data is live-queried
and not cached, it needs to be recomputed
each time we access it.
This increases our compute cost and reduces performance.
Now to ensure the right access control is in place,
our data engineers need to create multiple views
per table available based on the location.
This results in a high maintenance and storage overhead.
And of course, at the end of the day,
this isn't very scalable.
- Yeah, so it's kind of the opposite when we think about
role-based access controls.
You're actually having to create bespoke data sets
then for every individual access requirement.
So if you start to kind of extrapolate this
across eight different countries, eight views,
the same table when you factor that then by city,
it sounds like you have to create maybe
hundreds of views or more of the same data.
- Exactly.
We have plethoras of data that are
constantly changing every day, and we have around
1,000 active dashboards that our analysts are using.
Now our analysts are having to recreate the access control
literally one dashboard at a time.
- So how have you approached, then,
your solution and your evaluation with Azure Synapse
to respond to this and make things better?
- Right, so we've now started to develop a more mature model
for data governance and analytics in a single service.
The raw data comes into our data link on Azure,
and we are able to transform them using Synapse
in conjunction with Azure database.
We store our processed data in SQL pools
for data warehousing capabilities.
From here on, we're able to query the data
and visualize it through notebooks at scale.
And the most exciting part for us
is the governance capabilities that can be achieved
through row-level and column-level security.
This allows us to lop down the data at the cell level.
And we also have dynamic data masking,
which allows us to redact sensitive data
while it is being shared with analysts.
And we are also able to have
a unified authentication system with Azure Active Directory
with multifactor authentication across these services.
- That's a huge improvement from a security perspective,
and if you're new to row-level security,
you can make sure that users only have access and visibility
to those data roles that are pertinent to their location
or their vertical or their role.
Now this all happens against the same shared dataset
so there's no duplication that's required.
But can we take a look at what you've implemented here
using Azure Synapse?
- Yeah, absolutely.
So here, I have created what we call as a filter predicate.
This will filter the data before an analyst can query it.
It contains the logic for row-level security.
In this example, we are using RecordTypeId
column that represents country, city, and vertical.
Next, we create a security policy that calls the predicate
and applies it into our tables in Synapse.
So let me start by showing you the kind of filtering logic
that we have against our data.
So let's run this, and what you'll see is that
even though I'm an admin, I don't have
row-level access to this data.
Therefore, the query will return an empty set.
Let's try this out by running the same query
with two different AD authorized users.
The query on the left is for an analyst
that is the head of vertical.
For example, our right setting is this
across three countries.
It's trying to query metrics in units of hour
against our CRN system.
On the right, you see the exact same query
for one of our country heads is running.
So let's see what happens when I run these queries
for each analyst.
First, when I run the query on the left,
you see that the results are scoped to three countries,
which is based on what this analyst is allowed to see.
And then on the right, when you run the same query,
you can see that only one result specific to this country
are returned for this analyst.
This means it simplifies our audit process.
It simplifies our data governance process.
It improves our speed of execution and time to insights.
And it's also more efficient because we are using
less compute and storage.
- Now you don't need to create multiple views
against the same data but you also mentioned
that you're using dynamic data masking.
Can you show us how that would work?
- Yeah, sure.
So with dynamic data masking, we can ensure that
sensitive information is secured without limiting
the analysis that we can do.
If you're familiar with GDPR,
we also have similar regulations here called PDPA,
Personal Data Protection Act.
And here, let me show you the demo,
here we have a table with some information
that we want to mask,
such as phone number and email address.
I can mask the data with zero engineering effort.
I can choose from a number of preview functions to mask,
in this case email and phone numbers.
So now, as an analyst, let me run a query that includes
email and phone numbers.
And you see that everything is masked,
so the information is redacted.
What it means is I can still do analysis
but I won't be able to see sensitive information.
This gives data governance teams the peace of mind,
while at the same time empowering our analysts
to do their work efficiently.
- It's great to see these two different vectors of security.
You're able to provide the right access to information
at scale without any of the workarounds
that you were doing before.
But what's up next for Grab?
- So Azure Synapse gets us closer
to our data governance goal of having a data warehouse
as the central source of truth for powering,
reporting and analysis for our data scientists, analysts,
and data community's at Grab.
It also helps to establish a single source of truth
for clean, curated, and reliable
analytical metrics and data.
And now, access control can also be implemented
and maintained in a scalable, centralized way
with audit controls.
So yeah, we are pretty excited for the future.
- That's great,
and thank you so much for joining us today, Adhi.
And by the way, if you liked hearing from Grab
about their Azure Synapse evaluation,
please check out the rest of our How We Built It series
at aka.ms/AzureSynapseSeries to learn from early adopters
also kicking the tires of Azure Synapse.
That's all the time we have for today's show,
thank you so much for watching and goodbye for now.
(air whooshing)
(upbeat techno music)