- Businesses everywhere are looking for ways
to improve their operational efficiency
by enabling data-driven decision-making.
However, to achieve this accurately and responsibly,
it requires the use of high-quality,
relevant and governed data.
Hi, my name is Luv Aggarwal,
and I'm a solution engineer for IBM Data and AI.
And today I'm here to talk to you
about how your organization can deliver quality data
using our DataOps methodology.
So let's start with what is DataOps.
DataOps is the orchestration of people,
process, and technology to deliver trusted
high-quality data to data citizens fast.
The practice has focused on enabling collaboration
across an organization to drive agility,
speed, and new data initiatives at scale.
From an operational perspective,
DataOps integrates a continuous process of data discovery,
transformation, governance, integration, and curation,
and cataloging for self-service.
Our unified and extensible hybrid data platform,
IBM Cloud Pak for Data, supports all the services
that I mentioned and more as part of a DataOps tool chain.
Now let's take a look at a quick demo
to see how both a data engineer
and a data consumer in an organization would use
IBM Cloud Pak for Data's DataOps capabilities.
Okay, so first I'm going to log in to Cloud Pak for Data
as a data engineer and start by adding some data sources.
I'll go over to the Platform Connections tab,
and click on New Connection.
Now we can see the wide variety of IBM sources
as well as third-party sources that we can connect to.
And I wanna connect to a Db2 instance that I have.
So I'll input in all my credentials
and click on Create Connection.
Okay, now that we have our connection defined,
let's see how we can discover
data quality information from it.
So I'll jump over to the Data Discovery tab.
And now we have the option between quick scan,
which analyzes a sample of each table or file
to quickly provide analysis results
or automated discovery,
which provides detailed analysis results
of all assets from the data source
and is typically suitable for a subset of data.
So for now I'll run an automated discovery.
I'll select the data connection
that I set up and the parameters that I want,
including publishing the data asset
to our enterprise catalog
so that consumers can find this data,
and I'll click on Discover.
Okay, while this is running,
the platform is pulling metadata from our source
and assessing it for quality using metrics,
including data classes, formats,
frequency, distribution, and more.
So now I want to show you what that analysis looks like.
I'll open up the results
and we can see a dashboard
that summarizes our data quality and allows us
to dive deeper into the specific assets and relationships.
All right, now let's move on to,
how can we govern our data assets?
So two points I want to touch on here.
Data classes and data protection rules.
Classes allow us to specifically define
different types of data attributes
based on our business language.
So we could have something like customer number
or email address, which we can see as defined here
by a specific format.
Now that we have our classes defined,
I will switch over to the data protection rules.
Now these allow us to define custom rules
on how we wanna manage sensitive data.
So we can see, I already have some rules in here,
such as email masking,
which completely redacts any data fields
that matches with the data class email.
I could also substitute it for other values
or I could just obfuscate it.
All right, now I'm going to switch hats
and log into the platform as a data consumer.
So as a data consumer,
I'm looking for specific data
that I need to run some sort of analysis.
So I go to my enterprise data catalog,
and now I can shop for exactly what I need
by searching a variety of business terms,
data classes, or other attributes.
And I'm able to search across the entire data landscape.
Does not matter whether those data sources
are hosted on different clouds or on prem.
Everything is in the catalog.
Okay, so now I found the data set
that I'm looking for and I can preview it
to make sure this is what I need.
Now, if you have noticed,
I can't actually see all of the data.
There are particular columns in here
that are masks for me.
Now, this is because of the data protection rules
that we set up prior, which apply across
all data assets in the catalog.
So this allows our organization to maintain data governance
and compliance, protect sensitive data,
while at the same time,
exposing the data that our analysts
and data scientists need.
I can also take a quick look at the datasets profile,
which allows me to further analyze it for quality
and make sure the values match a particular class.
There are no missing or mismatching values
and reviews statistics about the asset.
Okay, this looks good to me.
So I'm going to add it to my data science project
and use it to start building insights.
Now, once I'm in my project and I decide
I need to make some changes to the dataset,
I don't actually have to go back to my data engineer
to request those changes.
I can use another self-service tool
to make those changes for my project.
So I will open up the data set here and click into refine.
Now this allows me to make minor transformations
to my dataset in a point and click manner
and save it back to my project.
Okay, so to recap, what we did as a data engineer,
I was able to leverage the platform
to connect to various data sources,
assess them for quality,
define specific data protection rules,
and then publish the assets
to an enterprise knowledge catalog
to make them available to all our data consumers.
And as a data consumer,
I was able to use the same platform
to search for the data that I need
and quickly get access to it
for my analytics project.
I was able to make the changes to how my data set behaved
without the lengthy process of having to go back
to a data engineer to request those changes.
On one single platform,
we were able to fulfill several different activities
of the DataOps life cycle
and take a process that traditionally can take days
into one that can be completed in hours.
While we only looked at a few
of the DataOps capabilities today,
check out the rest by requesting a no-cost trial
of IBM Cloud Pak for Data in the link below.
Thank you for watching.
If you have any questions,
please drop us a line below.
If you want to see more videos like this in the future,
please like, and subscribe.
And don't forget.
If you want to learn more about IBM Cloud Pak for Data,
please check out the links below.