- It's no surprise that the volume of data
across multiple stores, locations, clouds
and even vendors is accelerating.
But how do you manage this complexity
and make it simple to leverage your data?
Hi, my name is Luv Aggarwal
and I'm a Solution Engineer for IBM Data and AI.
And today, I'm here to talk about
one of our newest capabilities, AutoSQL.
So I wanna first start with how we got here.
Traditionally, we have seen many architectures
that have big data warehouses
with storage and compute tightly coupled
as well as data lakes in multiple clouds
with a lot of ETL pipelines
to move and replicate data around
for different BI and data science use cases.
This has led to increasingly complex data pipelines,
difficulty in scaling workloads
and unnecessary data duplication.
What we have seen become more common
is a new modern architecture
which utilizes separate compute engines
layered over inexpensive cloud object stores and data lakes
resulting in the concept of a data lake house.
So now let's get back to AutoSQL.
AutoSQL is our new unified compute engine
in IBM Cloud Pak for Data
that can query both structured and unstructured data
directly over your data lakes and cloud object stores,
leverage data virtualization
to access other external data sources
as well as support Spark.
In addition, AutoSQL brings integrated governance
as part of the Cloud Pak for Data platform,
which allows any ingested or connected data source
to be fully governed end-to-end with custom policies.
Now we have a single interface
and engine to support both data science
and BI across any data source environment,
whether that be your on-prem data warehouse,
S3 buckets in AWS, data lake in Azure,
Snowflake, Oracle, Teradata, it doesn't matter.
All right, now let me show you with a quick demo
how easy it is to access data
from various sources with AutoSQL
and our end-to-end hybrid data platform,
IBM Cloud Pak for Data.
So I'll start by logging onto Cloud Pak for Data
and once I do that, I'm presented with my homepage.
Now I wanna connect to some different data sources,
so I'll go over to platform connections under data
and click new connection.
So we can see there's an extensive list of both IBM sources
as well as third-party sources.
I'm gonna connect to an S3 bucket that I have it up in AWS.
I'll put in all my credentials
and click on create connection.
So this connection will allow us
to directly query a resource,
however, I also want to virtualize some data sources,
so I'll click over into the data virtualization tab.
Now, if I look at my sources,
we can see the many different instances
that I have virtualized in my constellation view.
And now I'm ready
to actually do something that's very powerful,
which is using data virtualization
to combine tables from multiple sources
into one virtual table for us to use.
So I'll go ahead and search for the tables
that I have virtualized
and join them into a new virtual table
in a way that allows me to pick and choose
exactly how I want it to be structured
based on the available attributes.
Okay, great, so now this new table is available
for us to start using to build insights.
So I'll hop over to the projects tab
and open one of the data science projects
that I've been working on.
We can see there are several data science assets in here,
but I'll go in and open one of the notebooks
that I've already been working on.
Okay, in here, we can see that I have the ability
to query that same S3 bucket that I connected to earlier
as well as that new virtual table that I created.
I can now use this data to build out whatever model I want
and deploy it directly in the platform
to make it available for consumption by my business analysts
or other data consumers in my organization.
All right, so to recap,
we connected to various data sources
in the Cloud Pak for Data platform,
we virtualized certain sources
and created new virtual tables
to interact with our data in new ways.
And then we were able to query those sources
right from our notebook to build and deploy models.
and by the way, all of this was done in a governed manner
where any governance policies
that were defined in Cloud Pak for Data
apply to all of the data sources that we connected to.
With AutoSQL, we're reducing costs by reduced migration
and significantly less data duplication,
we're reducing complex ETL work
as we saw when simply creating virtual tables,
we're automating security and governance
for trust and data validity and quality,
we're leveraging one performant and scalable query engine
for both big data and warehousing
that can execute distributed and virtualized queries
53% faster than the industry standard
and we're avoiding lock-in with our vendor agnostic design
that allows the same engine
to work with any data source on any cloud.
If you'd like to see more videos like this in the future,
please click like and subscribe
and if you want to learn more about IBM Cloud Pak for Data,
make sure to check out the link in the description.