In a previous video,
we talked about the data Lakehouse concept
and shared a story about how data lakehouses
are much like the operations of a commercial kitchen in a restaurant.
So definitely check out that video if you haven't seen it.
Today, I'd like to discuss more about the key drivers for
and values delivered by an open data lakehouse architecture,
as well as share a couple of examples.
And to help me do that,
I'm very excited to invite Edward Calvesbert,
Director of Product Management for IBM Databases, to join us.
Edward, thanks for being here.
Hey Luv.
So Edward, let's start with the major macro trends that we're seeing
and that are leading organizations to modernize their analytics infrastructures.
How has the use of data shifted
drastically in the past couple of years?
Well, there are three major macro
trends that we're seeing in the market.
First, the amount and
cost of data is exploding.
Second, data consumption
patterns are expanding
and changing.
And third, data architecture
is being disrupted and transformed.
So could you speak briefly about
each of these?
Sure.
So there's no doubt that the
amount of
data is expanding rapidly.
But also, it's coming from
a variety of different sources
and in all sorts of new
data formats.
What this means is that to manage
all of this data, enterprises
are spending more money.
And some estimate that that is in
the range of 30% year
over year.
Okay. So when you talk about the
cost of data, are you
just referring to the cost of
storing it in different repositories
like data lakes, warehouses
or other stores?
Or are you also referring to the
cost of managing and governing
the lifecycle of that data?
Well, it's actually both.
Let's talk about the patterns of
data consumption.
There's an ever increasing demand
for the use of data, especially from
business users.
There's no doubt that analytics
has become an essential component of
almost every job, and certainly
AI - the use of AI
is expanding rapidly.
Now, this doesn't mean that every
business user needs to become an AI
expert, but it does mean that
more and more we're seeing AI being
used to automate and optimize
certain decisions at scale, such as
advertising campaigns or supply
chain networks.
A.I. is also being used augment
human "in the loop" decision making
such as credit
risk underwriting.
This means enterprises are always
looking for more data
and use it to drive new
insights.
Right. And what about
the data privacy and data regulatory
concerns that are around AI?
Sure.
When you combine this with
increasing regulatory standards,
enterprises will require higher
levels of built in data
security and governance in
order to enable this data sharing
and consumption.
Absolutely. And you know, another
thing that we hear a lot is the
democratization of data.
So it's about create.
It's about time to value, right?
Business users, yes, they need data,
but they need it like yesterday.
Right. It doesn't do the user
or the organization a lot of benefit
if it takes a long, complicated
process for users to get access
to that data.
So to get the most value out of the
data, it's got to be consumed as
quickly as possible
and all while still adhering
to those governance and compliance
policies.
Would you agree with that?
Absolutely.
That leads us to our third point,
which is architecture.
Organizations are realizing
that the way that data is managed
needs to change.
The emergence of commodity
cloud, object storage
and the adoption of open data
formats
is really allowing enterprises to
increase the return on investment
in data management.
And they're doing this through
the optimization of the price
performance
of their workloads across
different storage and compu-tiers.
Now, what does that mean?
It means that organizations can
benefit from having the right tool
for the right job at the right
cost instead of
defaulting into a data warehouse,
which may be appropriate in some
cases, but can also become very
expensive and ineffective in others.
Right. So what I'm hearing is
more data, more users and
more uses of that data.
Right. And all while still
better ways to share and manage
access around it.
Right.
So, Edward, how are
these aspects related to
the key values that are delivered by
an open data lakehouse architecture?
Great question Luv.
We see three key values that are
delivered by an open data
architecture. First,
an open data Lakehouse provides
the foundation for users to easily
and cost effectively access,
store, manage and
unify large amounts of data
and from different sources and in
different formats.
Second, an open data lakehouse
can be easy to deploy within
existing environments, providing
users fast access to more data
without having long procurement,
onboarding or data pipelining
and wrangling processes, making it
much easier to consume.
And third, open data lakehouses
can optimize analytics workloads
to run where they perform the best
and are most cost efficient.
All this part of an integrated
architecture.
Wow. So I've noticed one
thing. You've said the word "open"
a lot.
Can you explain to us what the
difference is there?
Sure. I'm glad you brought that up.
A lakehouse should leverage the
capabilities across existing
data and analytics environments.
If you already have data and
analytics workloads in a data
warehouse or in a Hadoop data lake,
that's okay.
You shouldn't be forced to migrate
or rip and replace that environment
in order to get started with the
lakehouse.
But what about new data and new
workloads?
Well, a lakehouse should be the
starting point for new data and new
workloads, as well as provide a
modernization path for
existing environments over time.
Ya know, the other thing that we hear
often is customers getting stuck
with one vendor.
Well, open also means
that you're always in control of
your data and you aren't forced into
proprietary data formats or
specialized tooling in order to use
it. It also means that you can
maximize the use of your data
without having to make copies of it
and move it around.
And this provides for lower costs,
higher productivity and
better governance, which ultimately
leads to what enterprises are
looking for, which is more trusted
decisions.
Wow. So, Edward, what
I'm hearing is a data
lakehouse is like
a network of highways.
Some of them have tolls
on them and some of them don't
like regular freeways.
And a lakehouse allows you to
go on the toll road.
Pay when you need to get somewhere
really fast.
But when you're not in a hurry and
there's no traffic.
Take the regular highway.
Luv, that's a great analogy.
Think about using a data lakehouse
to cut your data warehousing costs
by up to half by optimizing the
price performance over your analytic
workloads.
Or like you're saying, saving money
by driving on the freeway while
there's no traffic in, you're not in
a hurry.
Wow, that sounds really exciting.
Let's hit the road.
Let's do it.
Thank you. If you like this video
and want to see more like it, please
like and subscribe.
If you have questions, please drop
them in the comments below.