hello everyone welcome to this session
on databricks lakehouse overview and
thank you for joining us today
my name is astrid i'll be your
instructor today i've been with
databricks since
2019 and i work on the curriculum
development team
so my role here is to put together
instructor-led
and self-paced content for our customers
and i will be
the main one presenting today along with
me today we have fantastic tas who are
going to help us out
they'll be answering your questions in
the summit platform
so hello and thank you to all of our tas
so let's go ahead and talk a little bit
about this session
there are two sessions that are being
offered for free during this data in ai
summit
and i want to make sure that everyone is
in the correct session
so the first of those complementary
sessions is the databricks like house
overview
that is this session this course was
developed for those of you who are
new to the big data in ai space so
perhaps you're in a more traditional
business management role where you're
helping your team move
to the cloud perhaps you oversee a data
science team
and your primary role is to make sure
that business needs are met
but you might not have a technical
background yourself
or perhaps you're looking to change
careers and you're brand new to this
space
and want to know more about big data and
ai
if any of those sound like you or
similar to you
this is a great course for you to take
our hope is that you walk away from the
session
feeling comfortable to have discussions
about and even come to decisions
about how to best position your
organization to make the most out of
your data
in this course we will start with the
basics so
in the first half we're going to take a
step back from databricks and
just introduce you to concepts like data
types
data formats the key roles and abilities
to look for
when you're building out your data teams
the different ways that organizations
today are working with big data
so for example what is machine learning
and how does that differ from deep
learning
we'll spend a good time talking about
data management systems
so looking at the pros and cons of data
warehouses and data lakes for example
and we'll talk about the issues that
organizations face when it comes to big
data and ai
so that's the first half of the course
in the second half of the course
we'll review how the databricks like
house platform fits
into an organization's big data
architecture
we'll talk about what it is what it does
and then we'll review some
customer case studies so that you can
see how real world customers are using
the data breaks like house platform to
streamline their big data and ai needs
everything we cover in this session will
be from a business or operational
perspective
so to be clear we won't be showing any
code in this course
and we won't have any hands on technical
labs in this course
so if you're sitting there and thinking
oh i actually wanted something more
technical
please look for the lake house with
delta lake deep dive course
that course is also complimentary this
year and
it'll provide you with an in-depth look
at delta lake features and
functionality in that course you'll
learn about applying software
engineering principles with databricks
and we'll demonstrate how to build an
end-to-end olap data pipeline
using delta lake for both batch and
streaming data
that course will also discuss serving
data to end users through aggregate
tables
and databrick sql analytics so again
if you're looking for a more technical
engineering course that is the one for
you
but if you're still here i'm assuming
that you're ready to get started with
this course
so let's go ahead and get started
so here's our agenda for the day our
session will be about three hours long
including a few knowledge checks and
several breaks
for the knowledge checks i'll ask you to
please type your answers in the chat
if you have questions about anything i
present please also use the chat
as well to ask those questions and our
tas will be there to answer those
for some of you this might be your first
session so i just want to make sure that
everyone is good to use a chat
and i also want to know a little bit
about you so please go ahead and let us
know where you're joining us from today
so please put that in the chat while you
do that
i'll let you know that i'm joining you
from the orange county area in southern
california
so i'll give everyone a few more seconds
to type into the chat
just where you're from it's great to
meet all of you again thank you so much
for being here
next i'm just curious if you could
please let us know what industry
your organization belongs to
so go ahead and type that into the chat
i'll give you about
10 seconds to do that
so again we're asking for what industry
your organization
belongs to
next if you know this might be fun
so what is the total volume of data that
your organization manages today
so go ahead and put that into the chat
again i'll give you about 10 seconds to
do that
and again what we're looking for is the
total volume of data
that your organization manages today
and then just so that we know a little
bit more on our end so
what do you aim to learn during this
session
so again i'll give you 10 seconds this
is really just so that we know
why you're joining us today and what
you're hoping to get out of this session
so if you could just add what you aim to
learn during the session in the chat
that would be great
so again welcome everyone and let's go
ahead and get started with our session
so we'll start with looking at big data
benefits for business and
i want you to think about this question
i want you to think about how well you
know this customer
so if i tell you that we have a customer
he is a male he's 35 years old
he's single he lives in new york city
and he makes about one hundred thousand
dollars per year
what do you think would happen if his
bank account
accidentally transferred a five thousand
dollars into his account
do you think he would give the money
back or
do you think he'd take the money and run
so just think about this for a few
seconds if anyone feels particularly
excited about either of these answers go
ahead and put it in the chat
and when you do that tell us why so why
do you think that he would either give
the money back or
run now
i'm sure that some of you are thinking
well we can't
really know there's too many unknowns
well what if we had more data so let me
give you a little bit more information
what if i told you now that this
individual
is darth vader yep so i'm telling you
that darth vader
packed it to death star and moved to new
york
how would knowing this change your
assessment of what would happen to that
money
so just in case someone in this call has
not heard of darth vader
he probably wouldn't care about the
legalities of keeping money that isn't
his
so how did knowing this change your
original answer
or did it change your original answer
knowing more about our customer allowed
us to really
up the certainty on what decisions they
would make
in a given situation now obviously this
is a simple example it's just a little
bit of fun
but think about this example on a much
larger
and much more complex scale and that is
why big data and ai
are becoming very important tools for
organizations today
as we'll discuss further big data is the
new norm
when it comes to data so if your
organization
is not currently prepared to
successfully process that data
or analyze that data or if you're
struggling it's really important to find
solutions that can help you
databricks is one of those solutions
that we'll talk about again
in the second half of our course today
so let's take a look at these statistics
today 83 percent of ceos say that ai
is a strategic priority for their
organizations
and then we look at this quote gartner
predicts the business value created by
ai will reach
3.9 trillion dollars next year
on the other side of that though gartner
also predicts that through 2022
85 of ai projects will deliver erroneous
outcomes
in other words they will fail as you can
imagine this is a complex landscape to
navigate which is why we built this
course
over the next few slides we're going to
look at some of the reasons why
every enterprise today wants to begin to
leverage their data
for ai
when you combine big data in analytics
you can answer questions to help you get
to know
again your customers better your
vertical better
your organization better your products
better
you can help protect your business and
stay ahead of the competition
you can ask questions like how are
customers actually using my product
how are my products performing is our
data secure
are we keeping up with industry trends
we list several questions like this on
this slide i'm sure that you can think
of others
and even though we've been keeping this
at a high level so far
these types of questions are relevant
across every industry
so right now i'll just give you one
example from each of the industries
shown on this slide
these are actual use cases on data
bricks but this is in no
way an exhaustive list so if you're
looking at the retail space
someone might be looking to use ai for
demand forecasting
to predict real-time demand in retail
stores
if we think about media and
entertainment we might have a streaming
platform
that wants to use our data to predict
which customers are at a greatest risk
for churn
and then proactively take steps to
prevent churn
maybe they'll do things like improving
personalized recommendations for example
if we look at the health and life
sciences we have research organizations
who might want to combine real-world
data sets
massive data sets with genomic data to
identify biomarkers that have
high probability of driving the onset of
disease for example then we have
financial services so we might have
companies that look to detect and
prevent fraudulent activities
things like money laundering credit card
fraud etc
they would leverage techniques like
machine learning for example to predict
anomalies in real time
so those are very different use cases
but they're all made possible
as long as you're using the right tools
on your big data
so before we continue talking about big
data let's just define what it is
and first i really want you to think
about how much data has changed over the
last
20 years and either in your head
or in the chat try to answer these
questions
so how has data changed over time
how has that affected what we can do
with it
how has that affected how we manage it
again please type answers into the chat
if you'd like
if not just think about these for a few
seconds i'll give you about 10 seconds
so over the last 20 years advances in
technology have completely changed the
world that we live in
today the amount of data being generated
every second has
increased dramatically and that's due to
the invention of new sources of data
that didn't exist before
almost everything we do generates data
if you think about the things that
you've done today
did any of you text someone or post
something on social media
did you exercise and log your workout
using a fitness tracker
have you used a smartphone today to
browse the internet or
pay bills or go shopping at a minimum
you logged into the summit platform to
take this course
and as you did all of these things you
along with the machines or the tools you
use to do them
generated data we typically think of
data as being
human generated machine generated and
organization generated
human generation or human generated data
is data that humans create and share so
some examples of human generated data
can be social media posts
emails spreadsheets presentations
audio files video files i will stop here
and point out that social media has been
a leading force in the propagation of
human generated data
just think every time you post a message
change your online status
you upload an image even when you like
or forward a comment
you are generating data
then we have machine generated data that
is data
coming from machines that don't rely on
active human intervention
so some examples of machine generated
data sources could be
sensors on vehicles sensors on
appliances
or industrial machinery what about
security cameras
uh satellites medical devices
uh personal tools such as smartphone
apps or fitness trackers
and you might be asking well what does
it mean for data to be generated without
active human intervention so if you
think of a fitness tracker
depending on the model you have uh it
might generate records for your heart
rate
your geographic location the calories
you burn
and more you don't tell your fitness
tracker to track these things
it comes programmed to do it and then it
does it on its own
and then we have organization generated
data and that is data generated as
organizations run their businesses so
if we think of some examples of
organization generated data
think about the records generated every
time you make a purchase at an online or
physical store we have things like
unique customer numbers
the items you purchased the date and
time you purchased the items
how many of each item you purchased
this type of data this organization
organization generated data
is often referred to as transactional
data and you'll hear that term
frequently
in the world of big data here's an
example of transactional data as you can
see we have a customer number the item
they purchased
the date and time they purchased it and
the amount they purchased
and one quick note here is that although
all of these sources
are constantly generating data not all
of the data generated
is necessarily considered big data so if
you think of yourself sending text
messages throughout the day for example
the collection of your daily text
messages is probably
not considered big data unless you're
sending millions of messages of course
over time those messages might be
considered big data
so we've referred to this term big data
for a while now since the very beginning
of the course
uh now we'll actually define it we'll
review characteristics of big data
so let's start off by just looking at
characteristics that distinguish
big data from just regular plain normal
data
one thing that is important here to note
is that
these characteristics and why we're
talking about this
these characteristics are what make it
really difficult to work with big data
and why equipping your teams to do that
work is not an easy task
so when we think about big data that
term big data
refers to data that is nearly impossible
to process
using traditional methods because there
is so much of it
coming in so quickly and in so many
different formats
we summarize these major characteristics
used to define big data
as volume velocity and variety
now depending on who you're talking to
these v's this volume velocity variety
can change
sometimes there are other ones that
people use to describe big data
in this course we'll just stick to these
three i think it gives a pretty
comprehensive overview of what we're
talking about
so we'll start with volume volume refers
to the incredible amount of data being
generated
every second of every day the
international data corporation
also known as the idc is a global
provider
of market intelligence they do advisory
services
in the i.t space and they forecast that
the amount of data that exists in the
world will grow
from 33 zettabytes in 2018
to 177 zettabytes by 2025.
just to put that into perspective the
computer i'm on
right now has about 256 gigabytes of
storage
and that's equivalent to chess point
zero zero zero zero zero zero zero
zero zero two five six data bytes that
was
nine zeros 33 zettabytes seems like an
impossible number
but if you think about how much data
each and every one of us generates every
day
through social media through cell phones
personal fitness trackers etc
it starts to make more sense if we look
at this from a business perspective
that's great right we have so much more
data coming in that we can use
however how are you going to store that
data
how are you going to capture that data
right these are some of the questions
that we start thinking about
the second characteristic that defines
big data is velocity
and velocity refers to the speed at
which new data is generated
and the speed at which it moves around
so again if we look at the idc
they tell us that in 2025 each connected
person
will have at least one data interaction
every 18 seconds
i mean even now think about the speed at
which credit card transactions
are checked for fraudulent activities i
know it's happened to me
that i might be trying to purchase
something out of the ordinary and all of
a sudden my credit card company reaches
out to me
within seconds to ask me if it was me
sometimes it's so quick that they
actually block the transaction
right there until i verify that it is me
and
while it's awkward to have your credit
card declined
you know i'd prefer that to a stolen
identity from a business perspective
how can you make use of data coming in
so quickly
and then also how do your teams process
such an enormous amount of data that is
coming in
both so quickly and then constantly
and finally when we talk about big data
we have to talk about data variety
there's so many different types of data
that are being generated today
we have social media posts credit card
transactions biometric data
customer transactions geographic
information and that's just to name a
few
and it makes it really difficult to make
sense of all of that data together
so again if we put on our business caps
and think from a business perspective we
ask ourselves
how can i combine all of these disparate
types of data
to extract useful information from them
we also mentioned at the very beginning
of this course that
there are different types of big data
and as you can imagine
different types of big data have
different storage and processing
requirements
today we'll talk about these three main
types of big data that i have on the
slide here
which are structured semi-structured and
unstructured data
every data science team today including
those in your organization are probably
using
all three of these and as mentioned each
of them require
varying amounts of processing before
they can be used to generate business
insights
so structured data the term structured
data refers to
any data that conforms to a certain
format
or schema it's usually clearly organized
and so because it's clearly organized
it's generally easier to analyze
which is why for a long time we kept our
data as structured data
if we look at this spreadsheet for
example we see that there are clearly
labeled rows and columns
and the information within those rows
and columns follow a schema
so for example the customer id seems to
be a five digit numerical value
the title of the movies are all letters
and numbers followed by
parentheses genres are also separated
so let's go from structured data to
unstructured data
unstructured data does not follow or fit
neatly into
a schema it's often referred to as messy
data
because it isn't easily searchable
compared to structured data
because of that this data is typically
the most difficult to analyze
of course it's also the most widespread
type of data
the idc reports that almost 90 percent
of data today is
unstructured and because unstructured
data is so readily available
it can be really valuable in terms of
extracting
meaning meaningful insights for your
organizations
one of the most powerful aspects of
machine learning for example
is that we can use it to analyze this
type of data
so again in the chat i'll give you about
10 seconds can you
add in some examples of unstructured
data
that you might be working with
so any unstructured data that you have
coming into your organizations or that
you'd like to use
to help drive business decisions go
ahead and type those into the chat
and we're seeing some of this come
through but yeah some examples of
unstructured data can include
social media posts we have photographs
videos emails audio files video files
images the list goes on then we have
semi-structured data
so semi-structured data fits in between
structured and unstructured data
semi-structured data does not reside in
a formatted table
but it does have some level of
organization
what we have on the screen here is json
you can see that our data is tagged but
the number of tags could vary
so in this particular example we have
middle name or
salary or birth date so we've covered
the three types of big data
again structured unstructured and
semi-structured
and of course due to the characteristics
we just talked about
we can't use traditional methods of
computing to handle this data
so now we'll pivot a little bit and take
a quick look at how we process big data
how we process that big data is called
distributed computing
i'll explain this by using an example
that we use a lot when we
talk about distributed computing i'm
sure it's something that you're all
familiar with i know i
am but you might have had a situation as
a kid where you and your classmates were
asked to guess how many pieces of candy
were in a jar
and the person who guessed the correct
number got to take that whole jar of
candy home
eat candy for days sure your parents
were super excited about it
but imagine this what if i gave you and
a group of your colleagues
an entire tub of candies so not just a
jar anymore
but a large tub of candies and i told
you that whoever finished counting the
candies first
was going to win some sort of prize the
caveat here
is that the count has to be accurate for
you to win you have to be spot on
so what would you do and as we're
talking through this
just think about what what you would do
if you would like to
please go ahead and post your answer in
the chat this might be fun to see what
different people would do
and actually let's go ahead and do that
so i'll give you about 30 seconds
write an answer in the chat the question
again
is if i gave you and a group of your
colleagues an entire tub of candies
and told you that whoever finished
counting the candies first
wins a prize what would you do
to count that candy to win that prize
so go ahead and type something into the
chat
so i'll start uh highlighting some of
these ideas but
you might line the candies up and count
them one by one
uh and that would eventually give you an
accurate count
really accurate fantastic but it would
take a really long time
right if we have thousands of candies
you're going to be counting for a long
time
you could also weigh the candy you could
weigh one candy
and then weigh the entire tub of candy
somehow
and figure out how many pieces of candy
there are based on those weights
now that would give you a count
relatively quickly
but i never told you that the candies
were the same types of candy
right so what if there are different
types of candy in that jar
then you can't guarantee that all of the
candy weighs the same
so you might not have an accurate count
now if we look at this differently we
can
have our colleagues help us right so
what if we split up the entire tub of
candy
between our colleagues we can distribute
the candies among all of them
so that instead of us having to do the
whole count on our own
our colleagues are all counting candies
at the same time
and that will obviously help us get
through that tub of candy much faster
and that idea is the idea behind
distributed computing
so you take data you divide it into
smaller chunks
into small chunks and you distribute it
among workers
which are just different computers we
can then perform whatever our
computation is
in this case it would be a count in
parallel
on smaller subsets of data
distributed computing is what makes it
possible to process big data
so for those of you who have heard of
apache spark or maybe you haven't heard
of apache spark
apache spark is the de facto standard
when it comes to distributed computing
tools
used for big data processing and big
data analytics
the founders of databricks are actually
the ones who invented spark
and for years spark has been the core
engine underlying data bricks
when we think in terms of big data
processing
there are two types of data that we
process
so we can process batch and streaming
data
and those terms batch and streaming
refer to the way that we're getting our
data
and the speed at which we're getting our
data
so batch data batch data is data that we
have in storage and that we process all
at once
or we process in a batch so say for
example that someone gives you a large
jar of candies and asks you to count all
of the candies
in that jar that is an example of a
batch job
because you take those candies that were
already present in some form of storage
in this case a jar and you are going to
count them
since you count them all at one time
that is considered
a batch processing job
a real world example of batch processing
is how telecommunication companies
process cellular phone usage each month
to generate a monthly bill
so to do this they process batch data
the phone calls you've made
the text messages you've sent etc any
additional charges you've incurred
through that billing cycle to generate
your bill
they process that data in a batch job
on the other hand we have streaming data
streaming data is data that is being
continually produced by one or more
sources
and therefore must be processed
incrementally at
it as it arrives so going back to our
candy example
what if instead of counting candy
sitting in a jar
we're asked to count candy coming
towards us on a conveyor belt
as the candy reaches us we have to count
the new pieces
and we're constantly updating our
overall candy count
in a streaming job our final count is
changing
in real time as more and more candy
arrives on the conveyor belt
so a real world example of stream
processing is how heart monitors work
all day long right you wear your heart
monitor
it receives new data dozens of thousands
of data points per day
as your heart beats every time your
heart beats
new data is added to your heart monitor
in real time
if your heart monitor has a display of
your average heartbeat for the day
then that average will be constantly
updated with the new numbers from that
incoming stream of data
and both batch and streaming data have
their place when it comes to
big data analytics so batch data
is used for things like periodic
reporting
streaming data is used for things like
fraud detection
right that's something that needs to be
identified in real time we want we can
go back to our credit card example
right and i mentioned that sometimes you
purchase something that might be out of
the ordinary
and instantly your card is blocked
so historically it's been difficult to
use these different types of data in
conjunction
thanks to new advances in technology
however combining
batch and stream processing is possible
and you can imagine that leads to
significant advantages in big data
analytics when you can combine those two
so next we're going to ask if anyone has
any questions before moving on to
what big data looks like in
organizations today
before doing that we'll take a 15 minute
break
so please take a look at the time right
now and we will meet back
in 15 minutes i'll give you a reminder
when we have 10 minutes left
and another reminder when we have five
minutes left see you soon
all right we have 10 minutes left of our
break
and here's your five minute warning we
have five minutes left on our break and
then we'll go ahead and get started
again
so let's pause now and ask if you have
any questions please remember to drop
any questions
into the chat so that our tas can answer
them
at this point we have defined big data
we've looked at characteristics of big
data
we've also talked about how we process
big data
and now we're going to move into more of
the big data analytics
and looking at techniques that
organizations use
to analyze data there are some terms
that are thrown around
quite a lot sometimes interchangeably
and we want to
start off by looking at some of these
terms
we'll review uh the relationship between
artificial intelligence
machine learning and deep learning
like i said a lot of the times these
terms are thrown around
but they do mean different things so
let's go ahead and start reviewing
these terms artificial intelligence
in essence artificial intelligence is a
branch of computer science
in which computer systems are developed
to perform tasks
that would typically need human
intelligence
artificial intelligence is a broad field
and there are a lot of subcategories
which we'll talk about in a couple of
minutes
to contextualize ai let's look at a
classic example which is a turing test
the turing test was developed by alan
turing
back in 1950 it essentially tests a
machine's ability to exhibit behavior
equivalent to human behavior
in a turing test a human evaluator asks
a question
using a text-based medium to both a
human
and a machine without being able to see
either one if after a preset period of
time
the evaluator is unable to determine
which responses are coming from the
human
and which are coming from the machine
the computer is said to have passed
the turing test and it's an interesting
thought experiment
right can we create a computer that can
act so human
human-like that an actual human just
can't tell
so far there are no uncontested results
where a computer has passed a turing
test but some scientists are predicting
that this will change in the near future
but now we'll talk about some specific
subfields of artificial intelligence
particularly machine learning and deep
learning
machine learning is like i said a subset
of artificial intelligence
the goal of machine learning is to learn
patterns
in your data without explicitly
programming a computer to do it
so there are a few types of machine
learning
i'll list them off just in case you hear
these terms so that you're familiar with
them
there's supervised learning we have
unsupervised learning
reinforcement learning the most commonly
used type
is supervised machine learning and that
is where you are given
a set of data with the correct results
already labeled and the goal is to
predict the labels or the outcomes
of new data of that same format so let's
go into a fraud detection example for
example
without machine learning people would
manually specify
rules for what constitutes fraud so this
could be the number of transactions per
month
maybe if the number of transactions per
month are more than 20
right that might be fraud um an average
balance
maybe if you have less than a hundred
dollars that
constitutes fraud in these rules um
these rules are brittle though
they're prone to human error they can
quickly go stale
with machine learning you can create an
algorithm though
with an input data set that contains
transaction information
that has already been labeled as fraud
or
not fraud after the algorithm is created
on that data
you can pass in new transaction data
that has not been labeled as fraud or
not fraud
and the algorithm can predict if the
data is fraudulent
so is this fraud is this not fraud and
that can be extended for millions and
millions of transactions
so here again we see that power of big
data you can process
far more data than any single human
analyst can
and you can detect changes in trends as
new types of fraudulent activity emerges
you should still include a human in the
loop
they are the ones that will manually
investigate and verify the model's
predictions and make sure that if the
model is saying something is fraud
it actually is fraud
aside from machine learning we also have
deep learning
while our supervised learning example is
very good with
tabular that structured data so for
example data that's in a table or an
excel sheet
we also have deep learning deep learning
performs well on complex data sets
things like images sequences or even
natural language
deep learning is a subset of machine
learning
that uses something called neural
networks which were inspired by the
structure of the human brain
they're much more complex than most
machine learning models
and they require significantly more time
and effort to build
unlike machine learning which plateaus
after a certain amount of data
deep learning continues to improve as
data size increase
deep learning is expanding into more and
more domains things like generating
movie scripts
predicting cancer based on mri scans and
making anyone feel like van gogh with
style transfer
so let's think about an example using
images
let's say that we want to build a model
to classify if an image contains a koala
or a panda
so if you were to describe a koala what
would you say it's small
it has fluffy ears it has a large oval
shaped nose
it has small beady eyes and that's all
true
and that helps us differentiate koalas
from pandas
but what if your image of a koala is
very zoomed in
um what if it zoomed in to a point that
it doesn't have
a left ear does that still make it a
koala
machine learning and deep learning
solutions need to be robust enough
to detect changes like this
if you show a human child an image of a
panda and a koala
odds are that they can distinguish
between those two really well
just based on the examples we talked
about right the large oval nose
and the short fluffy ears etc
to train a deep learning model to
distinguish between those
two however requires lots of examples
and lots of time
deep learning models learn differently
from humans so when we look at deep
learning models
the model might be looking at features
like patterns in the fur
or the shape of the eyes etc so over
time
the model learns what a koala is and
what it isn't
and it's able to make decisions quickly
based on that information
so we'll pause here a second and i'll
ask and go ahead
put this in the chat but can anyone
think of any examples
of artificial intelligence or machine
learning
that we use pretty much all the time
go ahead and post some examples in the
chat
again i'm giving you 30 seconds to do
this just think about
what you either experience maybe what
you witness day to day
ai and machine learning they're really
prevalent a lot i think
more than many of us are aware
so i'll just give you a few more seconds
to type your answer into the chat
just any example of ai or machine
learning
that you're aware of
so some examples are recommendations
online shopping centers whenever you see
those
you know based on your shopping cart you
might be interested in this
or based on what you watched you might
be interested in this
there's one example right there fraud
detection we've talked about this
a little bit already but financial
institutions using ai to teach computers
or have computers teach themselves what
constitutes fraudulent activity
what about facial recognition social
media sites or your phone
have you ever been on a social media
site and you get a tag that says hey
this
is you in a photo or this is your friend
in a photo
so we've talked about artificial
intelligence machine learning
deep learning where does data science
fit into all of this
so data science is a field that combines
tools and workflows
from disciplines like math and
statistics
computer science and business
and those tools and workflows are used
to process manage
and analyze data so data science is very
popular
in businesses today as a way to extract
insights from big data
to help inform business decisions how
does data science relate to the
techniques we just talked about
well machine learning deep learning
artificial intelligence those are common
tools in a data scientist toolbox
to help extract insights from data
what we've talked about so far is not an
exhaustive list
these are some of the most popular
techniques for working with big data
however
an important idea to note here is that
one of the benefits of using these
techniques particularly machine learning
and deep learning
is that they really help scale analytics
we've mentioned this already but
as you can imagine once machines learn
how to
detect patterns in our data they're able
to make predictions much faster
than human can and at a much larger
scale
so as we discussed all of these
techniques are used by data science
practitioners
to help extract insights from big data
they use these techniques as part of a
data science workflow
and that's a series of steps they follow
to process manage
and analyze data and we'll actually
take a look at the data science workflow
now
because now that we've established a
difference between ai
machine learning deep learning data
science
we should talk about what your data
teams are doing or what they can do with
these frameworks
the reason why we're going into the data
science workflow i think it's important
that we just chime in
and let you know you know why are we
talking about ai why are we talking
about the data science workflow
so when it comes time to make a decision
as to what technology or set of
technologies you need
to equip your teams with or when it
comes to creating a big data
organizational strategy it's important
to understand
what it is that each member of your team
does
because ultimately your choices are
going to affect how they work
and i know this is probably true of a
lot of people here i know i've had this
experience in the past that
your manager purchases a tool it's
supposed to help you do your work more
efficiently
sometimes they purchase these tools and
you had no input they didn't ask
right there was very little
communication with you and when you
start using the tool you realize
it actually doesn't help you with
everything you have to do
now you're left scrambling to try to
find out figure out
how to make up for the workflows that
the new tool doesn't cover
and what's more if it's not compatible
with tools you already use
then you might have an even bigger
problem
so again let's start talking about this
data science workflow
and this is the workflow that
individuals on data science teams
will typically work with as they're
working with big data
the steps in this data science workflow
can be broken down
into identifying business needs
ingesting data
preparing data analyzing data
and then sharing those data insights
so let's talk through each stage and why
it's needed
identifying business needs so this is
typically the first step
in the data science workflow it's the
first step that we'll discuss
in this step you typically have a
business leader
a manager for example it might be you
in your organization you come up with a
list of questions that you want answered
questions like should we make changes to
our product
which of our customers are at the
greatest risk for churn why
are they at the greatest risk for churn
maybe questions like
can we save money by changing the way
that we're pricing our products
in this phase of the data science
workflow managers or business leaders
are again identifying a set of questions
or business goals
that data practitioners will work to
solve
or will work towards
in the data ingestion phase your
organization is taking data in
there's often a combination of different
types of data
so you might have real time data which
comes in streams
for example a customer transaction that
gets added to your data store
every time a customer makes a purchase
you might have the continuous data from
a heart monitor or a fitness tracker
and other times you'll have data that's
ingested in batches we just talked about
this a few minutes ago
so chunks of data are being imported to
your data store at one time
an example of ingesting batch data could
be loading customer records
into your data store that might exist in
spreadsheets somewhere else maybe on
someone's local drive
for example at this stage at the data
ingestion
stage data is often raw
it's often messy then we go into this
data cleansing or data preparation
stage after your organization ingests
raw data
it needs to be prepared for use through
a deep cleaning
and preparation process that's often
referred to as data munging
munging data can mean anything like
cleaning it up
extracting data standardizing data
joining
consolidating or even filtering data the
point here
in the data cleansing and preparation
stage is that
you're preparing data enough so that
people using it
to for example train a machine learning
model or
generate a dashboard or a business
visualization
don't have to fix their input data or
don't have to spend a lot of time
fixing their input data
then we have this data analysis phase
during that data analysis phase
your teams are exploring this prepared
data
they want to find insights oftentimes
this is where
machine learning comes into play again
although machine learning isn't the only
type of data
analysis that can be applied to your
data it's becoming more and more popular
today
especially in conjunction with big data
aside from machine learning
individuals can also query data produce
visualizations etc
then we have the sharing phase sharing
insights so
once those business insights are
discovered they're often shared back to
you
if you're a manager perhaps another
stakeholder
who are going to use them to make
business decisions and that data is
typically shared through an
interactive dashboard it might be an
email it might be a presentation etc
so at a high level this is what your
data science teams are doing
with that big data
now we'll take a closer look at the
individuals
doing these things so who is responsible
for what
as we go through this section um
just think to yourself does my team look
like this
um especially in smaller companies we've
seen situations where
one person is fulfilling all of these
roles
but we do want to point out that these
are highly specialized roles
and as businesses grow they excuse me
they trend towards breaking out these
new worlds new
these roles into a variety of skill sets
some
organizations will break these roles out
even further but again
for the sake of this session we um just
want to show you an example
of these roles and these are the roles
that we see frequently
in the organizations that we work with
so let's start off with data engineers
data engineers develop construct test
and maintain data pipelines data
pipelines take data from its raw data
source
and they move it along to where it can
be used at different stages
of a data analytics or a machine
learning project
you can think of data engineers as sort
of your first line of defense
they are the ones that are taking that
raw messy data and performing
data munching to perform their duties
they use a set of tools to build and
maintain those pipelines
that can be anything from programming
languages like python or scala
different data storage solutions they
will also have different processing
engines so for example apache spark
data scientists then take that data
prepared by data engineers
and they'll use a variety of methods to
perform analytics
data scientists typically have a strong
background in disciplines like math or
statistics or computer science
and they often are tasked with building
machine learning models
testing those models and then keeping
track of their machine learning
experiments
to perform their duties they will use
tools like again programming languages
python r sql they'll use machine
learning libraries
and they'll use notebook interfaces so
if
you've ever heard of jupiter for example
then we have data analysts so data
analysts also take that data prepared
by data engineers to extract insights
typically a data analyst will also
present data
in the form of graphs it might be charts
dashboards
to stakeholders to help them make
business decisions
data analysts can also take advantage of
the work of machine learning engineers
to help derive
insights from data they're typically
well-versed in data visualization tools
and business intelligence concepts and
are often in charge of interpreting data
insights and effectively communicating
those findings
with their stakeholders to perform their
duties
uh data analysts often use sql
they'll also use visualization tools
like tableau power bi
looker and others
we also have platform administrators
platform administrators
um sometimes are called devops engineers
or infrastructure engineers
maybe cloud engineers but a lot of the
time they end up responsible for
managing and supporting big data
infrastructure
so that can be things like setting up
your organization's big data
infrastructure
updating maintaining and performing
health checks
on that infrastructure and also
providing governance
to data science team members around
changes or upgrades
to that infrastructure to perform their
duties
platform administrators often use tools
like the various infrastructure and
monitoring services
monitoring services that major cloud
providers offer
and that helps them keep data secure and
also scale and manage
their infrastructure
so while i was going through this i
asked you to think about you know does
this match
with my team does this seem familiar and
again we'll post in the chat i want to
know
are we missing any roles
can you think of anyone on your data
science team that doesn't fit
neatly into one of these roles
so go ahead and type that into the chat
maybe this aligned well with what you
see in your organizations
like we mentioned sometimes you have
people who do more than one thing
so just curious to see what those roles
are and how this aligns to what you're
seeing in your organization
so thank you to everyone typing those
answers into the chat
and now we will pivot and
talk about something that we've probably
all run into or at least
the data practitioners on our data
science teams have certainly run into
which are some of the challenges working
with big data
this is an important section because it
really helps at the stage for why
databricks was built in the first place
throughout this course we've made
reference to you know big data is
difficult to work with there's a lot of
different types and they have to be sort
in certain ways
so in this section we'll we'll dig into
that a little bit deeper and actually
give you examples of what we mean when
we say that
organizations have significant
challenges working with big data in ai
to kick this section off i do i just
want to ask you a question
um because i'm sure this is something
that everyone on the chat is familiar
with it's probably why you're in this
session
what are some of the challenges you've
seen organizations face
when it comes to big data big data
processing big data analytics
think about that question again write
your answer in the chat i'll give you
about 30 seconds
as you're writing in the chat i will
i'll give you some ideas
so maybe you've seen that even though
your team members know what they're
doing they're really talented they're
really smart
there just isn't enough of them to scale
right so you see some of these answers
coming through um
yeah there might be so many tools that
you can use that you just you don't know
where to start
so thank you to everyone posting their
answers in the chat
now in the early days of data analytics
simple relational databases historical
data
and spreadsheet expertise were used to
drive business decisions
and with the emergence of big data these
methods
are no longer sufficient today
businesses are spending
a large portion of time and resources
trying to piece together solutions for
how to extract insights from big data
and we already talked about this but
close to 90 percent of organizations are
failing
so why is this well challenge number one
we've said this multiple times working
with big data is not easy
it's very difficult to manage right so
we've talked about data coming in in
massive volumes
faster than ever before in a wide
variety of formats
and as data practitioners work to design
their organization's big data
infrastructure they're trying to answer
questions like
where are we going to store all of this
data how
are we going to store all of this data
how are we able to process
batch data with streaming data
how can we use different types of data
together in our analyses
so what if i want to combine
unstructured data with structured data
and then how can we keep track of all of
this work that we're doing
on our big data as you can imagine
there are many ways that an organization
can set up big data infrastructure
and getting it right is not easy what
frequently ends up happening
is that organizations will set up
different technology stacks
to handle their data workloads and often
these technology stacks don't work well
together due to
many different tools introduced that are
not compatible
or not easily compatible with each other
you might have proprietary data formats
that are not easy to translate from one
tool to another
and if you want to enable advanced use
cases
you might have to move across these
stacks so think about the work your
organization does
you probably don't have a data warehouse
use case
right or a streaming use case
you probably have a supply chain
logistics use case
or maybe it's a financial risk
assessment use case
so to address those challenges you need
to move across these technology stacks
we also have siloed roles that lead to
inefficiencies
so when we have these multiple stacks
set up
these multiple technology stacks set up
many organizations suffer from the
challenges
of having siloed functional roles for
individuals
on their data science teams what do we
mean by that so
it's not uncommon for a data scientist
to build and train a machine learning
model in a vacuum
on their computer with little to no
visibility
to related work being done by for
example
a data engineer who's preparing that
data for them
or a data analyst who might be using the
results from their experiments to
produce dashboards
and so that ultimately leads to
communication
downs you might have teams and this is
really common that you have teams
working with different versions of the
same data
this also leads to data security issues
and governance issues and
ultimately you will have a less
productive data team
and we talked about this a little bit
but definitely want to pull out data
security challenges
as its own section here because um
if we go back to we've referenced
gartner a few times in the session
already but
according to gartner 80 of organizations
will fail to develop a consolidated
security policy that leaves them and
their data vulnerable
to breaches security breaches and think
about the ramifications of a security
breach
beyond just the immediate monetary cost
there's a long lasting loss in customer
trust
and company reputation this has happened
to me
if you've ever been a customer of a
company that has suffered a security
breach
you know firsthand how long it can take
to rebuild trust
plus besides protecting data from just
leaking out
organizations also must ensure that
they're compliant
with data protection regulations we have
things like
gdpr so the european union's general
data protection regulation
we have the health insurance portability
and accountability act or hipaa
and one of the biggest issues
that organizations face and we have
referenced this
and this is again in addition to these
siloed roles and different technology
stacks being put together
security breaches etc is collecting
and storing data
it's incredibly complex as you can
imagine with data volumes
increasing data is becoming more
distributed
data types are changing setting up and
maintaining a big data infrastructure to
manage all of that data is
very complex and that's why today
many organizations struggle with their
data storage systems
particularly figuring out which
technology is right for them
and where to migrate their data to
in this session today we'll take a look
at data warehouses
data lakes and the lake house and the
lake house is the way that databricks
looks at the world
and we'll get to why in just a few
slides but first
let's start with data warehouses
so data warehouses have been around for
about 30 years
and if you work for a data-driven
organization
it's really likely that you're using a
data warehouse now
to power your company's decision-making
data warehouses
evolved as a set of guidelines to design
systems that control the flow of data
used in decision making and in many
companies
they delivered immediate benefits by
reducing the duplicated data found
in isolated data marts data marts which
had been developed for use
by a single department within the
company
and that contain highly specific views
of data
used to then generate reports or guide
business decisions the problem
is that if you have many departments
that need similar views
of data and if you work for a large
company
you might have dozens of data marts with
almost identical data
but then you have different naming
conventions you might have different
update frequencies
and those redundancies lead to high cost
for storage
and development and maintenance and
there's also confusion
right when you're trying to refer to
data that isn't managed by a central set
of rules
the data contained in data warehouses
comes from a variety of sources
generally referred to as operational
data and that's data that can be
anything that drives the functioning of
a company
so things like point of sale systems or
customer relationship management systems
or even inventory management systems
by combining data from diverse sources
into common tables
analytic insights can be used to drive
that decision making
so while glancing through a database or
a spreadsheet can provide a quick
glimpse
into the behavior of a given account for
example a data warehouse is going to
give you the ability to drill down
and look at seasonal trends in for
example how a subset of products have
performed
with a specific customer in mind
in various regions but why can't we
generate
reports like that from operational
databases
for example um how a product is
performing with an enterprise customer
in a specific region
so that really boils down to the design
and the intent
of those systems bi reports as you're
well aware can often take minutes
sometimes they take hours to generate
results and
data warehouses are designed to optimize
those queries they are optimized to
prevent
conflicts between concurrently running
queries
and data warehouses make the assumption
that the data that is entered
is unlikely to change with high
frequency so that's why if you aggregate
results over millions of records
data warehouses assume that those
records will not
change while they are being processed or
while they're being accessed
now let's go back to the retail point of
system
so imagine a retail pos
or a point of sale system for example
there are new sales that are constantly
being processed
and it's often the case that a recently
completed transaction
might need to be invalidated it might
need to be updated
let's say that a customer notices a
discrepancy on their receipt
trying to run a report on that type of
system
might result in inconsistencies in
results
it could prevent users from making
updates until a report is finished
now think about the front end how
frustrating that would be
for a customer and how expensive that
would be to a business
if a sales person is not able to
complete their transaction
or process a return anytime an analyst
runs a query on the back end
so that's where data warehouses come in
data warehouses help keep your
operations running smoothly
because they collect data from these
various systems your point of sale
systems your customer management systems
in a process called etl or elt
that's extract transform load or extract
load transform
basically there's a team of data
engineers and
database administrators and they define
how
data from different systems will be
processed and stored
they design that process so that bi
analysts can then count on data
in that data warehouse to be clean it's
validated
they know that it's being updated on a
consistent basis
and that kind of etl workload that
extract transform load workload
is vitally important to business
operations
any delays or errors in those processes
can lead to delays and reports or
inconsistent reports
etc now one thing to note here
is that data warehouses were originally
developed using
on-premises technologies so there
have had some pain points when it comes
to adapting to the modern data
atmosphere
we talked about this several minutes ago
but
the proliferation of smart devices the
proliferation of web-based commerce for
example
has really increased the volume and the
variety of data
data warehouses traditionally relied on
proprietary data formats
that means that they often do not
support videos
or images or free text files and
traditional data warehouse technologies
force customers to invest
in enough compute power and storage
to handle your peak user traffic
so as you can imagine as companies have
seen this exponential growth
in their data in recent years that kind
of design can become
extremely expensive
so then we have cloud-based data
warehouses and they have addressed some
of those challenges
because of the flexibility of the cloud
it's always possible
to scale up the amount of com computing
power
or the data storage that you need that
being said
many data warehouses still use compute
attached storage
so that means that scaling up and down
is not that easy
if you need to scale down the size of
your compute in storage for example that
can be difficult
so cloud-based data warehouses you know
a pro
is that you don't need to buy or
maintain the right number of
physical servers in order to scale up
but
you may find that your recurrent cloud
costs are growing exponentially
alongside of your data because of that
connection to the compute attached
storage
so another big point for data warehouses
and it's important we talk about this
one
is that they don't have native support
for machine learning applications
the databases and the tables that are
used in data warehouses
aren't well suited to working with
unstructured data things like user
comments things like images
we've talked about this in this course
and data warehouse vendors
generally don't allow outside
applications to directly access their
proprietary data formats
now machine learning algorithms we've
talked about this they need access to
huge amounts of data
neither the way that the data is stored
in a data warehouse
nor the query engine in a data warehouse
is optimized to return the data needed
by
data scientists in addition the open
source technology that powers a lot
of machine learning applications can't
actually communicate natively
with those proprietary data warehouses
so we've talked a little bit let's take
a pause here we've talked a little bit
about the pros and the cons
of data warehouses next we
will talk about data lakes and how they
evolved to address some of those issues
they've come into widespread use over
the last decade
data lakes are often defined in
opposition to data warehouses because
while a data warehouse delivers clean
structured data
for bi analytics a data lake allows an
organization to essentially store
anything they want
cheaply when we look at data warehouses
data warehouses require knowledge of
what data
will be used and how it will be used
before
any kind of engineering begins with a
data lake though
those decisions about how to use data
can be delayed by months or even years
and then uncovered later this uh process
is known as data mining so that process
of
analyzing these large amounts of
historical data and then figuring out
what are we going to do with this maybe
you uncover patterns or you find some
anomalies that may have predictive power
that is a core building block of many
successful machine learning applications
data lakes originally evolved as
an on-premise technology they were
closely tied to the apache
hadoop ecosystem i'm sure many of you
are are familiar with hadoop
many early internet companies had been
amassing these huge amounts of data
and they were seeking solutions to
process that data and monetize that data
that's where the hadoop distributed file
system or the hdfs
came in that allowed engineers to start
working with
data distributed across dozens
or hundreds of servers as if it were
part of the same machine
so what that meant is that that physical
infrastructure
was still expensive to purchase they
still had to purchase computers maintain
those
update those as well but that ability
to work with data distributed across
dozens or hundreds of servers
as if it were a part of the same machine
is what allowed
these big companies like facebook or
netflix or yahoo
to create and power these new
data-driven applications
there are a lot of companies in fact
your company might be
one that continues to run hdfs data
lakes on-premises
uh but most companies have transitioned
to working in the cloud or are in the
process
of transitioning to work in the cloud
each of the cloud vendors
offer their own take on object storage
it's particularly well suited for use as
a data lake
and there's a number of reasons for that
so the first is that
object storage doesn't rely on the same
relationship
between compute and storage seen in a
traditional
file system so if you break this down a
little bit
think about a file on your laptop when
the laptop is turned
off there's no way to gain access to
that file system and load the contents
right it's off
with object storage any computer that
has the unique address
for the data and the right security
privileges of course
can access it no matter where they are
so that means that any number of
computers can write to and read from
that shared object storage
simultaneously
and they don't need to worry about
running these expensive computing
resources
in order to keep that data available the
best part is that there's no
upfront investment in infrastructure or
in contingency planning for lost data
because
all of that is provided by the
individual cloud vendors
and while prices are obviously going to
vary by cloud and the specific terms of
you know a contract or your agreement
will vary by cloud
you are going to see a huge decrease in
monthly cost typically so
usually the monthly cost is around two
cents per gigabytes of data so it's
extremely cost effective
now a huge point here and something that
is uh the opposite of a data warehouse
is that
data lakes have the benefit of
supporting any kind of data
regardless of structure regardless of
format
as we mentioned before that's very
important for running machine learning
applications
in a data lake design data storage and
compute
are not coupled and so there are a
number of technologies that have evolved
around working with data stored in that
type of way
apache spark is one of those
technologies
it is the compute engine used by
databricks one thing to note and this is
an important distinction between data
lakes and data warehouses
a data warehouse is a specific
technology
supported by a vendor that's built
around a proprietary format
and a query engine a data lake
is more of a service model for
delivering data
regardless of the location of the data
lake the end user has a number of
options for the compute engines that can
process that data efficiently
and data will be stored in open formats
and that makes a migration of that data
simple
so it sounds like data lakes came around
they are the solution
to the traditional data warehouse
however for all of their strengths
they have not completely eliminated the
need for data warehouses
for bi analytics specifically most
companies continue to load
the relevant data from their data lakes
into a data warehouse
and it's in that data warehouse that
analysts execute queries
and generate reports so let's look at
this a little bit closer here
is a diagram that represents a common
pattern we see in the field
so you have data from numerous sources
landing in a data lake
a lot of that data will be stored here
indefinitely waiting for
the organization to figure out what to
do with it later on perhaps
for use in an ml application machine
learning application
an etl pipeline will load some of that
data from the data lake
into a data warehouse and in that data
warehouse is where the quality of that
data is
checked an analyst can have that
confidence
that that data is ready to be used to
derive insights
data scientists who want to work with
this data have two options at this point
they can either go back to the data lake
and implement new logic
to check and enforce the quality of that
data
or they can run additional etl
jobs to get data out of the data
warehouse
back into an open format so then it can
be used in machine learning
and that probably sounded really
convoluted and confusing but that's
why neither of these options is ideal
because it leads to a lot of data
redundancies
and a lot of additional engineering work
it also means that there could be delays
in data freshness or even a reduction in
data quality
what we see today is that many companies
and this might be true in your company
choose to use both a data lake and a
data warehouse in their architecture
and i do want to be clear that when
we're talking about data lakes
and data warehouses we are relying on
two assumptions here
one is that you have a data lake that is
implemented
using standard open source formats
without transactional processing
we're also assuming that data warehouses
are relying on data storage
in a proprietary format on an attached
storage volume
that attached storage volume is also
known as block storage
so if you remember uh maybe 10 15
minutes ago we described
object storage object storage is
that cheap cloud storage that allows
data lakes to remove
compute costs from storage costs
so the decoupling of that compute and
storage
cloud vendors will also offer block
storage
that is similar to if you think about uh
renting a portable hard drive in the
cloud
so you might be able to change the
computer
that the hard drive is connected to but
unless a machine is powered on
and running the right software it's not
possible to get your data
some cloud data warehouses have adapted
a hybrid model
and so you can actually leverage object
storage to cheaply store your data
that you use infrequently and then you
would only load it
into the more expensive block storage
when you need it
we also have some vendors that have gone
one step further and they
allow direct querying of data stored in
open source formats in a data lake and
this does solve some of the issues with
data warehouses
again it is not it does not provide an
answer for
those machine learning workloads
or working with truly unstructured data
like an image
at databricks we approached this issue a
little differently
so we've created the idea of a lake
house
essentially combining the best of both
worlds
so the data reliability in governance
that you get from a data warehouse but
also that flexibility
to store and work with different types
of data and different formats that come
with a data lake
we'll review more about the lake house
as we
dive into the databricks lake house
platform
so before we move on i've done a lot of
talking i want to make sure people have
the chance to ask questions please make
sure that you're
asking your questions in the chat so
that rtas can answer those for you
i also want to do a bit of a knowledge
check here because
we have talked about big data we've
talked about store ways to store that
big data ways to analyze that big data
so i'm going to go ahead and ask you a
couple of questions here
i'm going to read off some comments that
you've probably heard in the past
and then in the chat i'd like you to
come up with why
these statements either make sense great
good to go
or they do not sit well with you so
again please post your answers in the
chat
so statement number one in the long run
scaling up our existing on-premises
infrastructure
will be less expensive since we'll own
the machines
so go ahead does that sit well with you
do you think that's a fair statement
do you have any issues with that
statement and if you do
please go ahead and let us know why you
have an issue with that statement
you can go ahead and add that into the
chat
i'll give you about 10-15 seconds here
so
again in the long run scaling up our
existing on-premises infrastructure
will be less expensive since we'll own
the machine
go ahead and type your answers or
continue to type your answers into the
chat
so i am i'm going to say that's not
sitting well with me
um i think about the cost of owning a
machine
i'll have to maintain it i'll have to
make sure that they're upgraded
from time to time um anytime there's new
software that comes in i'll have to make
those updates
um as technology changes etc i have to
get staff
now to maintain and secure those
machines especially
you know i'm not going to have one i'm
probably going to have several or dozens
of machines so
i'm going to say this one is not sitting
well with me
the next comment you hear to help
streamline
our workflows we should make sure that
the technologies
staff are using are compatible with each
other
so again does that sit well with you
go ahead and let us know in the chat i
think this one's pretty straightforward
to help streamline workflows absolutely
you do want your
the technologies that staff are using to
be compatible with each other
as much as possible what about this one
to ensure data protection it's a good
idea to invest
in independent technologies that each
specialize in a specific part of our
data workflows
and can each be protected separately
so think about that one that's an
interesting question or an interesting
statement
so go ahead and type your answers into
the chat again that is
to ensure data protection it's a good
idea to invest
in independent technologies that each
specialize in a specific part of our
data workflows
and can each be protected separately
so as your answers come in i'm going to
say that's not necessarily a good idea
because if that's the path you choose
then you have to deal with different
tools
different methods of security trying to
coordinate users between those tools
and in this course we did talk about
those siloed technology stacks
the more tools you have the more you
have to think about
whether they're compatible to each other
for example how to get them to talk to
one another
so great job on that um now let's do a
little bit of review
when it comes to data lakes and data
warehouses
i'm going to read a list of descriptions
you tell me
and you can post this in the chat or
just sit there at home and
think about this but tell me if the
description describes
a data lake or a data warehouse
so if we're thinking about
a technology that is compatible with
unstructured data
think about it that would be a data lake
congratulations to those who got that
correct what about the ability to hold
any amount of data at a low cost
so this technology can scale to hold any
amount of data at a low cost
that again would be a data lake
uh what about um the user interface of
this technology
is familiar to users of traditional
databases
the user interface is familiar to users
of traditional databases
that is going to be a data warehouse
and then one last one which of these is
typically associated with
vendor lock-in
so that means that your data resides
with them
with the vendor
all right that one is going to be a data
warehouse
yep data warehouse associated with
vendor lock-in
okay great so let's go ahead
and move on we are getting to the point
of the course now where we have talked
about
the big data benefits for business we've
looked at big data characteristics
we've looked at big data types big data
processing we talked about batch data we
talked about streaming data
we've talked about artificial
intelligence machine learning
other types of analytics that are used
in analyzing
big data we looked at the data science
workflow
and the roles on a data science team as
they take big data through that data
science workflow
in this last section we looked at some
of the struggles with big data and ai
that many organizations are facing
we took a look at what happens when you
have siloed technology stacks
siloed roles on your data science teams
some issues that organizations face with
data security
and making sure that you're being
compliant with um
laws like gdpr making sure that you have
the right
security certifications you need to run
your businesses
now we're going to switch over and start
talking about databricks the moment has
come
so in our next session we're going to
take everything we've learned so far to
help us understand
what databricks actually is what that
lake house platform does
um and then again we'll take a look at
some real
world customer examples customer success
stories of customers that are using
databricks
before doing that we'll take a 15 minute
break
so please take a look at the time right
now and we will meet back
in 15 minutes i'll give you a reminder
when we have 10 minutes left
and another reminder when we have five
minutes left see you soon
all right we have 10 minutes left of our
break
and here's your five minute warning we
have five minutes left on our break and
then we'll go ahead and get started
again
okay so we're back uh ready to talk
about databricks and the lake house
platform
databricks is the data and ai company
it makes big data and ai easier for
organizations to manage
and the whole point behind what
databricks is and what we want to help
our customers do is enable that
data-driven
innovation in every enterprise
a simple way to describe the databricks
like house platform
is that it empowers everyone on a data
science team to work together
in one secure platform from the minute
that data is ingested
into an organization through when it's
cleaned up
when it's analyzed and when it's used to
inform those business decisions
so the databricks lake house platform
will allow organizations to
ingest process and transform big data
massive quantities and types of data
explore data through
data science techniques including but
not limited to
machine learning for example it will
also guarantee that data available for
business queries
is reliable and the most up-to-date data
an organization has
it also provides data engineers data
scientists data analysts
the unique tools they need to do their
work and
it helps organize and we've talked about
this but it helps organizations overcome
those traditional challenges associated
with data science and machine learning
workflows
and that's what we will explore in
detail in this lesson
so how this works so basically uh data
practitioners access databricks
functionality
using a custom-built web-based interface
uh this is an environment for accessing
all of your databricks assets
so things like notebooks you might have
libraries
you might have dashboards for example
and on top of accessing all of those
assets that you're using day to day
or that data practitioners are using day
to day
it also is how you will access the
computational resources you need to
process
your data and to process those queries
so while you might be using different
tools or functionality within the
databricks lake house platform
depending on your role everyone is
working in one platform and that's key
so here's an example of one of the user
interfaces you can use when you're using
databricks this is the databricks
workspace
when you log in this is what data
practitioners will see
and then they'll use different
functionality based on their roles
another concept we want to bring up now
and this is specific to the lake house
is that the data in a lake house is in
an open format
databricks invented some of the most
successful open source projects in the
world
you can see those in this image here and
we continue to contribute to those
projects regularly
when your organization uses databricks
you're getting managed
access to those open source data
projects through the platform
plus you are unifying your big data
system
with open source standards and by
aligning to open source standards
you're basically establishing building
blocks that help make your applications
more
interoperable with different systems
so here's a conceptual image of the
databricks lake house platform
what you see here is that we have
different types of data
coming into the platform we operate
across the major cloud vendors if you
have questions please let us know in the
chat
so we have all of this different type of
data
coming into the platform that's
structured semi-structured unstructured
and streaming data
and that data is landing in an open data
lake
on top of that data lake is delta lake
so you see this logo here delta lake
we'll talk about delta lake in a few
minutes
on top of delta lake are these
use cases including things like data
engineering
bi and sql analytics real-time data
applications and then of course your
data science and your machine learning
use cases
so we have two ideas to really unpack
here the lake house
which is at the heart of this platform
and delta lake
as we mentioned before lake house is
this new
open data management paradigm and it
combines
the most popular capabilities of data
lakes and data warehouses
per their design they implement similar
data structures
and data management features to those in
a data warehouse
but they do that directly on the kind of
the low-cost storage used for data lakes
so when you merge these ideas into one
single system
it means that data teams can move faster
because they can use data without
accessing multiple systems
and teams also have the most complete
and up-to-date
information or data available for
these different use cases like houses
have the following key features
so they have transaction support that
means that multiple parties
can concurrently read or write data
they have data schema enforcement that
helps ensure data integrity
so if you have a right to a table for
example
that right will be rejected if it does
not match the table schema
like houses also come with governance
and
auditing mechanisms to make sure that
you can see how data is being used
there's also bi support so bi tools can
work directly on
source data right on that freshest most
reliable data
and that really helps to reduce data
stillness huge factor here and we've
talked about this so
the idea that storage is decoupled from
compute
that means it's easier for your system
to scale to more concurrent users
and to different data sizes
this idea of openness so storage formats
used are open
they're standard plus there are apis and
various other tools that make it easy
for team members to access data directly
and there's support for all data types
structured
unstructured semi-structured streaming
etc
there's end-to-end streaming with the
lake house so real-time reporting and
real-time data can be integrated
into data analytics processes just like
existing or batch data is
we also talked about delta lake and so
now let's talk about
the relationship between delta lake on
data bricks
and a lake house that's a question that
we get a lot it's really
key to understanding what it is that the
data bricks lake house platform does
and how these components fit together so
first off
delta lake was developed at databricks
it was open source in early 2019
and delta lake on data bricks is a term
that we use for the managed version
of delta lake that comes with the data
bricks lake house platform
so i'm just going to refer to it to
delta lake but i do want to make that
distinction
with delta lake you are bringing
structure
and governance inherent to data
warehouses
to data lakes and that is what creates
the foundation
for a lake house when data practitioners
use
dell to lake they tend to follow this
architecture pattern that consists
of organizing data that's ingested into
your organization
into successively cleaner tables
starting with this idea of a bronze
table
moving into a silver table and then
stopping at a gold table
we're going to dive into this just a
little bit this won't be a super
technical discussion but
it is important to understand what's
happening on the back end because
the way that data is organized using
delta lake affects
how your downstream users in your
organization will interact with and use
that data
so we start with this bronze table this
bronze table contains
raw data ingested from various sources
it can be iot data for example maybe
their json files for example
data in those bronze tables are often
kept for years
this is what we talked about that an
organization can keep it in that table
for years until they decide what to do
with it
that data is kept as is so in other
words it's kept in its raw state
then we move into a silver table a
silver table is going to provide a more
refined view
of your organization's data that data
this data in a silver table is directly
queryable
and ready for big data and your ai
projects
data in that silver table is often
joined from various bronze tables
that's done to enrich records it can be
to update records based on recent
activity
but data in the silver table is clean
and it's normalized
and this is what we refer to as that
single source of truth
we also talked about the idea of data
marts
goal tables are going to provide that
business level
aggregate used for your particular use
cases for example
goal tables are what are going to
provide the business level aggregates
used for
particular use cases so for example if
the sales department
for an online retailer needs to create a
weekly dashboard for a specific set of
data every week
in that case a data engineer can prepare
this data
for that department and include
aggregations such as daily active
website users maybe they include
weekly sales per store maybe they
include gross revenue
per quarter for that department the data
stored in gold tables
are usually related to one particular
business objective
so there are several elements that are
sort of housed under this delta lake
umbrella
and what we'll review now are those
elements there's four of them
we have delta files you have delta
tables
the delta optimization engine and then
you have that delta lake storage layer
so first up we have delta files by
design
delta lake uses parquet files to store
an
organization's data in their object
storage
parquet files are a state-of-the-art
file format for keeping tabular data
they're fast
and they're considered more powerful
than traditional methods
for storing tabular data because they
store data using columns
instead of rows and all this means is
that when you're querying
this columnar storage allows you to skip
over non-relevant data very quickly
so your results are going to take
considerably less time to execute
delta files leverage all of the
technical capabilities
of parquet files so they're fast but
they also have an additional layer
over them and that layer is going to
track data versioning
and metadata and it also stores
transaction logs
so that you can keep track of changes
made to a data table
or to an object storage directory and
it's also going to provide acid
transactions which we'll talk about in
just a little bit
so then we have delta tables a couple of
things about delta tables so delta
tables are
collections of data kept using delta
lake technology
those delta tables will have delta files
that contain
the data your data and kept in object
storage
tables are also registered in a
metastore
a meta store being basically a catalog
that tracks
your data's metadata so the data about
your data
the delta tables will also have that
delta transaction log
that is saved with those delta files in
that object storage
then we have delta optimization engine
so delta engine it is a highly
performant query engine it provides an
efficient way to
process data in your data lakes and it
accelerates data lake operations and
also supports a variety of workloads
those can be large scale etl processing
workloads
they can also be ad hoc interactive
queries
and many of these optimizations just
take place automatically
so you get the benefits of delta engine
capabilities
by just using data bricks for your data
lakes
what the delta optimization engine means
for your business is that your data
workloads run faster
then we have this delta lake storage
layer when using delta lake
your organization stores its data in a
delta lake storage layer
and it accesses that data via data a key
idea here is that
an organization keeps all of this data
in files in object storage and it's
beneficial
because that means that your data is
kept in a lower cost
and easily scalable environment so
what we'll do now is review how
deltalate guarantees reliable data
part of diving into how
delta lake brings this reliability and
this governance to your data lakes
requires
going through some of the issues that
organizations face with data lakes
so one thing with data lakes is that
data can be hard to append
so what that means is that while it's
easy to write data
to a data lake from many sources
and then read that data with many
different systems
if you have an ongoing data job you
might need to write dozens
or hundreds of files before a
transaction completes
what can happen is that a query might
capture a partially written data set
and then that can lead to invalid
results with data lakes you can also
have jobs that fail
midway so when
you um are updating a data lake
you might have numerous files that are
written one after another
in that update if for whatever reason
that update fails
after some of the files have been
written it's difficult to recover
where you actually are what is the
correct state of that data
and oftentimes data engineers will spend
countless hours
troubleshooting failed jobs and trying
to figure out
what actually moved over what is the
valid state of this data
something else another challenge with
data lakes is that modifying existing
data can be difficult
so data lakes are designed to have
data written once and then read many
times
the concept of modifying or deleting
data
wasn't really of central importance at
the time they were designed
but now with regulatory measures like
gdpr
ccpa for example companies have to be
able
to identify and delete a user's data if
they request for that to happen
if the user doesn't want them to have
their data anymore or use that data in a
certain way for example
we briefly touched on this when we
talked about streaming data but
real-time data is difficult to leverage
um
even with data lake so many
organizations struggle to
implement a solution that will allow
them to leverage
their real-time data with their
historical records
a lot of the time that means that you
have to keep multiple systems
and multiple pipelines in sync in order
to be able to do that
that can also be that can be expensive
it's also difficult
um to manage and there can be some
consistency problems that just take a
lot of time from your data engineering
teams
in addition keeping historical versions
of data can be quite costly in a data
lake some industries
have strict regulations for data
reproducibility
and they require historical versions of
data to be maintained
with the huge scale of data present in a
data lake
it can be pretty expensive and difficult
to archive all of this data for
compliance it's also difficult to handle
large metadata
so we talked about this metadata being
the data about your data
if your organization has truly massive
data
reading that metadata can often create
substantial delays in processing
and overhead costs because there's just
so much of it
and then we have this too many files
cause problems so
some applications will write small files
out to data lakes and over time
you might have millions of files that
build up when that happens
your query performance is going to
suffer because there's a small
amount of data that's spread over so
many files
and then all of those files are being
read just to get to that data
you also have data quality issues and
that will affect your analysis so we've
talked about this but
data lakes don't have built-in
data quality and anticipating possible
issues
if the proper checks are not executed
that can be quite expensive you might
have long-running queries
queries that fail queries that produce
meaningless results etc
so right there we just took a more
technical dive into
exactly what is going on when we talk
about challenges with data lakes
and those are the biggest ones that
we've seen when we work with customers
in the field
so how does delta lake address those
pain points
one of the biggest benefits the most
powerful benefits of using delta lake
is that you get acid transactions it
adds
acid transactions to your data lakes
acid refers to a standard set of
guarantees
that most databases are actually
designed around
since most data lakes have multiple
pipelines that read and write data
at the same time data engineers often
spend a significant amount of time
to make sure that data remains reliable
during these transactions with asset
transactions
each transaction is handled as having a
distinct beginning
and an end that means that data
in a table is not updated until a
transaction successfully completes
and each transaction will either succeed
or fail
fully so in other words what that
transactional guarantee
does is it eliminates a lot of the
motivations for having both
that data lake and the data warehouse in
an architecture
appending data becomes much easier
because each new write
will create a new version of a data
table
and that new data will not be read until
that transaction completes
so again if a data job fails midway
it's disregarded entirely this also
simplifies the process of deleting and
updating records
because many changes can be applied to
data in one single transaction
and that eliminates a possibility of
incomplete deletes or incomplete data
updates
besides asset transactions we also
should talk about
schema management so delta late gives
you the ability to specify
and enforce your data schema it
automatically validates that the schema
of the data being written
is compatible with the schema of the
table it's being written
into so columns that are present
in the table but are not in the data are
set to null
and if there are extra columns in the
data that are not present in the table
the operation will throw an exception
what that does
is ensures that bad data that could
corrupt your system
is not written into those tables and
with delta lake
you can also make changes to a table
schema and then apply that automatically
there's another concept of scalable
metadata handling
with delta lake metadata is processed
just like regular data with distributed
processing
which makes things a lot faster then
unified batch and streaming data we've
mentioned this several times throughout
this course
so delta lake is designed from the
ground up to allow
one single system to support both your
batch
and your stream processing of data those
transactional guarantees of delta lake
mean that each micro batch
transaction will create a new version of
a table
that is instantly available for insights
so many databricks users use delta lake
to transform the
update frequency of their dashboards and
their reports
from days to minutes while eliminating
that need for multiple systems
we also have this idea of data
versioning and time travel
so with delta lake the transaction logs
used to ensure acid compliance
create an auditable history
of every version of a table and it
indicates which files have changed
between versions
that log makes it easy to retain
historical versions
of the data to fulfill compliance
requirements in various industries
again looking at things like gdpr and
ccpa
so in summary all of that functionality
we reviewed
helps organizations streamline these
operations
they are improving their data pipelines
they are unifying real-time
and batch data they're enabling the
performance of
bi workloads on directly on your data
lake
and it's enabling you to meet regulatory
needs
so delta lake and this lake house
platform enables
allows your data engineers to make sure
that
all of the data being used by downstream
users is that refined data that they
need
when we think about the databricks lake
house platform for
bi and for sql analytics we have
databrick sql analytics that is a part
of the
databricks like house platform that is a
sql native interface that provides sql
users the ability to write queries
in a familiar syntax and explore their
organization's
delta lake something really great about
sql analytics is that
regularly used sql code can be saved as
snippets
and so those sql analysts can quickly
reuse that sql code
query results can also be cached and so
that
will keep the query shorter once queries
are built
sql analytics also enables your data
analyst to make sense of those results
through a wide range of visualizations
those visualizations can be organized
into dashboards through
a drag and drop interface it's really
easy to use
and those dashboards can be shared with
other users to provide
insights they're also configurable so
they can automatically refresh and alert
your team to
any meaningful changes in your data a
key point here
is that sql analytics was designed to
work with delta lake
so you're getting all the performance
optimization
the governance the reliability of that
data
in your data lake but you're able to use
it with the sql native interface
plus the platform also supports existing
bi tools
so if for example your sql analysts have
a preferred visualization tool
they want to use to query your
organization's like house
they can use those preferred
visualization tools then
when it comes to the data lake house
platform for
data science and machine learning data
science and machine learning teams
use the data bricks like house platform
throughout the entire data science and
machine learning life
cycles they use a platform to explore
data
prepare and process that data build and
test machine learning models
put those models into deployment
optimize those models etc
that is done through various platform
components that we won't get into
specifically or technically right now
but things like
collaborative notebooks the databricks
machine learning runtime
we have managed ml flow we'll discuss
those
briefly here so collaborative notebooks
are web-based interfaces they contain
runnable code
they contain visualizations they can
contain narrative text
there those notebooks are used in data
science and machine
learning workflows to perform
exploratory data analysis
to clean up data to build machine
learning models
they support multiple programming
languages so
languages like sql scala r python
the machine learning runtime this will
provide data scientists
and machine learning practitioners with
scalable
computing resources that come with
built-in
popular data science frameworks we also
should mention managed ml flow that is
built on top
of ml flow which is an open source
platform developed by databricks
that open source platform was developed
to help data scientists
and machine learning practitioners
manage the complete
machine learning life cycle with ml flow
you can track machine learning
experiments
you can manage machine learning models
and you can also deploy
those models when data practitioners are
working
with the databricks lake house platform
they're sharing tools
found within the platform but they're
also sharing all of the assets that you
create
anything that is created inside of that
platform
whether they are machine learning models
prepared by a data scientist whether
it's a dashboard
prepared by a sql analyst
maybe a data engineer is working to
ingest some data he has some code or she
has some code in a notebook that they
want to share
perhaps there's some data sets that are
relevant to everyone on a team
all of those assets are shared with
everyone working within that platform so
that has been a high-level overview of
the databricks lake house platform
at this point basically everything we've
talked about in this course has come
together
so we started with this what is big data
what are the structures
what are the types of big data how is it
processed how do data teams work with it
um what are some of the techniques used
to work with it and then we came into
how all of those concepts are tackled
with the databx lake house platform what
we'll do now
is um give everyone a chance to just
look at some success stories from
databricks customers
before we jump into those customer
stories let's do
a knowledge check
so i'll post some questions here go
ahead and answer them in the chat
if you would that would be great our
first question here
and these will be about the material we
just covered so we'll be looking at
databricks and the lake house platform
so
in which of the following stages of the
data science workflow
do organizations use data breaks for big
data
and ai data ingestion
data preparation data analytics
or sharing data so go ahead and type
your answers into the chat
so the answer here is that organizations
use databricks for all
of these stages of the data science
workflow
organizations and data practitioners
within those organizations
use data breaks from the moment that
data is ingested
all the way through that sharing
insights sharing data insight stage
all right great job next question true
or false the databricks lake house
platform comes with different user
interfaces
tailored to data practitioner roles
true or false go ahead and put your
answers
in the chat
so that's going to be true and we
actually covered a couple of those
in this course so far for
extra points here can anyone remember
what those user interfaces were that we
covered in this course so far
so one is the workspace the databricks
workspace
there was one other one that we covered
that was the one that was a sql native
interface
databrick sql analytics
okay next question i'm curious if anyone
can name
some of the open source projects created
by databricks
that you have access to via the
databricks lake house platform
i showed a slide several minutes ago and
it had
some of these project on there and we
have talked about some of them so i'm
just curious
so which are some of the open source
projects created by databricks
that you have access to via the
databricks lake house platform
all right switch share this slide again
here's several of them
apache spark delta lake mlflow
in re-dash we did not talk about in this
course so far but those are some
additional open source projects
built by databricks
okay the databricks like house platform
is best suited for
which types of data
so in this course we've talked about
several types of data
go ahead and type your answer into the
chat the databricks like house platform
is best suited for which types of data
and of course the answer there will be
for all types of data so we've talked
about unstructured
semi-structured that structured data and
also streaming data
great job everyone let's keep going just
a few more questions
what are some of the benefits that the
lake house
brings to your organizations
so i'll give you a few hints here one is
transaction support
the other is bi support
this one might be a little tricky but
there was a slide i showed uh
several minutes ago and then we talked
through some of these benefits
so go ahead and drop your answers or
type your answers into the chat
so i'll bring this slide back up so that
we can just take a look really quickly
so
we talked about this transaction support
bi support
there's also that data schema
enforcement
it brings that governance and that
auditing capabilities to your data
this one's important so if you remember
we talked about storage
being decoupled from compute
using storage formats that are open to
make sure that your data and your system
is
interoperable with other systems
then we just reviewed this as well that
support for all data types
and this end-to-end streaming
all right which of the following lays
the foundation
for your lake house we have a few
choices here
delta lake apache spark
mlflow or the workspace
which of the following lays the
foundation for your lake house
go ahead and add your answers to the
chat
and the answer is going to be a delta
lake
congratulations to everyone who got that
one right delta lake lays the foundation
for your lake house
all right one more question uh thinking
about delta lake this time
so we talked about that delta lake
architecture of
landing data and successively cleaner
tables
which of the following tables is
considered an organization's
single source of truth so your options
here
bronze that bronze table the silver
table
and the gold table
let's go ahead and answer in the chat
and the answer here's going to be that
silver table
we also talked about organizations
potentially needing this business level
aggregate
data does anyone remember which table
that type of data would be found in
would it be the bronze the silver or the
gold
it's that business level aggregate data
that would be that gold table
so it's taking data from that silver
table and then setting it up and further
refining it for those specific
business use cases that typically have
uh one objective tied to them
one business objective tied to them all
right great
thanks everyone for your participation
next we're gonna take
one more break and when we come back
from that break we'll finish up the
course by looking at three of our
fantastic customers that are using
the databricks lake house platform to
streamline their big data and ai
workflows this will be a 15-minute break
one more 15-minute break and then we'll
finish up the course
all right we have 10 minutes left of our
break
and here's your five minute warning we
have five minutes left on our break and
then we'll go ahead and get started
again
here are some of the outcomes in a
variety of industries that we've seen
value can be quantified in terms of
productivity gains
cost saved revenue increases etc
the results can be pretty dramatic as
seen here
and we'll also look at some numbers in
the following customer examples
so we'll start off by looking at
h m so h m is a huge
innovator in the fashion and the retail
industry they rely on data
as the core for everything they do and
they have stores
opening up globally at a really rapid
pace
h m needed to improve their supply chain
and forecasting operations
to streamline their costs and also
maximize the revenue
they were working with an on-premises
hadoop system
and that older system was really
crippling
their ability to ingest and analyze data
being generated by
millions of customers and they needed
that data
to power their predictive machine
learning models
we're talking about massive volumes of
data
from over 5000 stores in over 70 markets
with millions of customers and that's
every single day
h m wanted to make use of all of that
data for ai
but again that legacy hadoop based
architecture was not sufficient for that
they just weren't able to scale to meet
their business requirements so
at h m data engineering was challenged
because they had
a fixed number of computing resources
that they could use
and they also had a complex
infrastructure which was really resource
intensive it was costly to scale and
they did
suffer from some data security issues
data sources were siloed and that
impacted the data science team's
productivity
and because of all of that it would take
the data science team
a whole year sometimes to go from model
ideation
to productionalization so databricks was
able to address
all of these challenges right out of the
box the lakehouse platform as we
mentioned
is a fully managed platform it
simplified
managing those compute resources it
simplified
infrastructure management and operations
at scale
and this collaborative notebook
environment because it supports multiple
languages
meant that a diverse team of users could
work together
in their preferred languages which meant
there was this
unification across a team environment to
fuel productivity now you can have the
data science teams working with their
data engineering teams
all working in their preferred language
and all working in one place
when h m moved over to databricks it
improved their operational efficiency
it actually reduced their operational
cost by about 70
the lake house platform dramatically
reduced the number of components needed
for data scientists and engineers to
work together and overall this had a
huge business impact
with faster time to insight so they had
this ability to be more granular
in their decision making all right we
also have a regeneron
regeneron takes genomic data
and they bring new medicines to patients
in need
one of the issues regeneron had was that
they had poor processing performance
and limitations when it came to
scalability and so
their data teams could not analyze the
massive amounts of data
of genomic and clinical data that they
had more than 95 percent
of all experimental medicines that are
currently in
the drug development pipeline are
actually expected to fail
to improve those efforts the regeneron
genetic center
built one of the most comprehensive
genetics databases
by pairing sequence exomes and
electronic health records
of more than 400 000 people and it was
a massive massive massive data set
one of the challenges that um regeneron
face is that
genomic and clinical data is highly
decentralized
so it's really difficult to analyze or
train models against again that entire
data set that was 10 terabytes of data
it was also difficult and very expensive
to scale their legacy
architecture to support analytics or
with a 10 terabyte data set
they were looking at over 80 billion
data points
they had data teams that were spending
days just trying to prepare data
so it can be it could be used for
analytics regeneron
uh choose to use databricks they
simplify their operations
to accelerate their drug discovery and
it really
improved coming over to databricks
really improved their data science
productivity
so some of the concrete outcomes the
databricks lake house platform reduced
the time it took
data scientists and computational
biologists
to run queries on their entire data set
from 30 minutes down to three seconds
which is a 600 times improvement
etl could be done in two days
versus three weeks and as you can
imagine
lowering that overhead has enabled their
teams to support
a broader range of studies now so
instead of spending so much time
just managing the overhead they can now
focus
more on these projects that can help
bring
treatments and medicines to people that
need it and then finally comcast
so comcast was working on a voice remote
customers could ask the remote for
personalized programming recommendations
as you can imagine that is a huge
amount of data if we dig into this a
little bit more let's just think about
what it means
for a voice remote to work so what what
happens is that
the remote is instantly answering a
customer's voice request for a
particular program
it's turning billions of individual
interactions
into actionable insights and that
strain comcast it infrastructure and
their data analytics and data science
teams
on top of that comcast needed to deploy
models to a disparate range of
environments so they had cloud they had
on-premises environments
and even directly to some devices in
some instances
so they had some very specific
challenges billions of events were
generated
by their entertainment system they had
over
20 plus million voice remotes and so
again that's where that petabytes of
data
originated that needed to be analyzed
they had poor collaboration
so they had a data scientist working
globally and working
in different programming languages it
was often difficult to share and reuse
code
which added overhead in time to
development efforts
and then the management of machine
learning models like i mentioned
they were developing training and
deploying hundreds of models
very manual it was very slow difficult
to replicate
and so it made it difficult to scale
those models to millions of customers
comcast moved over to databricks as i
mentioned before
with the databricks like house platform
you have an automated infrastructure
and delta lake made it possible to
create these more robust and faster data
pipelines
with databricks comcast built out these
rich data sets
to optimize their machine learning at
scale they streamline their workflows
across their teams
it improved collaboration reduce the
complexity of their infrastructure
and help deliver this awesome experience
to their customers
deltalake enabled them to optimize files
and that way
they had a much faster and reliable
ingestion of those files at scale
of that data at scale so again looking
at the numbers with databricks comcast
reduced their compute costs by about 10
times
delta lake enabled comcast to optimize
their data ingestion
it replaced 640 machines with 64
and at the same time it improved
performance
now that means that team members spend
more time on analytics and less time on
managing all of that infrastructure
prior to using data bricks it would take
approximately
five full-time devops employees to
onboard
200 users and that dropped down from 5
to .5
delta like also enabled the data team to
use data at any point within
that data pipeline created by their data
engineers
that just allowed them to act more
quickly when it came to building and
training those models
and it was faster to deploy those models
so
deployment times for models went from
weeks to minutes
with that we've come to the end of this
session and we'll leave you with this
so today more than 5 billion customers
interact with data every day
by 2025 that number will be 6 billion
or about 75 percent of the world's
population
and that's coming from an idc white
paper
and by 2022 more than half of major new
business systems will incorporate
continuous intelligence
that uses real-time context data to
improve their decisions
this is coming from gartner so today
as we saw throughout this course the
promise of big data in ai has never been
more achievable
putting it into practice is another
story that has never been more
challenging
you know the success of your
organization's analytics processes
hinges on knowing one the challenges
that lay ahead
so that you can avoid some of the common
pitfalls
that other organizations have faced and
then a huge one here is
making sure that you're using the right
technologies that can scale
with your business so we hope that you
enjoyed this session
we hope that at this point you feel
comfortable talking about high-level
concepts related to
the big data landscape just to wrap up
here in this session we talked about
big data benefits for business we talked
about big data characteristics that
volume that velocity
the variety of data big data types so
that unstructured data
structure data semi-structured data we
talked about
distributed computing and how
distributed computing enables us to
process batch data and streaming data
then we looked at
artificial intelligence we looked at
machine learning we looked at deep
learning
we talked about data science and we made
sure to differentiate between those
those terms used interchangeably we
looked at roles on
data science teams and talked about how
data engineers and data analysts data
scientists for example work together
on this data science workflow and what
tools they need throughout the various
stages of that workflow
we then moved into struggles that
organizations face with big data in ai
and uh then we moved into data bricks
and talked about how the databricks like
house platform
addresses all of those challenges and
also what it looks like in an
organization
so what a workflow looks like using data
bricks in an organization
if you have any questions please let our
tas know
if not stay safe we hope you enjoy the
rest
of data in ai summit and again
please let us know if you have any
additional questions that we can help
you with
here's a few next steps um if you
want a personalized consultation for how
databricks can help your organization
specifically please reach out to value
at databricks.com if you're curious
about
other databricks customers and you want
to look at some more success stories
please
view our customer success story gallery
at databricks.com
customers
you can also go to
academy.databricks.com
if you want to learn more about
databricks itself
maybe if you want a more technical
introduction to databricks
maybe you want to try some of the the
hands-on features or
perhaps you know people on your data
science team that might want to
try out some of those features again
that's academy.databricks.com
thank you so much for joining and please
make sure to fill out your evaluation
when you're asked to do so
have a great rest of summit