Today we're going to be talking about big data. How big is big?
so
Well, first of all, there is no precise definition as a rule. So kind of be standard what people would say is
When we can no longer reasonably deal with the data using traditional methods
So that we kind of think what's a traditional method? Well, it might be can we process the data on a single computer?
Can we store the data on a single computer? And if we can't then we're probably dealing
With big data, so you need to have new methods in order to be able to handle and process this data
As computers getting faster and bigger capacities and more memory and things that the concept of what becomes big is is changing, right?
So kind of but a lot of it isn't really as I'll talk about later isn't how
Much power you can get in a single computer
It's more how we can use multiple computers to split the data up process everything and then throw it back like in the MapReduce framework
Then we talked about the for in with big data
There's something called the five es which kind of defines some features and problems that are common amongst any Big Data things
We have the five es and the first three that were defined. I think these were defined in 2001
So that's kind of how having talked about four. So first of all, we've got some volume. So this is the most obvious one
It's just simply how large the dataset it's the second one is
velocity
So a lot of the time these days huge amounts of data are being generated in a very short amount of time
So you think of how much data Facebook is generating people liking stuff people uploading content that's happening constantly
All throughout the day the amount of data they generate every day
It's just huge basically so they need to process that in real time
And the third one is variety
Traditionally the data we would have and we would store it in a traditional single database. It would be in a very structured format
So you've got columns and rows everywhere. He would have values for the columns these days
We've got data coming in in a lot of different formats
So as well as the traditional kind of structured data, we have unstructured data
So you've got stuff coming like web dream cliques, we've got like social media likes coming in
We've got stuff like images and audio and video
So we need to be able to handle all these different types of data and extract what we need from them
and the first one is
value
Yeah, so there's no point in us collecting huge amounts of data and then doing nothing with it
So we want to know what we want to obtain from the data and then think of ways to go about that
So something some form of value could just be getting humans to understand what is happening
In that data. So for example if you have a fleet of lorries
They will all have telematics sensors in that we collecting sensor data of what the lawyers are doing
So it's of a lot of value to the fleet manager to then be able to easily
Visualize huge amounts of data coming in and see what it's happening. So as well as processing and storing this stuff
We also want to be able to visualize it and show it humans in an easily understandable format
Oh, the value stuff is just finding patterns machine learning algorithms from all of this data
see then the fifth and final one is
Veracity this is basically how trustworthy the data is how reliable it is
So we've got data coming in from a lot of different sources
So is it being generated with statistical bias?
Are there missing values if we use think for example the sensor data, we need to realize that maybe the sensors are faulty
They're giving slightly off readings
So it's important to understand how?
Reliable the data we're looking at is and so these are kind of the five
Standard features of Big Data some people try and add more. There's another seven V's a big data at the 10 meter producer
I see. I'm sure we will keep going up and up
They are doing things like don't like vulnerability. So
Obviously when we're storing a lot of data a lot of that is quite personal data
So making sure that's secure but these are the kind of the five main ones
The first thing the big big data obviously is just the sheer volume
So one way of dealing with this is to split the data across multiple computers
So you could think okay. So we've got too much data to fit on one machine. We'll just get a more powerful computer
We'll get more CPU power. We'll get larger memory
that very quickly becomes quite difficult to manage because every time you need to
Scale it up again because you've got even more data you to buy computer or new hardware
So what tends to happen instead and all like they see all companies or just have like a cluster of computers?
So rather than a single machine
They'll have say a massive mean warehouse
basically
If you wind loads and loads and loads of computers and what this means that we can do is we can do distributed storage
so each of those machines will store a portion of the data and then we can also
Do the computation split across those machines rather than having one computer going through?
I know a billion database records you can have each computer going through a thousand of those database records
Let me take a really naive way of saying right. Ok, let's do it. Alphabetically, I'll load more records. Come in for say Zed
That's easy. Stick it on the end load more records coming for P. This Y in the middle, right? How do you manage that?
and so there's
Computing frameworks that will help with this
So for example, if you're storing data industry to fashion than this the Hadoop distributed file system
And that will manage kind of the cluster resources where the files are stored and those frameworks will also provide fault tolerance and reliability
So if one of the nose goes down, then it you've not lost that data. There will have been some replication across other nodes
So that yeah losing a single node isn't going to cause you a lot of problems
And what using a cluster also allows you to do is whenever you want to scale it up
All you do is just add more computers into the network and you're done and you can get by on
relatively cheap
Hardware rather than having to keep buying a new supercomputer in a big data
System there tends to be a pretty standard workflow
so the first thing you would want to do is have a measure to
Ingest the data remember, we've got a huge variety of data coming in. It's all coming in from different sources
So we need a way to kind of aggregators and move it on to further down the pipeline
So there's some frameworks for this. There's an Apache Capra and like Apache flume for example and loads and loads of others as well
So basically aggregate all the data push it on to the rest of the system
so then the second thing that you probably want to do is
Store that data so like we just spoke about the distributed file system
you store is in a distributed manner across the cluster then you want to
Process this data and you may skip out storage entirely
So in some cases you may not want to store your data
You just want to process it use it to update
Some machine learning model somewhere and then discard it and we don't care about long-term storage
So you're processing the data again do it in disputed fashion using frameworks such as MapReduce or Apache spark
Designing the algorithms to do that processing requires a little bit more thought than maybe doing a traditional algorithm with the frameworks
We'll hide some of it but you need to be thinking that even if we're doing it through a framework
We've still got data on different computers if we need to share messages between these computers during the computation
It becomes quite expensive if we keep moving a lot of data across the network
So it's designing algorithms that limit data movement around and it's the principle of data locality
So you want to keep the computation close to the data?
Don't move the data around
Sometimes it's unavoidable, but we limit it. So the other thing about processing is that there's different ways of doing it
There's batch processing
So you already have all of your data or whatever you protected so far
You take all of that data across the cluster you process all of that get your results and you're done
The other thing we can do is real-time processing. So again because the velocity of the data is coming in
We don't want to constantly have to take all the day to Detective
Well produce it get results and then we've got a ton more data
I want to do the same get all the data bring it back process all of it
So instead we would
Do real-time processing so as each data item arrives?
We process that we don't have to look at all the data we've got so far. We just incrementally process everything
And that's coming up in another video when we talk about data streaming
So the other thing that you might want to do before processing is something called pre-processing remember I talked about unstructured data
So maybe getting that data into a format that we specifically can use for the purpose we want to so
That would be a stage in the pipeline before processing the other thing with huge amounts of data
There's likely to be a lot of noise a lot of outliers so we can remove those
We can also remove one instances, so if you think we're getting a ton of instances in and we want them she learning algorithm
There'll be a lot of instances that are very very similar see an instance is say in a database
It's like a single line in the database. So for HTV sensor reading it would be everything for that
Lorry at that point in time CS speed directions traveling reducing. The number of instances is about reducing the granularity
so part of it is saying
if we store a rather than storing data for a
Continuous period of time so every minute for an hour if those states are very similar across that we can just say okay for this
period this is what happens and put it in a single line or we could say for example a machine learning algorithm if there's
Instances with very very similar features and then a very very similar class
We can take a single one of those instances and that will suitably represent
All of those instances so we can very very quickly reduce a huge data set down to a much smaller one
By saying there's a lot of redundancy here and we don't need a hundred very similar instances
When we one would do just as well
So if you've got a hundred
Instances and you reduce it down to one is does not have an impact on how important those instances are in the scheme of things
Yes, so techniques
That deal with this stuff. Some of them would just purely say okay now this is a single instance and
That's all you ever know others of them would
Have yet have a waiting?
So some way of saying this is a more important one because it's very similar to 100 others that we got rid of this one's
really not as important because there are least three others that were similar to it so we can wait instances to kind of reflect their
Importance. There are specific frameworks with big data streaming as well
so there's technologies such as the spark streaming module' for apache' spark or there's newer ones such as
Apache plink that can be used to do that. So they kind of abstracts away from the
streaming aspects of it so you can focus
Just in what you want to do a little thinking all this data is coming through very fast, obviously
My limited brain is thinking streaming relates to video. But you're talking about just data that is happening in real time. Is that right?
yes, so
Going back to the Lori's as they're driving down the motorway. They may be sending out a sense of read every
minute or so and
That since the reading goes back we get all the sense readings from all the lorries coming in as a data stream
so it's kind of a very quick roundup of the basics of Big Data and there's a lot of applications this obviously so
Thanks, we'll have huge volumes of transaction data that you can extract patterns of value from that and see what is normal they can do
Kind of fraud detection on that again. The previous example of fleet managers understanding what is going on
basically any industry will now have ways of being able to extract value from the data that they have so in the next video we're
Going to talk about data stream processing and more about how we actually deal with the problems that we all time data can presenters
over very very large BIOS
This kind of computation is a lot more efficient if you can distribute at because doing this map phase of saying, okay
This is one occurrence. The letter A that's independent of anything else and see most
Interested in you're probably only interested when a button is pressed or so on the only times positive