Hey guys welcome to yet another interesting session by Intellipaat.
Today, we are going to discuss about Apache Spark.
In today's session we will cover
why Apache Spark?
What is Apache Spark?
Career opportunities and hands-on exercise
Apache Spark Vs. Hadoop, Different features of Apache
Spark.
Spark Architecture and its components, Career opportunities and hands-on
exercise
Now, let us look at the use case of Spark
One of the world's leading and pioneering
e-commerce giant is eBay which facilitates both business to business and
business to consumer sales.
eBay is a marketplace with a hundred and sixty-two million active customers
and 25 million sellers.
eBay wanted to make their data as efficient as possible for their buyers and sellers.
As they have a total of eight hundred million listings on their site and 8.8 new million
listings are being added every week.
Initial user pattern of eBay was search oriented,
in which the users had to search through the search listings to find the products.
With the transition in the e-commerce, the users just wanted to browse and
buy the products instead of searching in the list.
To do this, eBay had to transition the entire data model.
Initially, eBay was using Hadoop architecture which was not compatible with the real-time
data model, hence eBay was looking for real-time architecture tool.
Well, Apache Spark was a handy tool for them.
eBay uses Apache Spark to provide offers to target customers based on their earlier
experiences and tries to leave no stone unturned in enhancing the customer experience
with them.
This enhances the customer experience in providing what they might require in a proactive manner,
also helps them to efficiently and smoothly handle customer's time on the
e-commerce site.
eBay does this magic by letting Apache Spark leverage through Hadoop YARN.
What is Apache Spark?
Do you have any idea what it is?
Apache Spark is an open-source engine developed specifically for handling
large-scale data processing and analytics.The large amounts of unstructured data and the
need for increased speed to fulfill the real-time analytics have made this technology a real
alternative for Big Data computational exercises.
When it comes to Real Time Data Analytics, Spark stands as the go-to tool compared to
all other solutions.
You might be thinking when we already have
Hadoop to process the data, why do we need Spark?
Let us see what makes Spark more popular than Hadoop?
To answer this, we have to look at the concept
of batch and real-time processing.
Batch processing is nothing but collecting the data over a period and
processing it.
For example, payroll and billing system.
In case of real-time processing, the data collection and processing occurs
at the same time.
You can think of ATM as an example of a real-time system.
As soon as you enter your password and other details you will receive the desired amount.
Hadoop is based on the concept
of batch processing where the processing happens on blocks of data that is already being stored
over a period.
However, Spark can process data in real-time and it is about 100 times faster
than Hadoop MapReduce for batch processing large data sets.
One more advantage of Spark is, it can deal with both historical and real-time data.
The image on screen will clearly depict the
major difference between Hadoop and Spark With Spark, you can handle multiple applications
based on both historical and real-time data.
Spark is easy to use and a faster processing engine when compared to Hadoop.
Let us look some of the features of Spark.
Spark code can be developed in Java, Scala, Python or R language.
Spark runs 100 times faster than MapReduce.
Spark supports different data sources like text, CSV, RDBMS, JSON, Hive,
Cassandra etc.
Spark provides real-time computation.
Spark also supports machine learning.
(Machine learning)
These are the plus points which made Spark a booming technology in the IT sector.
Now, we will move on to the Spark architecture.
Spark architecture includes the Driver program,
cluster manager, and various worker nodes.
The driver is the central and
entry point of the Spark shell.
It runs the main () function of the application and creates a Spark context.
The driver stores the metadata about all the Resilient Distributed Databases
and their partitions.
Cluster Manager is responsible
for acquiring resources on the Spark cluster and allocating them to a Spark job.
There are 3 different types of cluster managers a Spark application can leverage for the allocation
and deallocation of various physical resources.
We will discuss this in the next slide.
The executor is a distributed agent responsible for the execution
of tasks.
Every Spark application has its own executor process.
Executors usually run for the entire lifetime of a Spark application and
this phenomenon is known as "Static Allocation of Executors."
Let us understand how Spark runs.
Apache Spark runs in three different ways Standalone, On Hadoop
YARN and Spark in MapReduce The simplest deployment of spark
is standalone in which a Hadoop cluster is equipped with all resources statically and
runs in parallel with MapReduce.
Spark can also be executed on top of YARN without any pre-installation.
Even though, if you don't have YARN, you can also use Spark along with MapReduce.
This reduces the burden of deployments.
Anyway, Spark is deployed, the configuration allocates it the necessary resources.
The moment Spark is connected, it obtains executors on the node.
These executors are nothing but the processes running computations and securing the data.
Now the application code is sent to the executor.
Let us now look at the Spark components that makes spark ecosystem.
Apache Spark Core is the fundamental processing engine of the Spark application.
Its most important features include the in-memory processing and the data referencing from external
data sources.
Spark Streaming displays its high-speed computational prowess.
It works excessively well with streaming data for providing real-time data analytics.
The data is segregated into multiple batches and using the RDD abstraction the data is
processed in a massively parallel fashion iteratively to suit the needs of stream data
processing.
Spark SQL creates a new level of data abstraction called the Schema RDD
for working exclusively with both structured and semi-structured data by deploying SQL
querying language.
GraphX is a Graph processing capability of Apache Spark framework that
is the amalgamation of iterative graphical computation, exploratory analysis, and ETL
capabilities.
Spark can also be used for Machine Learning algorithms and applications using
its MLlib library that provides a Machine Learning framework for Spark in a memory-based
distributed environment.
These features of Spark have
been rapidly adopted by enterprises cutting across a wide range of industries.
Internet powerhouses such as Netflix, Facebook, and eBay
have deployed Spark at massive scales.
It has quickly become the largest open source community in big data, with over 1000 contributors
from 250+ organizations.
PayPal also uses Spark technologies to manage
and process its large-scale data.
Being a closed-loop payment platform, it processes billions of data events every day
around users, risk, payments, web behavior, and identity.
To help their merchants in maximizing the number of successful transactions, they had
to use data processing solutions.
Well, Spark plays a major role in that and helps PayPal to process large-scale data to
enable impactful merchant solutions.
Now let us go on to the hands on exercise