Welcome to Apache Spark in RStudio IDE.
In this video we'll look at how to run SparkR code in RStudio IDE.
To run Spark, you just need to enter "sc" (if you get an error, you may need to restart
RStudio IDE).
To restart, select File > Quit Session If the sc command is successful, the JavaSparkContext
id is displayed, as shown here.
Let's briefly explore the automotive data that's pre-installed in R. To load this
particular data set (or a data frame as it's called in R) enter "head(mtcars)".
And run this command within double quotes.
When you do so, the Console displays this.
Then, to work with dataframes we need to create the SeQueL Context, which is created using
spark RSQL.init(sc) within parenthesis.
SQLContext uses SparkContext, which has already been created, and named "sc".
Running this command, you'll see in the console log, that SparkRSQL.init(sc) within
parenthesis, has successfully been assigned to SQLContext.
Then we'll need to create the Spark DataFrames by running these two commands.
After running these commands, the console log shows how the variables have been assigned.
Next, run these spark head commands; the first command prints the contents using spark R,
the second selects a column, and the third filters based on a condition.
To operate on columns, SparkR also provides a number of functions that can be directly
applied to columns for data processing and aggregation.
This example shows the use of basic arithmetic functions to convert lbs to metric tons.
And the console log displays the output, as follows.
For Grouping and Aggregation, SparkR data frames support a number of commonly used functions
to aggregate data after grouping.
For example, we can compute the average weight of cars by their cylinders as shown with this
statement.
And in the console, we see the output that is, the average weight by cylinders.
And finally, here is how you can run SQL Queries from Spark DataFrames.
A Spark DataFrame can also be registered as a temporary table in SparkSQL.
Registering a DataFrame as a table allows you to run SQL queries over its data.
The "sql" function enables applications to run SQL queries programmatically and returns
the result as a DataFrame.
In the console output we see the gears where cylinder was greater than or equal to 4 or
less than or equal to 9.
We've come to the end of this video.
We encourage you to practice on your own in RStudio IDE.
We're sure that if you do, you'll be up and running in no time!
Thanks for watching!