BALAJI SIVAGANESAN: Have you ever wondered of loading data
from Cloud Storage to Bigtable using the power of Spark
by just passing some arguments?
Stay tuned.
This is Balaji.
I'm a Cloud Migration Consultant with Google Cloud.
Dataproc templates are an effort to solve
simple but large inflow data tasks including
data import, export, backup, restore,
and bulk AP operations.
The technology under the hood, which
makes these operations possible is the serverless part
functionality based on Google Cloud's data hub.
Without having to create them from scratch,
we can run typical use cases on dataproc serverless
using Java and Python, thanks to dataproc templates.
With the help of these templates,
we can simply customize under common Spark workloads.
Now we'll focus on GCS to Bigtable Java template.
This template is written in Spark which is open source
and can be customizable based on the requirements from the user.
These are driven by configurations
and are ready to use.
This offers the user the power of Spark
and customizing for advanced use cases is super cool.
One of the common [INAUDIBLE] tasks
is to load data from GCS to a low latency
database like Bigtable which can power up any UI or real time
query platforms.
This template supports CSV, Parquet, and Avro file formats.
These templates are ready to use powered workloads,
and they can be altered with connection parameters for use
cases with identical requirements
and the development can be completed in no time.
Here's the flow of data in the GCS
to Bigtable dataproc template.
The source is the GCS storage, which
holds the file to be loaded into Bigtable in a storage bucket
inside a specific folder.
As of now, the template supports loading
of CSV packet and out of files.
The file has to be uploaded in a folder,
and the exact location has to be passed as an argument.
Unlike other storage things, Bigtable is quite different.
It is known for its low latency performance,
and the overall query performance
depends on the row key.
The data is stored as column or storage format,
and each row has its unique row key.
The beauty of this NoSQL lies in its columnar axis of data.
So each column can alone be accessed with its row key,
column family, and column name.
[INAUDIBLE] the goal of this template
lies the power of Spark provided by dataproc serverless.
It accesses the data in GCS and loads the data in a [INAUDIBLE]
at a much faster pace as it's driven
by the number of workloads which are handled internally
by dataproc serverless.
Once the batch job is started, the data serverless
looks into the job configuration,
and its starts the Spark session.
Right now there is no logic written for any transformation.
It just loads the data into big tables from GCS.
But in case of any enhancements required,
it can be carried out in no time.
Here is a sample CSV file and how
it gets loaded into Bigtable.
As the row key needs to be unique,
the first column in the CSV file is taken as the row key,
and it gets loaded.
Further, the header details are taken automatically
as the schema.
The column family is specifically
passed as an argument.
As we see, the various values gets
loaded as columns under the given column family,
and the first column of the given row is the row key.
Here is a sample Avro file and how
it gets loaded into Bigtable.
From the input, we can see the schema has a timestamp, tweet,
and the user name.
Since the username is the first level value in the input,
that's taken as the row key.
The various values gets loaded as columns
under the given column family.
The column names gets populated automatically from the schema.
Here is a sample packet file and how
it gets loaded into Bigtable.
From the input, we can see the schema has continent
and the country array.
The continent is the first value,
and it gets loaded as the row key.
And the array of countries gets loaded as a single value.
Further, there are two rows with the same row key.
So it gets loaded as two separate set
of columns under the same row key, as it has to be unique.
In the previous video, you would have
seen it's a three-step process to execute.
First, the clone the repository get authentication credentials,
and to execute the template.
Now let's focus on the third part,
and how to execute the template.
First, create the GCS bucket, then the input folder,
and upload the required file into that folder.
Doing this in a console is the most preferred one.
Next, create the Bigtable instance
if it's already not available in the cluster.
Then within the instance, create the Bigtable table
with the required column family.
This column family has to be passed in the arguments.
Set the configuration parameters like the GCP project,
region of execution, and the GCP staging location
for internal use to upload the jars and temporary storage.
Then pass the arguments and execute the template.
This is to Bigtable Java, post which the batch Spark
job has to be monitored.
Here are the list of arguments that
needs to be passed while executing the template.
The major ones are the location of the input file, the details
of the Bigtable instance, the table name, and the column
family.
The respective project tallies of the GCS storage
and the Bigtable also needs to be passed.
Here are the commands used to execute the template.
The ones in red shows the configuration parameters
that needs to be set one time during the session.
The start script will trigger the template,
and the name of the template has to be given along
with the required arguments.
Once this is submitted, it builds the jar, copies to GCS,
and then triggers a batch job in the dataproc serverless.
In this section, we will cover hands-on on the GCS
to Bigtable template.
First, we will start with the Cloud Shell,
and after authentication is done,
let's clone the dataproc template repository
into the Cloud Shell.
Then let's traverse into the folder,
check the branch details using the key status,
ensure that we are in the main folder.
Let's ensure that we have the required input
files in the input folder of the GCS packet.
For instance, let's take a CSV, Avro, and packet file.
Let's see how the input CSV file us-states2 looks like.
It is a header and some record.
So we can enter into the Bigtable instance
in a Bigtable, and let's create a table if it doesn't exist.
For now, use the table bus-data.
Please note that the column family
needs to be given correctly while creating the table.
Now let's see the contents of the bus data table.
You can see there is no data in the table.
You would proceed with the configuration parameters
to set up the configuration parameters like GCP project,
region, GCP staging location.
Once the basic setup is done, next
is our template specific processes.
Populate the required documents for running the template
like the location of the input file in GCS,
expand, then the details on the Bigtable table instance,
and the column family.
Copy the whole command for executing the template.
And the logs of the Spark job execution are shown here.
Once the job gets completed, it clearly
provides the details of the completed
job along with the additional details.
The data from UI shows the logs along
with the number of executables that
are running in the back end.
In case, if we need to view the detailed worker
level logs from Spark, it can be seen from the view log section.
Now let's check how the data were loaded into the Bigtable.
In our input we have two records that we
can see that two records got inserted into the Bigtable.
As shown in the example previously,
we can see the schema got updated into the column
names automatically.
With this, we are coming to an end for the demo of GCS
to Bigtable template.
We are happy to seek your feedback, queries,
and solutions based on your requirements and usage
of these templates.
Please feel free to reach us at Dataproc templates
support external@Googlegroups.com.
Cheers.
Have a great day.