Welcome to “This is My Architecture”.
Today’s guest is Mr. Hiroaki Idobata,
CTO of From Scratch Corporation.
Pleasure to meet you, Mr. Idobata.
Nice to meet you, too.
Now, please tell us about From Scratch Corporation and
today’s topic b→dash (b-dash).
Sure. From Scratch offers a service called b→dash,
which is a SaaS style marketing platform.
b→dash manages all the marketing data such as
web data, customer data and
purchase data in one place.
Then, it provides an all-in-one service with
marketing automation or bI recommend.
Today, I would like to explain resource manager.
It handles b→dash's core competence big data.
In b→dash, this resource manager actively controls
a large volume of EMR clusters to
handle a several TB data on a daily basis.
Please tell us the overview of the resource manager.
Sure.
First, when input data is placed in the S3 packet,
a message appears in SQS.
When an instance in EC2, a worker, sees the message,
it responds to the message and submits
data to be processed in EMR cluster.
The data processed in EMR cluster is
placed in a different packet in S3 for output,
then subsequent applications
utilize the data.
Now, I would like to review it from the beginning.
After S3 event, it's sorted out into a few SQS queues.
How does it sort out?
Yes.
First, there is a separate queue for each b→dash application.
Among those applications,
several queues are used depending on priority.
For example, batch processing is not
required to be real-time.
So, the data is saved and processed in
the low priority queue.
On the contrary, when a user’s screen operation is triggered,
a prompt response is expected.
The data is stored in high priority queue and
processed with priority.
This way, the user’s stress is eased.
I see.
They are sorted out by the type of processing or
high-low priority.
Then, how does the worker process a queue?
Before submitting the data to EMR cluster,
the worker calls API, and the resource manager will
inquire the cluster’s end point.
Resource manager stores cluster mapping data
for each type of task, it responds with
an appropriate cluster’s endpoint for the task.
Worker processes and submits data toward the endpoint.
Then, the result is stored in S3.
When several clusters are available,
resource manager observes the availability, and
returns the endpoint of the available cluster.
The data is effectively sorted out for
efficient processing.
I see. Resource manager controls the assignment of tasks.
Please tell us how EMR cluster is
created for better processing tasks.
It takes an enormous cost to process
a large volume of EC2 instance on EMR.
Therefore, b→dash tries to use spot instance by
using instance fleets in EMR.
Resource manager also controls the status of EMR.
When a cluster falls for not being able to acquire
a spot instance, resource manager detects it and
automatically recover the cluster.
This is how it provides stable processing.
With this, 70% of the whole can be
processed with spot instance,
we achieved 50% cost reduction,
compared to processing all on demand.
I see.
The use of spot instance from instance fleets provides
a great merit in cost reduction.
Now, please tell us the value this resource manager
brought to b→dash as a whole.
Certainly.
In order to process big data, it is imperative to
sort out data for processing.
However, it always accompanies
issues of stability and cost.
b→dash was able to overcome the issues
by implementing resource manager, and achieved
high stability and low cost.
High stability and low cost. Both are very important.
At last, please tell us about b→dash’s outlook.
Certainly.
Not only improving the sort out system in batch processing,
we also enhance real time data processing, so we can
provide a platform where users are able to more freely,
interactively handle data and accelerate their business.
Let’s make it come true together.
I understand that resource manager is the key in
b→dash and is optimizing the resource management in
this architecture.
Thank you, Mr. Idobata.
Thank you.
“This is My Architecture” by From Scratch.