Welcome to 'This is My Architecture.'
I'm Orit, joining me today, Tomer from Fundbox.
Hi, Tomer, tell us about Fundbox.
So Fundbox is a startup company.
We are using AI and machine learning to help our customers
manage their credit lines.
What are we going to talk about today?
Today, we are going to describe our backend platform
we built in the data organization.
This platform orchestrates data jobs.
What was the challenge that you wanted to solve?
Prior to this platform,
we had a different set of EC2 instances in AWS.
And as a user, I had to SSH into one of those EC2 instances,
upload my files whether if it's SQL file or Python script or anything else
to manage everything by myself
and hope that my job will execute on time and nothing fails.
And that was challenging.
And this is why we decided to shift a bit
and to separate the business logic from the underneath infrastructure
and to help users progress in a more faster fashion.
Let's dive into the flow.
As a BI developer or machine learning engineer
or data engineer, I have a file that describes my job.
It could be a Python script, a SQL script, a Spark job, or anything.
And along with this file that describes my code,
I need to create another file.
This is a configuration file in which I need to define
the type of the job I have-
if it's a SQL, so SQL.
If it's a Python, a Python and so on.
Those two files are committed into a dedicated GitHub repository
and GitHub then triggers a Jenkins job
that will take those files, wrap them in a Docker container,
upload the Docker container directly to ECR,
create the ECS Fargate task for me
and create a Step Function for the job.
The Step Function needs to manage all the infrastructure
and decide when to execute the job.
From then on, I have few options to run my job.
I can just type a Slack command
that sends an HTTP request to API Gateway.
And API Gateway will trigger a manager Lambda.
I can wait for a cron expression to kick-in and then do the same,
just trigger the manager Lambda.
The manager Lambda will pick all the job configurations
from a database and will trigger the Step Function for me.
The Step Function will do all the Docker related things,
will grab the ECR repository,
and run the ECS task on Fargate.
And we'll eventually send an SNS notification
describing the job status, whether it failed or succeeded.
The SNS message is then sent out to a set of Lambda functions.
The first Lambda function is an audit function
that takes the job result and writes it to an S3 bucket.
Another Lambda function is a Slack Lambda function
that eventually sends notification to the user's channel in Slack
and tells the user if his job failed or not.
And another job, another microservice is a dependency Lambda.
This dependency Lambda decides
if we need to execute more jobs downstream.
I mean, if my job, it needs to trigger another job that I configured,
so this dependency Lambda will decide and then send back the event
to the manager Lambda and then all the flow runs again.
Where does the data sit?
So data can reside on a few data sources.
We have our warehouse or S3 bucket or external APIs.
So the code needs to interact with the warehouse
or S3 through a set of APIs that we provide to the end users.
How do you manage the permissions of the users to the data?
Okay, so inside GitHub we have a structure of directories.
Every job role in the company
needs to put these files inside a specific directory.
Jenkins then decides according to this directory,
which IAM roles and which permissions to grant to this user.
The IAM roles are shared with the Fargate task
and the Step Function and this is what limits the user
to access our warehouse or a data lake.
So you actually separate the data access from the users?
- Yes, users...
- They write the jobs, they don't need to care about infrastructure,
they don't need to care about permissions,
the job orchestrator actually runs their job.
- Exactly.
All the interaction they have is with GitHub or via Slack.
They also get logs through Slack.
They can browse the history of the job through Slack.
So as a user, all they have to do is to have Slack open,
and GitHub, and commit and push commands on my hands.
Sounds fun.
How does the platform manage different volumes of jobs?
So because we are leveraging serverless technologies,
we don't care what happens down underneath our jobs.
I mean, if we have at a point in time, one job or hundreds of jobs,
it doesn't matter.
The platform runs smoothly.
We depend on ECS Fargate and Lambdas,
these can scale very easily.
So basically we reside on serverless technologies.
Who are the users of the platform?
As I mentioned, users might be machine learning engineers.
They can write jobs like data extraction.
If I'm a data engineer that uses the platform,
I can write my ETL jobs.
If I'm a BI developer analyst, I'm using SQL to describe my jobs.
These are the set of users we are serving today.
What type of jobs does the platform support?
The platform supports, actually, any type of job
because we are using Docker containers.
We can compress into the Docker container
any kind of language or runtime we want.
Today, we actively support SQL and Python,
but users can also provide a Docker container,
Docker file themselves,
and describe any runtime they want, such as Node.js code or a JavaScript
or anything that they want to execute.
And the system will treat this job as if it was a normal job,
wrap it in the container,
and let the platform execute this job for me.
I assume they are happy users.
Do you have any future plans for the platform?
Definitely.
First, we want to create some console, I mean, a UI interface
that will help users to understand what is the status of their job
if there are other jobs that need to execute as a consequence.
Another feature we want to add is automatically resource tuning.
I mean, if my job failed for resources,
I mean, I didn't allocate enough memory to the job,
so we want the platform to manage that automatically,
to tune the Fargate task automatically for me.
Thanks, Tomer, for sharing that.
And thank you for watching 'This is My Architecture.'