Hi and welcome to 'This is My Architecture'.
I'm Andy, I'm here with Rishabh from DataCoral.
Hi, Rishabh.
- Hi.
It's great to be here, Andy.
- Excellent.
If you can tell me what is Datacoral.
- Of course.
So, Datacoral is a data engineering platform that allows our users
to extract data from 80 plus different data sources
and load that into the destination data warehouse of their choice,
like Redshift or Snowflake.
Now, the data sources can be databases, such as Postgres or MySQL,
they can be SaaS applications like Salesforce, Marketo,
or they can be generic file and object stores like S3.
- Excellent.
So, I see a largely serverless architecture in here,
if you wouldn't mind, can you tell us how you actually get the data
from your customers database into a different data store.
- Of course.
So, let's imagine a use case,
let's say that we want to move data from a MySQL database,
let's say it has a few 100 tables into a Redshift cluster right here.
And we want to be fetching data every five minutes
because data is changing rapidly.
So this is our architecture,
and one of the things that you'll notice here is that
it's fully deployed in our customer's VPC.
And this means that no data ever leaves their system.
In terms of the installation itself,
central to it is this orchestration service.
Now, this is two parts.
One, we use a Lambda which generates in response to all kinds of events.
And we have a DynamoDB, which basically acts as a config and state store.
Now coming back to this data source that we want to replicate data from.
So, because we wanted to fetch data every five minutes,
this orchestration service at the fifth minute will send an event
to our data connector, which, again, as you see will be a Lambda,
and it's using Lambda for processing, and it's fully serverless.
Now, this connector will do a few interesting things here.
So first, it will take this incoming request, and it will batch it up
into multiple pieces that can run independently and in parallel.
So you might imagine every single table being a separate batch,
and sometimes for large tables there can be multiple batches for the same table.
So these are going to be running independently in parallel.
And as these are in progress,
state information about how much progress has been made
on interesting metrics, such as how long a particular sync took,
how many records were fetched
whether schema changed, all of this gets saved into DynamoDB.
- So you mentioned batching your data before, I'm just curious,
what are some of the other reasons why you ended up doing that?
- So, there's two reasons.
One, because they could these can run in parallel,
we get very large speed ups in how quickly data gets fetched.
And the second is, if something were to go wrong,
it's only that particular batch that would need to be rerun,
because the rest of the data flows are all going fine.
- Excellent.
So after you have this initial Lambda, what happens next here?
- So the data that is being fetched as raw data
gets staged in S3.
And once the sync for a particular table is successful,
our connector will alert the orchestration service saying,
“Hey, the data has been written to S3, ready to go for the next step.”
And for the next step,
the orchestration service will then send an event to the Datacoral
Redshift Service, which you'll notice is yet another Lambda.
Then this service will pick this data up and issue a copy statement into
Redshift such that the data can be successfully loaded from S3 into Redshift.
- Can you give me a sense of how much data you can actually move with this?
- Yeah.
So, across all of our customers,
we are replicating over 500 billion records per month.
And as you can see, it's all done in a serverless fashion.
- Excellent.
Before, one of the things that you mentioned is
that you're able to take care of schema changes.
Can you walk me through how that actually happens?
- Yeah.
So, our data connector is essentially storing information
about whether schema for tables at source had changed.
And so, what will happen before this load is
we'll make sure that schema hasn't changed.
But if it has changed,
this Lambda will be sending certain DDL statements,
which will go ahead and modify the table at the destination.
and then the load from S3 into Redshift will be successful.
- Excellent Rishabh.
Thank you very much for walking through this.
It's great to see how Datacoral is using an entirely
serverless architecture to not only provide data pipelines,
but also metrics and information on the status of it as well.
- It's great to be here, Andy.
Thank you for having me.
- Excellent.
Thank you.
And thank you for watching 'This is My Architecture'.