Welcome to 'This is My Architecture'.
I am Hawn, and today I am joined by Sathish from BioPharma.
- Welcome. -Thank you, Hawn.
Before we get into your architecture, can you tell me a little about BioPharma?
Yeah. Our company is in the biopharma industry.
We develop targeted, personalized treatments in the area of neuro disorders.
By neuro disorders I mean diseases like Alzheimer's, Schizophrenia, and Parkinson's.
So I see that you have a data lake.
So once the data lake has been populated, how do you orchestrate and curate it?
That's a great question.
Once the data lake has all the data, we need to get the data to the processing layer.
The first thing what we do is for MRA data processing
which we are discussing right now for this architecture,
we have the AWS Batch service which access Orchestrator.
So, when we run a job with AWS Batch, it creates many EC2 instances.
In this case we are talking hundreds to a few thousand of EC2 instances.
So the computing area is instantiated.
The computing area requests are stored, for which we are using FSx.
And FSx is the one which obtains the data from the data lake in this case.
So this is enabled at the time of provisioning the FSx storage.
At the time of provisioning we have the capability to sync with a particular input S3 for that.
AWS Batch orchestrates this download and it reads the data.
Once it has entered FSx storage, it is mounted on an EC2 instance.
This is where the processing happens, and the curation also happens during the time of processing.
So as a batch process, it must be, a long process, that you may have quite a bit of errors.
How do you manage that?
At the time of processing, definitely this is a multi-day job for most of it.
Depending on the data sets it can take anywhere from 24 days to 7 days.
At the time of processing, if errors happen, we identify it this way.
We actually have AWS Batch sending events into EventBridge.
Even when notifying the change in status of AWS Batch.
These are the events that are communicated, these events are being sent by Lambda
onto ElasticSearch where they get indexed.
That's how we get to know anything that's failing, we immediately get to know the events through ElasticSearch.
So, from the data science perspective, how do they visualize and consume the data sets?
Our data scientists and our biologists, are the primary consumers of this pipeline.
So, how do they come to know?
We have all the logs being sent from AWS Batch in the ElasticSearch.
That's when they can actually look into the logs to see, if there are more technical, they would like to know,
what subjects have been processed.
That information is communicated, again, from AWS Batch to Lambda,
and Lambda indexes. That's one.
The second thing is output that's produced during the processing.
The output that's produced is actually stored in FSx,
which is again, synced into the data lake,
using an activity called a DRT or a data repository task, which is provided by AWS.
This automatically syncs terabytes of data that's produced on FSx back into the data lake.
And this data is something that contains
all of the required correlation matrices required for visualizing this data.
So, our biologists can look into the data lake to see the data.
Awesome. So it sounds like you're able to not only
ingest, curate, and orchestrate the data; you're able to handle the errors and process it
where that the data scientist can visualize the data
and also see the status from the long-running batch so that they can see the petabytes of data that you have
for the MRI scans to help with that accelerated processing.
Exactly.
It all helps.
All the things that we do is to help our biologists and data scientists take advantage of the pipeline.
So they are going to trigger the pipeline, and they can monitor the entire pipeline for instance,
and also look at the visualizations that are enabled by ElasticSearch,
where the Kibana dashboards are being exposed to visualize.
And we also have the data being visualized in our data warehouse that's powered by the data lake.
So thank you so much for going over your architecture.
That is amazing how you're able to help the data scientists
not only visualize the data, but also help with that rapid, accelerated research, and development
for personalized neuro disease and also drug discovery as well.
- Thank you so much. - Thank you so much, Hawn.
And thank you for watching 'This is My Architecture'.