Welcome to 'This is My Architecture'.
I am Hawn, and today I am joined by Chris from Biopharma.
Hello, Chris. Hello Hawn.
Before we get into your architecture, can you tell me a little bit about Biopharma?
Sure, Biopharma is about drug development
specifically in the neurodegenerative disease space.
And we're looking to develop drugs to help things like Alzheimer's.
Okay.
So with that, how do you ingest all that data?
It's a great question, Hawn.
So most of the data that we work with genomics data in particular
starts off as A, T, Cs, and Gs,
and basically hundreds of billions of these.
Typically, in the amount of samples
that we're ingesting, which is hundreds to thousands.
And so that's coming into our ad hoc EC2 VMs.
So we spin these up as data scientists and data engineers,
and then we write the data to FSx.
So how do you curate all this data?
Great question, Hawn.
So once the data is in FSx, because there's so much of it,
we're talking again, hundreds of thousands of samples,
we can't just manage a few ad hoc'd VMs.
We need to actually have a more scalable process, and that's where EKS comes in
and our container repository that we use our private repository.
And so what happens is that we take the set of software that we're using,
and we freeze it into version containers
and that's stored in ECR, and we have both versions for a particular pipeline
and multiple pipelines stored here.
Then those are pulled from ECR into EKS
and data that we had ingested earlier in the step
is then run on because these are all mounted,
FSx is mounted onto the different nodes, and we're talking now hundreds
or potentially hundreds of nodes that we are going to scale up to using auto-scaling,
to actually do the curation of the data, to go from A, T, Cs, and Gs
down to matrices
one through 10, 100.
And this is the kinds of things that our data scientists who do statistical analyses
can then reason over and then give gene targets of interest to our biologists
and these will end up in the Data Lake.
Okay.
So are you able to leverage the EKS to do that rapid research and discovery
of the data sets that you get?
Absolutely, and that EKS really makes this possible in a very easy way.
So it's easy for me as a data scientist and engineer,
once I have versioned these containers to be able to spin up these EKS,
thanks to auto-scaling.
So to go from zero to 100 instances, without really even needing to think about it.
And then to know that those will scale down when we're done,
because these are sporadic pipelines, we don't run them every night.
They don't need all those machines running and that saves us in cost.
And then the outputs of that being written to the data lake,
of course, we're saving them, we're persisting them
and these matrices for the downstream users and that specifically the data scientists
who are doing the statistical analysis, and eventually the biologists. Okay.
So you're able to take the version from ECR, train on a model
to do that rapid discovery and populate your data lakes
for your data scientists to consume from. Exactly.
Awesome, so what are some of the benefits that you've seen from this architecture?
Well, I've alluded to the scale up possibilities of EKS and auto-scaling are huge.
So we love that feature, and we're using it as much as we can.
That's the first thing.
The second thing is the versioning aspect.
Is that to be able to do reproducible science, because this part of this is research
and development, we need to be able to come back and say, this is the version we used
this is the set of software we used, and that's really what the versioning is,
locking down those software versions for all the tools that we're using in these pipelines.
And then the democratization, so that I'm not always the one that has to run this.
I'm the one that knows how it works from end to end,
but I'm gonna be working on other things and that we can get other people
to do this who don't need to necessarily understand the whole process.
Great, awesome.
So what's next for Biopharma?
Mainly just refining this workflow, making it more stable,
making it easier to go from this part, the ingestion and this sort of exploration
to a container and then to scaling it up.
Well, thank you so much for going over your architecture.
That is amazing how you're able to curate all the genomic sequencing data
so that you can basically do rapid discovery for your data scientists
to have that R&D for the genomics discovery.
So thank you for that. Thank you, Hawn.
Okay, and thank you for watching 'This is My Architecture'.