Hi, welcome to 'This is My Architecture'.
I'm Sudipt.
This is Roberto from Vertex.
Roberto, how are you today?
Well, thanks.
Good, well tell us a little bit about what Vertex does.
Sure, so Vertex is a global biotechnology company headquartered in Boston,
that specializes in focusing on scientific innovation
to create transformative medicines for people with serious diseases.
Excellent, today, we're here to talk about the molecule search platform,
designed and deployed on AWS.
But you could tell us a little bit about the application,
and then we can talk through the user flow.
Sure.
So, for some of the disease areas where Vertex is conducting research,
we want to look for a potential small molecule
that could become a therapeutic.
There are 10s of billions of potential small molecules to evaluate.
So our computational chemistry team asked to work with us to create a platform
that would let them more intelligently search that space
to find molecules to synthesize in the lab.
Excellent, so these are probably not technical users, right?
Maybe you can talk through how they're interacting with the platform.
Sure, so our chemists have a client that runs on their laptop
that allows them to submit a request to an Amazon API Gateway API.
They in that payload, submit a post request with some JSON that includes,
the type of search they want to run,
as well as the size of the fleet of workers they want to create for that search.
That request comes to an AWS Lambda function
that enriches that search with some additional details from a relational database.
And then creates this environment on the fly
in response to some of those parameters.
Excellent, so actually dynamically creating this environment
based on the parameters that were placed by the chemists.
Interesting.
Okay, great.
And I'm noticing you've spinning up Fargate here,
which is also spinning of workers, the Elastic Container Service.
So maybe you can kind of talk through how Fargate is spun up
and what the worker nodes are actually querying.
Sure. so as I mentioned, the Lambda will spin up this environment
based on parameters of the search that includes,
the size of a Fargate Spot Fleet over here on ECS.
It spins up one task that essentially drives the search,
we call this the Orchestrator.
So Lambda will spin up the Orchestrator,
and the Orchestrator communicates with the search workers
via RabbitMQ on Amazon MQ.
So the Orchestrator will tell the search workers where to look next,
the workers pick up that work and execute the search.
And this is actually an intelligent search.
So the workers will report back, details of how their search is going.
That information comes back to the Orchestrator
who then further influences the search by telling them where to look next,
based on what is hearing back from there- Excellent.
So interesting use of MQ, not just queuing requests,
but actually helping you with that intelligent pattern detection.
That's right. Excellent.
So you mentioned billions of molecules,
maybe you can talk to the size and scope
and kind of scale of what these worker nodes will query.
Sure, so as I mentioned, there are libraries of 10s of billions of molecules,
that our chemists want to search with these searches.
And depending on the complexity of search,
and the details of the search parameters they pass to us,
we'll spin up anywhere from 50 to 500 Fargate tasks,
which ends up being about up to 1000 CPUs worth of worker power.
Excellent, so maybe you can talk through
like what kind of impact has that had on performance and cost in general,
compared to maybe how you were traditionally leveraging this.
So as you mentioned earlier, the whole architecture is serverless
and that's by design.
We want this environment to come alive when there's a search to be done,
and then to disappear when the search is over.
Also, using Fargate spot,
further, improves our economics for this architecture.
So not only are we getting better performance than we were before,
by being able to scale up very wide into AWS,
and to fine tune size of this cluster,
depending on the search parameters
between using spot and serverless,
we're both more performant and more cost-effective than before.
Excellent.
Now let's talk through when a pattern has been found.
So what is the interaction?
How does the Orchestration process ultimately,
let the chemist know that, "Hey, we potentially have a result?"
Right, the Orchestrator here is tallying the promising results
that it's hearing back from the workers.
And when the search is complete,
the Orchestrator writes those results to Amazon S3.
One of the parameters that was passed in the JSON payload from the scientists,
was an email address where they want to be notified when the work is done.
So the Orchestrator lastly, will send an email to the scientists
with a pre signed download URL for S3 where they can download the results.
Excellent.
This is just an excellent example of using a serverless architecture,
to meet your initial design goals of performance, cost optimization,
as well as democratization of access by your chemists.
So thank you, Roberto, for helping us walk through this today.
And thank you for joining, 'This is My Architecture'.