MARK MIRCHANDANI: Drug research and development
is more challenging than ever, requiring massive amounts
of data and complex analysis.
Join us as we talk to Recursion Pharmaceuticals about how
they're using machine learning to speed up the drug discovery
process on this episode of Stack Chat.
Scott, thanks so much for joining us today.
Can you tell us a little bit more
about what Recursion Pharmaceuticals does?
SCOTT NIELSEN: Yes.
Recursion is-- we're trying to bring new drugs to market.
And in order to do that, we have to find new drugs,
discover new drugs.
And we're trying to turn that problem into a machine learning
problem.
What that means and how we do that is,
we are creating a large data set consisting primarily
of cell microscopy images.
And so that means we take a lot of pictures of a lot of cells
in a lot of different states--
so healthy cells, sick cells, sick cells
combined with drugs, healthy cells combined with drugs.
And try to learn more information
about how those images change under different circumstances.
MARK MIRCHANDANI: So what does the architecture
look like that does the eventual machine learning on these?
SCOTT NIELSEN: Currently, we're taking about 60 terabytes
of images a week.
And so at the beginning, that can
take across seven different microscopes
about five days at times to take that many images.
And so we're constantly-- as soon as those images are
available, we're uploading them into GCS,
and sending off events that those images are
available for processing.
And then we have a couple of different Kubernetes clusters
running on GKE.
All our software is deployed in there,
and we have a number of applications
that are reacting to these events,
aggregating them at different levels,
and then scheduling processing in our kind of jobs
Kubernetes cluster to do different computation
on the images or the results of a previous processing,
until we boil those down into more refined features
through this feature extraction process.
And then those features are used, and analysis is done.
Runs through-- depending on the type of experiment,
it runs through different analyses
where all these results, all these metrics,
quality metrics on the images, and a lot of different things
are stored in a combination of Cloud SQL and BigQuery.
And then all that data is used to make decisions and design
our next round of experiments.
And so that's kind of the process
that we go through at Recursion on a weekly basis.
MARK MIRCHANDANI: So it sounds like you did a lot of data
processing directly in Kubernetes with Kubernetes
jobs, rather than using a traditional big data
solution like Hadoop or Spark.
What made you kind of choose to write your own Kubernetes?
SCOTT NIELSEN: I think there is a number of factors.
So the two primary factors that pushed us in that direction
was that we have all these images coming off
the microscopes constantly.
And so we have this kind of streaming flow of images.
And in order to wait for the right batches that
are the most important, it can take sometimes hours.
But we want certain feedback on a lot of those images
much more quickly.
And so we wanted a streaming solution
so we could get faster feedback on a lot of these quality
metrics and stop problems and save ourselves
money from taking pictures that aren't valuable.
The second part of that was we had a number
of existing services that weren't written
in languages or in ways that integrate nicely
into some of those tools.
And so some of those were Cloud Functions or Lambda jobs
and other things that we wanted to be able to not
have to rewrite all of that in order to just orchestrate
this pipeline.
MARK MIRCHANDANI: Awesome.
Well, thanks so much for coming in and telling us
a little bit about how you built this entire solution,
taking a real big problem, big data problem,
and turning it into real insights.
SCOTT NIELSEN: Yeah, thank you.
MARK MIRCHANDANI: Check out this quick guide
to try running Kubernetes jobs of your own.
Thanks for watching, and make sure to subscribe
for more great Google Cloud Platform content.
We'll see you next time on Stack Chat.