[MUSIC PLAYING]
SPEAKER: OK, moving on-- last talk for the evening.
We have Randall here.
Randall Lin is an ML engineer at Fathom Health.
He previously studied physics and computational neuroscience
at Cal Tech.
Before Fathom Health, he worked at Halo Neuroscience.
RANDALL LIN: Yeah.
So I'm here to talk about Airbender
and how this tool enables us to isolate developer working
environments while we work on Kubernetes and Airflow.
So who are we?
At Fathom, we use machine learning and NLP
to standardize physician notes for medical billing.
I'm Randall.
And here's our tech stack.
We do everything on Google Cloud.
Everything is run on Kubernetes, scheduled through Air Flow,
and then everything ends in Python,
which is important to note just because everything
will be interpreted.
So that lets us do certain things we get away with.
So our main goal here is to make development easy and fast.
And in the spirit of that, we want dev and prod
to be equal almost always.
And also, as a machine learning shop, we experiment a lot.
And as a side effect of experimenting a lot,
we also run our pipelines a lot, rerun them a lot.
And we do this on multiple versions
of the pipelines or the tasks.
And to do that fast, our DAGs have to reflect the latest
commit on that branch for fast iteration.
And as we're dealing with medical data,
security is also of utmost importance.
So our solution is Airbender.
And it addresses the problems.
It isolates our Air Flow work environments.
On demand, you can spin them up, put them back down.
And they're tied to specific GitHub branches isolated
in Google Cloud Storage.
Their code and their secrets all synced.
And so here's the user experience for the developer.
Make sure you're on the branch you want to be doing things on.
Then all you do is run Airbender Up.
You name your name space that you want to be doing things in.
And after a few minutes, you'll have an IP you can go to.
And you'll have your DAGs.
And two things to note-- the code changes
will be immediately available to the task,
because they're spun up as they're run.
And any DAG structure changes are
dependent on Airflow just refreshing the DAGs.
And let's say you're experimenting things.
Maybe you have another branch you want to run.
The other run's going.
Do the same thing; just change branches and spin up again.
And when all's said and done, everything's run.
Results are good.
You want to bring your environment down,
just Airbender down, and it will delete all the Kubernetes
resources that you've been using.
So that all seems easy.
But behind the scenes, what's happening is,
as you run Airbender Up, it's a local Docker
that defines your Airflow pod spec,
chooses your GitHub branches, decides if you need OAuth,
if you need firewalls.
Then it passes it to GKE and creates your full Airflow pod
in the specific namespace you've decided on.
And it also syncs your code.
After that, the load balancer, you ask it for the IP,
and then you can add it to the DNS for a subdomain of Airflow
that you, as a developer, are working on.
And then our Secret Syncher deployment on GKE
will push the secret stall namespaces so that your service
accounts are always up to date.
And as a developer, you can just,
using that URL or IP address, you can hit your web server.
And how does the individual task, how do they get run?
Three things happen when you run each task.
Service account is chosen so that the permissions
are minimally permissive.
GitHub branches are mounted to the new containers
that are spun up.
And I/O is isolated by every DAG is a unique GCS workspace path.
That way, all the files going in, coming out
are under that workspace.
Certain gotchas that we ran into--
two main things.
One is being able to sync your code is a blessing and a curse.
So you might be changing your DAG structure
and, all of a sudden, you had something running,
and you change the DAG, and now, they're orphaned tasks.
But that's pretty easy to revert.
You can just sync again.
Dev being close to prod is also good.
But that makes devs want to iterate remotely on this
on Airflow.
So that's going to be still inherently
slower than local iteration.
So certain to-dos we have there, like Minikube or other things.
Questions?
AUDIENCE: Is the database shared with--
[? do you have ?] to have your own?
RANDALL LIN: Yeah, it's not like Cloud SQL or anything.
AUDIENCE: This is [INAUDIBLE]?
RANDALL LIN: So it's on one pod, but it's, like, each container.
Yeah?
AUDIENCE: [INAUDIBLE]
RANDALL LIN: Yeah.
AUDIENCE: This was totally sweet,
by the way, this presentation.
RANDALL LIN: Thank you.
AUDIENCE: I guess, how much constraints do you
put on your actual business logic
code within the tasks in order to-- you said you
had to do namespacing in GCS.
Do you have APIs in your actual business logic code
so that you have to actually use that namespace?
Because Airflow is just arbitrary Python, right?
So it can touch whatever external resources you want.
RANDALL LIN: Yeah.
It's a combination of the namespace, the DAG ID, runtime.
But that's all wrapped, like, go get stuff from GCS function
that we have.
And it will just look in your environment variable,
and it will know you're Airflow, and you're
doing certain things.
And it will inject those directories.
AUDIENCE: So you use a bunch of internal APIs,
and then, because you're good at stuff,
you just remember to use them?
RANDALL LIN: Yeah.
AUDIENCE: Everything's ephemeral, right?
So that's for one-offs.
If you want to schedule something every day
for the next year--
RANDALL LIN: It's all different, yeah.
AUDIENCE: Then you'd get a little bit more [INAUDIBLE]??
RANDALL LIN: Yeah, the typical use case is, we're
going to explore all these models.
One crazy use case is like [? Tensor ?] broke
some of our veins, and we wanted to get
bisect, but the whole pipeline.
And so just spin up a bunch of these and see what's happening.
Yeah?
AUDIENCE: This was awesome.
I just wanted to say that too.
AUDIENCE: I have a [? naive ?] question.
First of all, how long did it take you to [INAUDIBLE] this?
RANDALL LIN: Well, we're still working on it.
But I'd say, between me and the other guy, it's several months.
Yeah.
We're still always adding and picking up bugs.
AUDIENCE: So the [? naive ?] question
is, why not go the other way, which is just multiple Google
projects, multiple clusters, and you don't
have to do the namespacing?
RANDALL LIN: So it's dev-dependent.
So I guess, yeah, if you're in a world where you're
giving each dev their own project for whatever they're
working on at the time, that might work.
Yeah.
Yeah?
AUDIENCE: [INAUDIBLE]
RANDALL LIN: It's not released yet.
But yeah, we're certainly open to--
AUDIENCE: [INAUDIBLE]
RANDALL LIN: Cool.
SPEAKER: Awesome.
Thank you.
[APPLAUSE]
[MUSIC PLAYING]