[MUSIC PLAYING]
SPEAKER 1: Next up is Barni.
Barni Seetharaman is a software engineer
at Google, where he works on the open source Kubernetes project.
He has experience building network protocols,
distributed systems, and multi-cloud orchestration
platforms.
More recently, he has worked on enabling data processing
work loads on Kubernetes.
Quick note-- these are the lightning talks right now.
So each of them are just five minutes.
We've got just three more talks to go.
BARNI SEETHARAMAN: All right, hello, everyone.
Thank you for staying back for so long.
I know some of you may be sleepy.
But I hope it's not as boring.
So thanks for the introduction.
Most of you know Kubernetes.
If you don't know, it's an open source project from Google.
And it is a production-grade container orchestration system.
The key components you need about Kubernetes
is the API server, core controllers,
and the [INAUDIBLE] and the resources.
The resources are what we call declarative specs.
Essentially, you define your workloads and infrastructure
as specs.
And the controllers work in satisfying what
you've declared as your intent.
And we have something called Kubernetes Operators.
The operator pattern extends the core Kubernetes concepts,
so that you can build your own controllers, your own specs,
and extend the Kubernetes API to build interesting applications.
And one such application is the Airflow Operator.
This operator is different from the operator
that is used in Airflow.
It's confusing, but just hold that thought.
All right, so for Airflow Operator,
we define two custom resources.
One is the Airflow Base and the Cluster.
Output is show what they do.
And I hope I have time to give a quick demo.
I'm going to quickly skip over it.
So you have a controller that talks to the Kubernetes
Core, the API server.
And then you have the kubectl, which is the CLI.
So through the CLI, when you post an Airflow-based CRD--
a CRD is nothing but a YAML sign--
so the Airflow controller picks it up and then
works to schedule these instances for you.
So you could schedule MySQL, an NFS, etc.
And then you have the other CRD, which
is the CRD, through which you could
deploy the scheduler, the readers, workers, and the UI.
The reason we partition this into two separate specs
is you could deploy multiple clusters against the same base.
Meaning you could have just one MySQL,
and you could deploy multiple clusters schedulers against it.
We provide isolation.
Because when you deploy an Airflow cluster,
it dynamically creates a database
and then feeds it into the scheduler.
So they are isolated from each other,
even though they use the same database.
In Airflow, a pod is a unit of execution in Kubernetes.
Think of it like a VM.
Within the pod, you could run any of your services.
It could be the scheduler or a worker container, for example.
And we have something called sidecars
in Kubernetes, which is nothing but another container which
assists the main container.
So when deploying on Kubernetes, we
introduce the DAG sidecar through which
you could pull your DAGs from different sources.
It could be from the GCS bucket, or Git, or an NFS or even a DAG
volume that's on your hard disk.
The advantage is its a separation of concerns.
So it pulls all these DAGs and then starts
your scheduler or worker.
And the DAG sidecar also has a mode where it periodically
monitors for changes and then posts the latest files for you.
But we haven't figured out the easy way of restarting
scheduler and worker.
Maybe we will look into that.
So there are multiple uses cases.
I'll quickly touch upon them.
It's ease of deployment when you're containerized,
and you're trying to use Kubernetes.
The Airflow Operator makes it easy
for you to deploy Airflow on Kubernetes.
And it also allows you to spin up a temporary Airflow set up,
work on it, and then trash it away.
It's as simple as just kubectl create something.
Kubectl deletes something.
And the other thing is it allows you
to customize when you have different Airflow images,
like, your own custom images.
Each Airflow spec can have a separate image,
and then that's the way you can have--
Say, user one will use have his custom image.
And user two can use his own custom image.
Makes it easy to spin it up.
Some companies may have an admin model,
where the central IT spins up Airflow clusters for users.
And they could use the Airflow Operator to do that.
And it also provides isolation.
Because when you're working on the same Kubernetes cluster,
different people can spin up their own clusters
without tampering on each other or stepping on each other.
And also, it helps in automation.
The custom resource plus the kubectl in Kubernetes APIs
are easily pluggable into automation CICD.
If you want to set up a DAG, CICD could use this.
You have a Kubernetes cluster, and you have your DAGs.
If you make any changes, you could
use those to trigger tests on the Kubernetes cluster.
The use cases are still fluid.
But you could use it in different ways.
There are links to the pre-recorded demo.
If you get access to the slides, you can see a pre-recorded demo
here.
It's an Alpha version.
Its open source.
Got this on Google Cloud Platform and on GitHub.
There's a Slack channel, where we are taking messages,
and people are interacting already.
We welcome contributions.
We want users of Airflow on Kubernetes
to try it out, give feedback, and see how we can improve it.
So we think it's a good way to run Airflow if you're
setting up Kubernetes on prem.
And for cloud, of course, you have Cloud Composer,
which is a managed and well-thought out
production grid solution.
Any questions?
[APPLAUSE]
AUDIENCE: Hi, just a real quick one.
Is this going to be the default way
to run Airflow in the far future?
BARNI SEETHARAMAN: No--
That depends.
If you are using Kubernetes, we would recommend this
as a good way to run Airflow.
We would love for this to be the default.
So we are already seeing contributions
from the community for adding support for other databases.
But if you are using Kubernetes, and you are using Helm Charts,
we would recommend you do move to this.
Nothing wrong with the Helm Charts,
but just this is a little bit more powerful than that.
AUDIENCE: How much flexibility do
you have to configure your instance,
or how much can you [INAUDIBLE] use different executions.
BARNI SEETHARAMAN: That is a good question.
There is, in fact, a PR open for that.
But essentially, the good thing with Airflow
is everything can be set through ENVs.
And in the custom resource, you can inject any environment
variables you want.
So technically, I hope it's a good thing.
I'll see if I can connect and give a demo later.
But thank you for that.
[APPLAUSE]
[MUSIC PLAYING]