[MUSIC PLAYING]
MOFI RAHMAN: Hello, everyone.
My name is Mofi.
I'm a Developer Advocate at Google Cloud.
My focus area is mostly on Kubernetes and running
AI workload on Kubernetes.
So quick hands on how many people have heard of Kubernetes
or used Kubernetes before.
Keep those hands raising for a second.
How many of you have used GKE?
If you haven't, drop your hands.
Ooh.
Ah, still quite a few number.
So GKE is Google's managed Kubernetes.
So we're going to talk about Kubernetes, how
to run AI workload on Kubernetes,
and specifically GKE in this case.
When you talk about building cloud-native applications,
there is--
Kubernetes is probably the first thing a lot of people
think about when they think about cloud-native application.
And when we're talking about AI workload on Kubernetes,
I specifically think of three separate use cases.
Number one is you have a model that you want to just make sure
that other people can access, so the serving
inferencing use cases.
Number two is you have a model, or don't have a model,
but have data that you want to train the model with.
So you want to run fine-tuning or training
job, both full fine-tuning or LoRA fine-tuning of that case.
And then, number three, which is probably the most common use
case, you have an application that you
want to serve somewhere.
And that application these days might be talking to something,
so they access AI-related APIs and other things.
And that API might not even be running in your cluster.
That API could be a Gemini API or something like OpenAI API
that you're calling from your application.
So we want to make sure that all three of these use cases
are well served.
In that regard, why would you want
to even use Kubernetes for things like this?
So if you look at a spectrum of things
that are fully self-managed to fully managed for you,
something like Vertex, that my colleague just talked about,
would probably be in the furthest end,
where you just care about your application AI data and models,
everything else is taken care for you.
Your own data center or something like a bare metal
server probably will be all the way in the other end, where
you manage everything from updating your Linux
distribution, installing wiring, figuring out how much network
bandwidth you need, all that stuff.
Kubernetes probably sits somewhere in the middle.
You have a lot of things you can tweak and control.
But for the most part, a lot of the upgrade and update
gets taken care of.
So you have the flexibility of bringing the tooling
that you want to bring in, like the library that you
want to use, the software you want to use.
And once you build the container,
that becomes that portable unit of operation
that you can take from this cluster to other provider,
to your own data center that's running Kubernetes.
Performance-wise, again, Kubernetes just
talks to the underlying infrastructure.
So at GKE we can give you all the necessary resources
like GPU and TPU for you to get the best
performance for your application that you want to run.
And, finally, the efficiency is both efficiency
in terms of getting the right resources,
as well as being able to manage an autoscale environment that
can scale down to give you the best
cost-efficiency to be able to run things at large scale.
Now, at Google, even before Kubernetes,
we have been doing things with containers for a while now.
At Google, we have been running containers for roughly 20 years
for pretty much all the big applications that we run.
And all that institutional knowledge
have been baked into our managed service for Kubernetes,
which is GKE.
We can provide unparalleled scale
compared to the rest of the industry at this point,
as well as a new mode of operation
for Kubernetes called GKE Autopilot, where you don't even
have to think about the nodes you're running.
You just define the application you want to run in a YAML format
and Autopilot takes care of provisioning the hardware,
running your application, and scaling them down
when you no longer need them.
So we talked about the three modes of operation
I think about when we're talking about running
AI workload on Kubernetes.
So the first one we're going to talk about is taking a model,
like Gemma in this case, and fine-tuning it.
I think we saw an example that Keelin just
showed to the notebook.
But when you want to take that same experience
and somehow package that in a container, the task
itself is not that much different.
We want to just take the same code, package that in a Docker
file, get a container out, and that container then
can run in a Kubernetes environment.
So we're going to look at an example of fine-tuning
Gemma on GKE.
Before I start the video, which is going to load in a second,
in the screen, what you see is the Google Cloud Console
for GKE.
And I have a Kubernetes cluster with three node pools.
One of the node pool is just your general CPU nodes.
Two of the other node pools, the G2 standard 96 and G2 standard
8, one of them has eight L4 GPUs and one of them has one L4 GPU.
The one that has eight L4 GPU, that's
the one I'm going to try to use to fine-tune a Gemma model using
some data set from Hugging Face.
And the reason I'm trying to fine-tune this model, I'm
going to take the pre-trained model of the two billion
parameter model, but fine-tune it using a data set
called SQLContext.
So what that does is, if I give it a question about some table
and give it a table structure, it
should be able to give me some SQL query
that I can then run against my database.
So I'm going to start the video.
So this part goes pretty quickly.
So I'm going to select, again, these are the two node pools
that I have with GPUs.
And the code is taken from a notebook.
I just copied all the codes of the notebook blocks
and put it in a Python file.
And I'm setting most of the variables
that I need, like my LoRA config.
If I'm using a float point 16, some optimizers.
I'm also going to set up some of the data set that comes in.
I'm going to change the data set into some different format
that I need, as well as the LoRA config.
I'm going to have some training arguments.
All of this stuff you have seen if you went through the notebook
yourself.
But again, all of them are now in a single data set file.
Also I run the trainer [INAUDIBLE] train.
After the model training will finish,
I will then combine the model into a single fine-tuned model,
and finally upload that all up to Hugging Face.
I will need a Docker file to make that in a container.
So that's using the Docker file there,
and put that into YAML to talk to Kubernetes.
After all of that is done, the only thing left to do
is to run the command.
In real life, this command, end to finish,
takes about 33 minutes.
But, for time's sake, I have sped up the video.
So this whole thing is going to end in about 30 seconds.
But what you will see if you can read the font-- which is not
super big, I notice now-- is, again, the training
is happening.
I have 5,000 total rows in my data set,
which is about 5,000 steps.
But, because I have eight GPUs, you're
going to see my total step is actually 625,
because each GPU breaks it into--
each step is about eight steps in total in real time.
In about 33 minutes, I have my loss function,
I have my model created.
After that is done, my model weights from the LoRA weights
will get combined with my original model
and then pushed back up to Hugging Face.
And in a second, I'm going to also go to Hugging Face.
The last time I had pushed this was about four days ago.
But when the video was recorded, the last time
I had pushed this was 38 minutes ago,
and which updates to now less than a minute ago,
so after I finish uploading.
If you were to go back to this link,
which is MofiCode/Gemma-2b-sql, you should see that the file was
updated about four days ago.
So that was the first part, we fine-tuned our model
and then pushed it back to Hugging Face.
Now anybody on the internet can go and grab that model
and set up their own serving that
will give them a model that is now
equipped to handle SQL queries.
Now, before we go do that, let's just talk
about what all of that, as a big picture, looks like.
At the very bottom layer, we have our accelerators
that we want to use.
We have our GPUs in this case.
We also have TPUs, which are custom-built hardware
for doing AI workload like training and fine-tuning.
And, on top of that, we have our Kubernetes platform,
that is GKE, that you can have not only your application
running, but as well as AI workload running.
On top of that, you can think of different types
of workload you want to run.
You have your training, fine-tuning,
that's one type of workload that requires a lot of resources
for kind of a long time.
And the other side, you have inferencing,
which is more things that needs to be running constantly
as deployment.
So in that case, being able to optimize resources with that--
if you are using an 8-GPU constantly,
that's probably more expensive than having
smaller GPUs that can do serving constantly as a deployment.
And finally, on top of that, you can bring any kind of tool,
like your PyTorch, your Keras, other tools in the open source
community as well as closed source,
that can run on Kubernetes that can give you
the complete picture of your AI ecosystem.
And as for this release for Gemma, from the GKE team
we released four different guides
that allows you to figure out how
to do serving on GKE using open source tools that
exists in the community.
So we have, for GPU, we have vLLM, Hugging Face, TGI, as well
as TensorRT-LLM on Triton server from NVIDIA.
And for TPU, we have the example with SaxML
that we can run to see how serving the same model
can be a little bit different across these four
different options.
Now that we-- the last thing we're going to talk about
is the fine-tuned model that we just pushed up,
how do we go about serving that model on GKE?
So the four guides I talked about, the one of the guides
is serving open models using GPUs
on GKE using Hugging Face TGI.
TGI stands for Text Generation Inference.
I think a previous speaker already talked
about that world of serving large language models.
So this guide goes through surveying Gemma 2B, 2B-IT, 7B,
and 7B instruction 2 inversion of the model on TGI.
So for us to get started, all we basically
need is that YAML that we are using here.
But in this case, instead of using the Gemma 2B model
directly, we're just going to swap that out
for the model we just tuned, the model we
have under Mofi codes, SQL, something, something.
So we can copy this whole YAML over to there.
And we're just going to replace this value with the value here,
MofiCodes/Gemma-2B-SQL.
And after all the secrets are in the right places,
all you need to do then is run this application.
Slowly type that.
And again, this process to download and set up serving
takes about four minutes.
Again, through the magic of video editing,
we can get this done in about four to five seconds.
But the basic idea is the same.
You deploy, it goes to Hugging Face, downloads the model,
sets up all the weights and tokenizers in the right place
so that it's ready as a server for us
to go then make HTTP query against it.
Now I set up a port forwarding over there
so that I can call it from my localhost.
You can also set it behind the load balancer
if you want to get the same experience across the internet.
Now, I talked about how this data set was about SQL,
so the user prompt I have is some question about a data,
and I'm also providing the context of what
does the database look like?
So I have a database that is, create table farm of some data
set.
And then, when I make this query against my table,
the model that previously was just giving random answers,
now it can answer things like, give me a select average
working horses from my table.
So again, I haven't actually tested,
but from the looks of it, first glance,
it looked like it's going to work.
But if it doesn't, I can always go back
and fine-tune it even more to fit the need I have.
So in this short, 7 to 8-minute video,
we saw the whole flow of--
you have data, you take that data, and fine tune any model
that you want to.
And then push that model up to Hugging Face.
And then also set up serving so that not only you,
but other people also can take advantage of a fine-tuned model
to use in an environment they want to.
So if you can set up a load balancer against it,
other people on the internet can be using it.
Also locally from your cluster, other applications
can hit that endpoint to get data out.
So we're super excited to see what you will all be building
with these kind of tools.
Again, in a GKE cluster, you can have
a node with a huge amount of GPU,
not just L-4s, but H100, A100s to fine-tune really big models,
not only from fine-tuning with LoRA,
but also full fine-tuning or training
and get similar results and great results.
[MUSIC PLAYING]