So, the URL that you’re seeing here, the website that you’re seeing here, is Intel®
Developer Cloud.
So, this provides you with an interface to access the latest and greatest hardware that’s
available, Intel for data center requests requirements.
So, all the way from the Intel® Xeon® processors to the GPUs, the data center GPUs, series
GPUs, [and] the Max series GPUs.
Also, you could access a bunch of instances from this URL.
I'm just going to go sign in here.
I’ve already signed in, and it gives me the page where you have the option to select
the type of instance that you want to launch.
Like I said, you have access to Intel® Data Center GPUs Max Series, Intel® Data Center
GPUs Flex Series, and the last generation of Intel® Xeon® processors available as
bare metal systems out here, right?
So, for this example, we are going to select this configuration of the Intel® Data Center
GPU Flex Series 170 along with the 4th generation processors.
Yeah, 4th generation processors.
So, this comes with an operating system of Ubuntu 22 and has all the basic drivers that
you need to run your demo.
Right.
So, once you select the requested system, you can just close [it] here and then launch
and click here to request access.
You type in your company name and allow where you want to.
The way you want to access these systems and when you click on request access request instance
it goes for approval and should be done in 24 hours.
So, I’ve already done this process, so I have an instance listed to me which is available
out here.
So, what I’m going to do is I’m going to access the system now.
OK, go to terminal.
And access the server.
Great.
As you can see, I have a bunch of variables that have been set and... sourced here as
such through Shell script.
So, for this demo, we’ll be using the Big DL framework.
So, the Big DL framework has multiple components in it, and the latest component is the Big
DL LLM.
So, this enables you to run large language models on Intel® hardware.
So, this could run your models that are in 4 quantization.
And specifically with GPUs, you have the option to access it through the Transformers plugin,
and also you have native support for LangChain plugins, right?
And as you can see that like you have examples to run various models, so you have the option
to run Llama 2, but by extension, all its variants, as well as Code Llama, the latest
model for code generation, right?
So, you also have other models like StarCoder, Falcon, and also Whisper, right?
So, you got all these different models out here, and let’s...
We click on Llama, you can see these steps as such to run the model, right?
So, first, you go ahead and create an account environment and install the required libraries
for it.
Specifically, we are going to be installing a Big DL LLM, so when you install this version
of the Big DL LLM, by default, installs the Intel® Extensions for PyTorch 2.0 for XPU.
So, this utilizes the optimizations available through oneAPI library.
So, once you get everything installed, you source the environment variables and set a
couple of configurations for optimization.
And once you do that, you’re good to go.
So, what you are going to do is we’ve already done that, and we’ve logged into the system.
So, let’s go and start our Jupiter lab.
So that it can run the instance, right?
I’m selecting this URL and just pasting it over here.
See, that notebook is getting loaded.
Fantastic.
Now, let’s add another terminal.
So, this terminal, we’re gonna watch the GPU utilization, and to do that, the command
would be xpu-smi.
So, xpu-smi gives you information about the hardware.
So, about what is the utilization for our various engines that started the GPU, the
utilization from my memory standpoint.
And as you can see that the memory utilized is zero, and the compute engine utilized is
also zero over here, right?
All right, let’s jump into the code.
So, from a code standpoint, if you look at it, the first 4 imports are pretty straightforward,
right?
So, if you’re going to use a Llama model, then you would have to use a LLaMa tokenizer,
and we’ll also see this demo on how you can use the Big DL-based wrapper to run the
Llama 13 billion model on your Intel® Data Center GPUs Flex Series 170.
Right?
And the way we’re going to run it is through creating a LLM chain and sort of running inference
to show that you can directly use this as a drop-in replacement for the default AutoModelForCasualLM
that comes with the Transformers library.
Right?
So, these are the regular ones and two additional ones, this one specifically for the additional
optimization that you get from Intel® Extension for PyTorch.
So, you import that, and also, you change the instance of the default AutoModelForCasualLM
being imported from Transformers you imported from the Big DL library, right?
So, let’s go ahead and run the imports, and we specify what model we want to run here,
right?
Point to node is [a] 13 billion chat model from Llama.
That’s where you’ll learn.
And the rest of the stuff is, like, pretty typical, right?
So, if you look at the utilization for AutoModelForCasualLM, you specify the model name for the... to load
it from the Hugging Face hub.
So, load it in 4bit indicates that you want to do... you want to update precision type
of int4, right?
And also, the regular other parameters are like just the regular ones that typically
do it, right?
Once you created...
once you’ve instantiated your model, you moved it to your XPU, meaning here you’re
going to be moving it to your GPU device, and then you instantiate your tokenizer as
well.
Let’s just do that, since a model weights have already been downloaded, it’s just
directly loading the checkpoint straight away.
OK, now what we do.
As then, we create a custom wrapper for the LLM class that we’ve imported from the LangChain,
right?
So, this is where you create, you take your input prompts, tokenize it, and move tokenized
prompts to the XPU device.
So, once you have the input tokenized, then you pass it to the model to generate your
outputs, right?
And here, we’re going to set the maximum token size as 512, and once you have the results,
you’re going to decode it with the tokenizer and then publish the results, right?
So, this is a simple function within your class, the custom LLM class, because this
is the configuration expected by LangChain, right?
We’ll just create this custom LLM class, and then we go and create our prompt template,
right?
So, we give the basic system instructions for what’s needed for the model and the
text; the input text that we pass as an input for the model gets passed as this variable
text as a placeholder text, right?
So, you do that, and you also specify what prompt, what’s the final prompt, and what’s
the LLM that’s going to be used by the LLM Chain.
So, you create those.
Now, you start your inference.
Now, before starting the inference, you can see that the GPU is closely like 50% utilized,
and the GPU... from a memory standpoint after LLM computed in standpoint it’s like 0%
utilized, right?
So, let’s go ahead and run our inference.
It’s running.
See that close to 60% is being utilized here.
Then increased to 64.
And finally, you have the output, right?
So, you have the list of different recommendations in terms of series that you would like to
watch and say if I go and query the model, what, who are the main characters of Stranger
Things, it’s going to give me a result.
And you can see that the GPU utilization is like going up to 100% here, right?
So, all the computation is happening in the GPU, and yeah, you have your output as the
main characters, which... who are like being listed here.
So this way, you could create a custom LLM wrapper, and you can use the LangChain.
You also have instructions here on how to... how to use it with LangChain natively, right?
So, you have LLM’s...
LLM drop-in placement for LangChain LLMs as well as drop-in replacements for your embedding
models.
So yeah, this is solved.
You could use your Intel® Data Center GPU Flex Series 170 for LLM inference.
Thanks.