What skills do I need to build a talking
chatbot?
You will need to have knowledge
of programming languages,
as well as experience
with natural language
processing and machine
learning, understanding of chatbot
development frameworks and platforms.
Building a conversational
AI is a complex task
and requires
a lot of trial and error…Thank
you chatbot at Intel.
Our goal is to empower developers
to bring AI everywhere,
so I will show how you can create
and deploy your own
talking bot on an Intel device
within minutes.
It starts with a foundation model
that we will first customize
with our own data,
then add voice input and output,
and finally optimize
for the device we’re running on.
This probably sounds as hard
as the chatbot describes,
but it’s made much easier
with the Neural Chat library from Intel.
This is a set of
application building blocks
that work closely
with Hugging Face Transformers.
You can add plugins to the core models
to customize its functionality
and you can optimize it
for the device you're running on.
This is all built
using frameworks and engines
powered by oneAPI that optimize for Intel
CPUs and GPUs without changing your code.
So let's get started.
This is running on a 4th
Gen Xeon instance
on Intel Developer Cloud.
Here's a simple Python script
that will load the default model,
which is limited to 7 billion parameters
from Hugging Face.
Then it will send a text question
and receive a text answer.
Since it takes some time to load
the model and run.
I'll cut ahead, but
show the inference times.
So pretty simple.
Just a few lines of code
and we have a chatbot.
But what if we want
to customize the chatbot
with our own data?
There are a range of methods
to customize chatbot output.
.
If we want it to draw on our own data
but we don’t have the resources
to train a large LLM from scratch,
we could either fine-tune it
with our data,
or use Retrieval Augmented Generation.
Both of these can be done
using just CPUs.
Here we will use RAG,
which combines the language
capabilities of the foundation
model with our own knowledge base.
This doesn’t
require any re- training,
so our knowledge is not embedded
in the model
and we can easily update
the knowledge base.
Implementing RAG with Neural Chat
only requires
another couple lines of code –
to enable the retrieval plugin
and then configure the pipeline –
note that we’re using a different model
now, one that’s
going to be retrieving the data
from our little collection of docs
instead of it
all being statically contained
in the model.
And again,
skipping ahead to the inference output.
We can see some different results
which are drawn from our knowledge base.
Now to add on the voice IO,
there are actually two plugins,
a speech recognition
model to generate textual
input from the audio.
What is mixed precision inference
then one
to convert the text output to speech
and you can customize the voice.
In machine learning,
mixed precision inference
refers to the process
of using both floating point and integer.
Inference refers
to the process
of using both floating point.
You might have noticed
that the voice IO models
add runtime overhead
and maybe you need
the response to be quicker
or you need it to run
on a smaller device.
Neural Chat makes it easy to add
Intel Neural Compressor model
optimization techniques
that are specifically tuned
for transformer models.
We can just add to the config to tell it.
For example, to use mix precision
or in this case here to perform
weight only quantization.
This runs quantization
during model loading.
So this sped up the inference pipeline
by almost two x and it still produces
quality speech output.
Mixed precision inference
refers to the process
of running deep
learning models
with both floating point FP.
Now, these have been simple
Python scripts, which load in the model
every time we run inference.
Neural Chat also provides a model
serving capability, either
with Python or from the command line.
This also allows you to enable
and configure the plugins in a yaml file.
Here we're using the text based chat bot
with the retrieval plug in.
The server is already
running on this machine
and we run the query from this machine.
That was a brief introduction
to neural chat.
To get started yourself,
there are sample notebooks
in the GitHub repo
along with example Python scripts
which you can access with a QR code here
or from the links below.