[MUSIC PLAYING]
ANU SRIVASTAVA: Hey, everyone.
I'm Anu.
NIKITA NAMJOSHI: And I'm Nikita.
You've probably been hearing a lot of buzz
about large language models, or LLMs.
ANU SRIVASTAVA: So we're here to show you
how to take the flexibility and versatility of these models
and use them in production applications on Google Cloud.
Your machine learning on Google Cloud begins with Vertex AI.
Vertex AI provides ML tools, services, workflows,
and infrastructure, all accessible through
a simple and unified platform.
Now Vertex AI supports generative AI
with two new products--
Model Garden, which gives you access to Google's latest
foundation models, like PaLM, and a variety
of machine learning APIs that span text, image, code,
and more.
The other product is Generative AI Studio,
which provides a simple interface
to explore, prototype, and customize these generative AI
models.
Both of these together make it easy for developers
to build with generative AI.
So let's dive in and check out Generative AI Studio,
which lets you quickly explore and customize
generative AI models.
Today, we'll be working with large language models.
Then, we'll show you how you can leverage these models
into your applications on Google Cloud,
going from prototype to production in just a few steps.
NIKITA NAMJOSHI: One of the many reasons
people are so excited about generative AI
right now is that it allows you to prototype applications
really fast.
With the right tools, you can experiment with new ideas
in minutes instead of months.
You can find Generative AI Studio and the Vertex AI
section of the Cloud Console and select the type of content
for your use case.
To get started experimenting with large language
models, click on New Prompt.
In the world of generative AI, a prompt
is just a fancy name for the input text
that you feed to your model.
You can get one of these models to take
on all sorts of different behaviors
based on how you structure your prompt.
This art and science of figuring out the best prompt
for your use case is called prompt design,
and it often involves a lot of experimentation.
But that's where Generative AI Studio is here to help.
So let's start with a free-form prompt.
One way to design a prompt is to simply tell
the model what you want--
in other words, provide an instruction.
For example, create an outline for an essay
about hummingbirds.
We click Submit, and you can see that the model returns an essay
outline.
So let's try something a little more complicated.
Here's a synopsis of a fictional TV
show about a team of biologists working in a lab.
This text snippet includes both the names of the characters
as well as the actors that played them.
We can instruct the model to extract
the characters and the actors who played them
from the above message.
We send this text to the model, and you
can see that the model does, in fact, extract
the characters and the corresponding actors that
played them with a friendly note at the end.
To make this information more structured,
we can even instruct the model to create a table.
We just need to modify the prompt a little bit.
For an added challenge, let's extract the job
that each character has in the show as well.
We'll change the prompt to say, create a table
and then provide the fields we're interested in--
actor, character, and job.
And this time, our model outputs a table
with the information we requested.
This approach-- writing a single command
to get the LLM to take on a certain behavior--
is called zero-shot prompting.
Now, for this use case, our model
was able to complete the task with just an instruction.
But depending on your use case, it
can be helpful to provide some examples
and show the model what you want.
That's where few-shot prompting comes in.
In this approach, you provide the model with a few examples.
And to make it easier, you can use
a structured prompt template in Generative AI Studio.
The structured prompt contains a few different components.
First, we have the context, which instructs
how the model should respond.
You can specify words the model can or cannot use,
topics to focus on or avoid, or a particular response format.
And the context applies each time
you send a request to the model.
So if we wanted to use a structured prompt for our use
case, we'll need to add a synopsis as an example input.
Let's use the same one from before.
We'll need to also provide the corresponding output
text in the format that we want the LLM to use.
So let's say we want to establish the text pattern--
actor, character, job-- so we can easily
parse the output from the LLM and store
the information in a database.
Once we've added examples, we can test our prompt
with a new summary.
The snippet here is about another imaginary TV show
and similarly contains the names of the main characters
in the show, their jobs, and the actors who played them.
When we click Submit, you can see
that the model extracted the relevant information
in the format that we wanted.
It's important to note that adding instructions
and a few examples tends to yield good results,
but at the end of the day, there's currently
no best way to write a prompt.
Experimenting with prompts is a core part
of being a generative AI developer.
You'll want to try out different structures and formats
and examples and see what works best
for your use case and the specific model
that you're working with.
So when you have a prompt that you
think is working pretty well, you
can save it and return to it later.
Your saved prompt will be visible in the Prompt Gallery,
which is a curated collection of sample prompts
that show how generative AI models can work
for a variety of use cases.
Finally, in addition to trying out
different prompts and prompt structures,
there are a few model parameters you can experiment with
to try and improve the quality of responses.
First, there are different models you can choose from.
Each model is tuned to perform well on specific tasks.
You can also specify the temperature, top-P,
and top-K. These parameters all adjust the randomness
of responses by controlling how the output tokens are selected.
When you send a prompt to the model,
it produces an array of probabilities over the words
that could likely come next.
And from this array, we need some strategy
to decide what to return.
A simple strategy might select the most likely word
at every time step, but this method
can result in uninteresting and sometimes repetitive answers.
On the flip side, if we were to randomly sample
over the distribution returned by the model,
you might get some unlikely responses.
By controlling the degree of randomness,
you can get more unexpected-- and some might even
say creative-- responses.
How you set these parameters will depend on your use case.
Lastly, if you want to explore additional customizations,
you can tune one of the base models for your use case.
First, you provide a model name.
Then, you point to the local or Cloud Storage
location of your training data.
Your training data should be structured
as a supervised training dataset in a text-to-text format.
Each record or row in the data will contain the input text--
in other words, the prompt--
followed by the expected output of the model.
After specifying the path to your dataset,
you can start the tuning job and monitor the status in the Cloud
Console.
When the tuning job completes, you'll
see the tuned model in the Vertex AI model registry,
and you can deploy it to an endpoint for serving
or test it out in Generative AI Studio.
And there you have it.
You've prototyped a data extraction and organization
system in just a few minutes.
ANU SRIVASTAVA: Now, prototyping is great,
but to really get the most out of your LLM
is to integrate it into an app or automate a workflow.
Large models on Vertex AI are ready for production use.
Let's now see how we can scale what we did in the Generative
AI Studio by using the API.
Nikita was experimenting with prompts
analyzing TV summaries because, let's say,
in our demo scenario, we run our own content platform.
And we work with creators to get their scripts into production--
a different type of production with actors and filming.
We need a system where we can quickly process and store
new files with key metadata, such as a synopsis
of the shows we're working on.
And we want to automatically extract the character names
and actors and store this in a database.
Let's take a look at how to use the API.
If you select View Code, you can see
the snippet you would need to use this LLM in your code.
We're taking a look at Python today.
This code snippet is handy because it shows us
how to set up the client library and make the request.
Let's take a look at the key parameters.
We concatenate the prompt and the user input
and pass through the parameters.
And we're ready to go so that when we call the API,
we get back a structured response.
And we can use the output to merge into a database schema.
We're using BigQuery, a Cloud relational database,
in this application.
Our processing workflow can update the database every time
a new summary comes in.
Using some common Google Cloud components-- like our Google
Cloud Storage, which stores our scripts,
and Cloud Run, a serverless compute platform--
we can create a processing architecture like this.
The Cloud Run job takes a summary,
runs it through the LLM by calling the Vertex AI,
then stores the output data in BigQuery
using the BigQuery API.
With Cloud Run, we can set up a trigger
so whenever a new file is stored in the Google Cloud Storage
bucket, the file is automatically
passed to our Cloud Run job.
Voila-- our BigQuery table is automatically populated
and can easily be rendered in a webpage or app
so we can keep track of our scripts.
That was our walkthrough of our Cloud-based application
that incorporates our LLM to get the maximum use out of it.
There are several benefits to using large models
on Google Cloud, such as state of the art models
Google maintains and upgrades for you.
Privacy-- my data is stored in my company's
private, protected environment, and Google will only
use it with my permission.
By enterprise ready, we mean the models can be used out
of the box, so you can count on SLAs
when you take your app to production to serve your users.
In conclusion, Vertex Generative AI Studio
helps you design and prototype with large language models.
And with the power of Cloud, you can elevate those experiments
into production as a key component of your apps.
Try your own prompts in the Generative AI
Studio in the Console and sign up for Google Cloud Innovators
to stay in the loop with the latest updates.
[MUSIC PLAYING]