[MUSIC PLAYING]
JEFFREY HUI: So I'm Jeffrey.
I'm a research engineer at Google.
And on the side, I love building side projects.
I love building little software tools for myself
and that sort of thing.
And one thing I've been really enjoying recently
is using LLMs within my development loop.
But calling an API for a model for prototyping toy projects
can get pretty expensive.
Or you might just be on a plane or a train or something
and not have access to stable Wi-Fi or internet.
So I'm here today to show you a quick example
of how you can use Gemma locally using the llama.cpp ecosystem.
So llama.cpp is an open-source inference framework
that allows you to run large language
models on your own machine.
So it's hyperoptimized for on-device, especially
for MacBooks.
It's written in C++ and has minimal dependencies,
making it efficient and portable.
It also supports a wide range of popular open-source and open
models, like Gemma, and has a thriving tooling ecosystem
built around it, thanks to a great open-source community.
And we made it really easy to work with Gemma in llama.cpp
by releasing the model on launch day in GGUF format.
GGUF is a self-contained checkpoint format
that's designed specifically to be used with llama.cpp.
It bundles together all of the necessary information
for loading and running a model in a single file,
eliminating the need for any other configuration files--
if you've worked with other checkpoints,
types like tokenizer, JSON, that sort of thing.
So it makes it very easy to use and share LLM models.
You only need a single file to get started.
Ours is hosted on Hugging Face.
So you can grab it right there from the Hugging Face repo.
And once you've downloaded it, it's very easy
to get started as well too.
You can clone the llama.cpp repository off GitHub
and then build it.
And then in one line, you can perform inference
on the model right on your device.
And so now let's dive a little bit
deeper into a demo of how you can actually
use the model in your own environment.
So before we start, before we go into the demo itself,
I want to preface this with a story.
When I was writing this demo, I was
trying to figure out what we should do here.
I was thinking about the 11-hour flight
that I had to take from the US West Coast over here.
And I thought about how the Wi-Fi might not
be as good on the plane, and I needed something
to do to pass the time.
Anyone here heard of the game Connections
from "New York Times," the minigame?
Anyone?
Yeah?
OK, we got some hands.
All right, cool.
So it's this really fun word puzzle game,
where they give you a bunch of words
and you're trying to figure out-- but the point is
that I'm addicted to it.
I'm obsessed with it right now.
And I want to play it more than once a day.
It only drops once a day.
So today what we're going to do--
if we can switch to the laptop now--
we're going to try to build a portion of this using Gemma.
And so for the uninitiated, Connections looks like this.
This is my very minimal version of it
that I've built locally on my laptop here.
You're given 16 words, and you're
trying to find four patterns where four words match
into the pattern.
And no word can be used twice.
And no word can go unused.
And so in this case, maybe a match
would be lawsuit, action, claim, and complaint.
Those are all legal words.
OK.
Correct, awesome.
I got it right.
[SIGHS] I was worried about that.
So on the back end, how this actually looks
right now is very not complicated.
I have all of the solutions hard coded.
I can add more puzzles, but then it's
just kind of a test of my memorization skills
as opposed to my word-association skills, maybe.
And so what we're going to do today,
we're going to do a little part of this project where
we're going to try to get it so that it's not fetching
the connections from the disk-- things that I'm writing--
and instead fetching from Gemma, hopefully with solutions
that I don't understand so I can figure them out.
So first of all, let's jump over to setting up the model.
So I'm going to set up the model in an interface.
Is the text big enough there?
No?
OK.
Sorry, give me one second.
Cool.
I've already preloaded the model in there.
You can see at the top there's--
I think that's just a bit of a UI glitch.
The model is loaded into RAM on this MacBook right now.
And I'm using this interface called LM Studio.
On the back end, it's still using llama.cpp.
But I thought this would just be nicer
to show in this demo environment.
So what we could do first is we can just
write a quick prompt, just to make sure everything's working.
Write a poem about Python 3.
OK, cool.
So I want to emphasize that this is running on this computer.
And this is, like, a MacBook that you can just
buy at the Apple Store.
It's not this special computer or anything.
And what's cool about this interface
is we have this nice chat interface
that formats everything, and we can show it real nicely.
But at the same time, we also have the ability
to set up the same model loaded into RAM as a model server
instead and access it programmatically.
So this will become a bit important later,
and we'll go back to this.
Do we have a beautiful poem?
We do.
Awesome, so this works.
But let's not get distracted.
Let's go back to the task at hand.
So I've written down here a series of patterns and words
that I've seen in past Connections puzzles.
And what we're going to try to do
is we're going to use these as part of the prompt
to try to get Gemma to generate more.
So let's make sure--
so let's tell Gemma, below are a series of patterns and words
that fit those patterns.
And then we'll say, generate four more patterns
and four words for each.
And we'll see what that does.
[LAUGHS]
There we go.
OK, cool.
So we're generating patterns pretty well.
And I think some of those are--
they might be a little bit easy.
But it's doing it.
It's doing it correctly.
And it looks like it's generating things
that I wouldn't expect, maybe.
That last one is interesting.
So this is good.
This is awesome.
But since we want to access it programmatically,
we want to hook it up to the front end.
Maybe we want to put it in a format that's a little bit more
friendly for programming.
So why don't we go back up here?
And we'll double click.
And we'll put it in JSON.
Generate four words for each in a JSON format.
And we'll edit the prompt like that,
and then we'll go ahead and regenerate that.
And it looks like we're successfully generating a JSON.
And that's really good.
And so one thing I want to emphasize here
as we're watching this generate is that what's important
is that there's a lot of this going on as you're building
something like this, right?
[LAUGHS] So there's a lot of noodling
about when you're building a prompt to use
a model in your app.
And so you're doing this a lot.
And if you're calling an API for this, it can get expensive.
You might be disincentivized to use it because you're
paying money for each call.
And so with a local model, you're
able to iterate really quickly and for free.
It's just on the computer.
And so what I'm going to do here is--
I'm not going to bore you with me iterating over
this prompt a little bit more.
But you might iterate a couple more times
and end up with a prompt that looks like this.
It might look like this.
And so we're going to paste it in, and I'll explain.
So this is a prompt where--
I've actually only worked at it for about five minutes or so,
just changing certain things.
In this case, I'm purposely telling the model
that I want things in JSON by formatting
all of the examples in JSON instead of mentioning it
at the bottom.
And I've added a couple extra things at the bottom just
to make sure we're taking care of that extra verbosity.
And so we'll give that a run.
As we can see here, it looks like we're
able to generate some interesting patterns here.
And I can generate this a couple times.
And you'll note that, while we have
a couple of different examples, it's
pretty robust in that it always generates in the same format.
And this will take a little bit of tweaking to do,
and that's what I've done behind the scenes with this prompt.
But we can get to somewhere where we can use it in the API.
So what we're going to do now is,
we're going to take all of this.
And we're going to start up our server as we--
and this is going to be the same exact model that's
loaded into RAM.
And now we're going to switch back to our code.
And we're going to try to use this fetchConnectionsFromGemma.
We're going to put our prompt right here.
And then we're going to pass this into this function
and now call Gemma instead.
We're going to save it.
And now when we refresh the page,
we're loading because we're generating the patterns right
here.
And, again, I want to say, this is a normal laptop.
There's no trickery behind it.
We're generating the tokens on this laptop.
And when it's done generating, I can go back to my game,
and I have a brand new game.
And I can just do this as many times as I want.
[APPLAUSE]
What's most important that you should
take away is that now I have unlimited games, and I'm not--
anyways, no.
The important thing that we want to take away from this
is that when you have a model that
can run in local development, it really
unlocks a lot of things that you can try,
because you're no longer bound by calling
an external server that might be expensive, might be tedious.
And you can also use a model--
I used the model 7B on this computer.
It's a very good model that you can also deploy.
So now you can do a one-to-one local environment
with your prod environment.
And anyone who's developed any number of apps
will tell you that that is the best thing to have.
And so I'm very happy that we're able to run this like this,
and I'm very curious to see what everyone builds with this.
[MUSIC PLAYING]