(bright music)
>> Hello, and I'm John Furrier with theCUBE.
We're here, Palo Alto, California,
for exclusive news around generative AI.
Of course, we've been covering it like a blanket
for many months now, and it's been a surge of activity.
We're here with two CUBE alumnis,
the CEO and co-founder of OctoML, Luis Ceze,
who's been on theCUBE many times before.
Going back before the craze started,
we were chatting a lot about AI.
Welcome, Luis, to theCUBE.
And of course, the investors was Madrona Ventures,
and Jon Turow was on the board of directors of OctoML,
also an expert in AI as well,
former AWS, now a VC partner.
Matt McIlwain did the original investment,
give him props there.
We're here to discuss the hot news in generative AI
that's being released today.
Luis, Jon, thanks for coming on theCUBE.
>> Jon: Glad to be here.
>> Okay, let's get into it.
Luis, we've talked many times
about what your role has been in the industry,
obviously with OctoML and that startup.
Generative AI has been going crazy.
You have news that you're announcing today.
First, optimized compute service for AI,
which just sounds like a great product,
I want to hear more about it.
A platform you guys say that's going to help developers
have a fully managed cloud infrastructure
to make it easy to write AI apps.
Basically take all the heavy lifting
and abstract away all the complexity.
Let's get into it.
What's the news?
This is the hottest area, generative AI.
People want to do more.
GPUs are on allocation, you can't get any.
There's all these different services out there,
a lot of confusion, a lot of noise, but it's hot.
What's the news?
>> All right, so we are releasing today
the Octo AI Compute Service.
So it's the first self-optimizing compute service
for generative AI.
It offers freedom because it allows users
to go choose their model or bring their own custom models.
Second, offers efficiency because we optimize the models
and we choose the right hardware,
and we make sure that it gets the right
performance-efficiency tradeoffs.
And then second, it's very easy to use.
So we make it very easy for folks to get started.
We offer a collection of super-optimized models
like Stable Diffusion, LLaMA-based LLMs,
Whisper for audio transcription, and so on.
So with freedom, efficiency, and ease of use,
offering this service that makes very easy for folks
to build really amazing generative AI applications.
>> There's a lot of confusion out there
around how to get into the gen AI business.
Some people are leaning into it,
have been doing well with it,
but the rapid pace from idea to code to product,
I've never seen this kind of acceleration.
What problem are you solving for the developers?
Obviously this interest is high.
What problem are you solving specifically?
>> Yeah, so several problems.
First, abstracting away the complexity
and helping clear this confusion, right?
So we offer the ability for developers
to come to the platform, select a use case,
for example, text-to-image or text-to-text,
and very quickly get started
with state-of-the-art models ready to go
and ready to be integrated into their environment.
And then we also abstract away
all this incredible complexity
that is involved in putting a model into production.
Because once you have a model to deploy,
the path from there to actually deploying
in a way that, first, has the SLA that you need
to make your application usable,
and then two, has the right cost
and scalability of properties, is a lot of work today
and we completely abstract it away
and make it fully automatic.
>> Jon, you've been covering this area,
we've been on theCUBE talking about this before.
Madrona, obviously, early stage investor here
in the company.
The world spun right in the front doorstep of OctoML.
What's your assessment?
What do you see in this new platform?
What's the outlook?
What should people think about it?
What's your perspective?
>> Well, what's really exciting
about the world that we're in right now
is there's almost sort of an Android moment
that Luis and I have written about before.
That, look, we have very exciting models,
you might call them the iPhone kinds of models,
things like GPT-4, an AI model from OpenAI,
and ChatGPT, Anthropic, Cohere,
lots of other models like that
that are genuinely really powerful, really exciting, good.
There is also a world of open source AI models
that are really useful in their own right
and together with the closed models.
And the open source models are special
because they let developers and scientists
push the boundaries on functionality,
on data types, on cost, and on latency.
And you can assemble these models
into what are called ensembles,
some people call them cocktails,
of lots of models that work together.
And while that is good and we've seen great companies
like Runway ML and Midjourney and some others
built on top of open source AI, it's been really difficult
to get started with that until now
and to manage it and to run it all.
And so what's exciting about what Octo
and Luis and the team are doing
is that they're going to be able to give,
for the first time, the kind of ease of use
with open source AI
that you're getting with the closed models.
And that's going to unlock lots of new innovation.
>> Let's get into the open source content here, Luis,
because, you know, Jon,
we were talking before this interview
on text and also in person about the long tail.
There's a lot of long tail stuff coming out.
You got the power law.
I've never seen power law
applied to this kind of phenomenon before,
but open source is making a real impact.
More people are coming in, a lot of experimentation,
people are looking at different scopes
of the size of the models.
They don't need to be huge, big models.
They can work with existing.
There's a blending of cocktails, if you will.
Why is this important?
What's the big thing that's going on, Luis?
Why is this so successful?
>> Well, I mean, the pace of progress
in these open source models is just absurd.
And they're showing to have value, right?
So I'm talking about literally on a weekly basis,
there is new models that are released
and these models are actually very useful
for problems that businesses have today, right?
So now, the big problem here,
and the anxiety this generates
is that how do you write all of that?
How do you actually, you know,
how do you harvest value out of these models
and add it to your business
in a way that's sustainable
is exactly our goal at Octo
and now we're extremely focused
in making these open source models
that are moving so fast work for you
to work for the business, right?
So it'll enhance the application creator's focus
on what experience they want to offer their users
and abstract away all of the complexity
of the infrastructure and details
involved in getting these models
to actually function in production, right?
So that's our laser focus here
is let businesses extract value from these models
that are moving really fast.
>> Jon mentioned proprietary Android, iPhone.
I like the analogy, but it's interesting.
OpenAI doesn't run on AWS yet,
Anthropic does.
OpenAI is on Azure and you got Cohere out there.
They interplay and there's interaction between them.
What is this?
Is this part of the equation?
Because I see lock-in there,
I don't see a lot of choice
if it's something that runs only on AWS
or something only runs on another cloud,
what does that mean?
'Cause there's consequences of that.
I mean, OpenAI is not going to go and run on any cloud,
what good is it?
Or is that going to be proprietary?
Are we back to the old network protocol stacks
where you had to pick something and stay with it?
How do you guys talk about this
and how do you see it unfolding?
Because it doesn't make any sense to me.
>> I think there's a very early moment that we're in
where a lot of these questions are open.
You can use OpenAI from AWS,
it's just a little harder, things like that.
And that's the moment that we're in.
But the clear movement
from enterprises and from developers that I speak to
is they want to access this powerful technology
and they want choice of the best model
and the most useful model
and combination of those models
for a use case that they're building.
>> Luis, what's your take-
>> Luis: Yeah, and if I may-
>> On theCUBE that developers in open source
are the defacto standards bodies these days?
They're the ones driving the change.
If you look at just what's happened
in the past four months alone in open source,
it's been pretty incredible.
Explain the phenomenon there.
>> Yeah, so let me answer a mix of the two questions
that are in the air here, right?
So I'll start with first, I mean,
it's undeniable that these closed proprietary models
offered by, say, OpenAI,
have incredible functionality, right?
So, and you know it takes a lot of money
and a lot of effort to actually build those models
and understand, you know?
And rightfully, they're building a business out of it.
And these models, say, models like GPT-4,
they do truly surprising things
that are very, very powerful.
But what Jon said is really important,
which isn't like, you know,
it's important for users to ask the question,
"What is it they want to solve?"
Like do they want to solve a question answering service
over a set of documents?
Do they want to do a text summarization service?
So they want to do text classification?
Do they want text-to-image?
And once you understand your use case,
it became really clear
that there's a large set of open source models
that quickly progressing
that can solve those specific use cases
and address them very, very directly.
Now, and the end user advantage here
is the following, right?
So you can, for the use case
that you know that can be fulfilled by an open source model
or by a customized version
or a fine-tuned version of those models,
you can control the deployment of that model.
You have control of where it runs and how it runs,
what data it's going to touch.
You have full transparency on what it does
and you can assign, you can use that
in components of your application that make sense.
That said, it's also important to realize
that once you need the functionality that, today,
is only available in the large proprietary models,
you can still call them.
The reality today that Jon alluded to
is that people are building with an ensemble of models
where you combine a collection of, say, open source models
that do things that you know that they do well,
we can validate that,
but then once you need this functionality
only offered by the closed models, you can call them, right?
So all that said, I think like if you zoom out
on the market and the timeline too, it's interesting to see
that the gap between,
this is a point that Jon and I made before,
and supported by others in the community,
that the gap in functionality
between closed proprietary models and open source models
in terms of functionality is shrinking, right?
So, and then if you project this out over time,
I think it's easy to see a world where yes,
you can probably fulfill all of your,
you'll probably be able to fulfill all of your AI needs
out of open source models.
And that means that the big challenge then
is like do you have a platform to run them on, right?
>> Yeah, and I think this is where I like the news.
Let's get back to the news.
So GPUs are on allocation, they're hard to get,
they're jacking up the prices,
NVIDIA's stock is soaring,
although they just invested in Cohere recently last week,
we saw that news.
NVIDIA's a player, okay?
Amazon has a cloud and they're a great footprint,
carbon neutral, they say, by 2025.
And these large language models aren't helping the world.
I mean, talk about carbon footprint impact, you know?
I was talking to a customer,
they said the sustainable budget,
their budget and their metrics are exploding
off the charts because the sustainable goals
are being not met
because they've been blown out by all the compute and that.
So it's a lot of pressure.
At the end of the day, the developers just want to code.
What do you guys offer the developer?
What's the pitch of this platform?
Why use you guys?
What does this mean for me?
I want to get in, get my model's nailed down,
I want to understand how it all fits,
I'm tinkering, I'm kicking the tires.
How does it work?
>> Yeah, absolutely.
So, and this topic's obviously near and dear to my heart
because of Octo, the Octo AI Compute Service,
but also because I'm a computer architect
by training, right?
So I like seeing chips being so important
in this new phase of the world here.
So what we offer to the developer, again,
is the ability to not have to worry about infrastructure.
What does this really mean?
It means that really making the deployment viable,
which, in a first instances,
you have access to the silicon, right?
So because of our ability to deeply optimize the model
and how it runs on the actual hardware,
we can actually offer choice.
You may not need an A100
and you might actually use an A10G, for example,
or maybe even a T4 that's readily available.
So giving, offering the optimization as such
that use less computes,
coupled with the ability to move the actual work around it
and abstract that away from the end user point of view
gives you access to more silicon, fundamentally, right?
So, and then that directly leads
to more cost efficiencies, right?
So cost efficiency
will come from literally making the code faster
and run better, that's one.
And then the multiplicative factor there
is the ability to run this
on silicon that has better pricing, right?
And more availability.
And then all that said, of course,
we're supporting and working with Amazon
and supporting Inferentia,
and then we're going to expand that offering
to other providers as well, right?
So abstracting away the choice of hardware
such that users can focus on building their application,
which is what really matters,
is a significant part of our mission.
>> And the use cases, you guys,
what do you see coming out the gate?
Customization, is that the purpose?
Finding their differentia-
>> Yeah, I mean, one clear use case
that we see extreme need for,
and it is very much featured in our launch,
is text-to-image.
You know, it's just interesting how,
in fact, text-to-image is a great example
of an open source model like Stable Diffusion
leapfrogging what was around there that was closed before,
then showing that it can offer amazing results.
So text-to-image is a very significant workload
in the world right now, we support that.
Users can come, build with Stable Diffusion 2.1.
We also guarantee you have
the latest and greatest optimizations,
the ability to fine-tune that model,
and the ability to change the model weights
and replace them such that it can customize
for a broad set of users and quickly switch between them.
So that's one clear use case is text-to-image.
Another use case that we're already supporting today,
ready to go from pre-optimized models,
is large language models for, say, text-to-text, right?
To do summarization, to do text expansion,
text classification, and so on.
The third one is audio transcription,
turning audio into text.
We support that as well.
But we also support customers bringing their own models
and building their own custom containers, right?
So go and run it on the service.
So we have an altering service for that.
Anyways, yeah, so you can, those are some of the use cases.
I also want to point out
that we've been releasing steadily
some exciting demos built on our Compute Service.
One of them is the InkyMM multi model.
It's the first open source
commercializable multi-modal model that you can upload.
You can offer images and ask questions about it.
It's called InkyMM
and we show how that was built.
There's another demo, it's cool,
you can make your photos look like it's being cartoonized,
and then we show what the code looks like there.
So these are some examples of use cases that we see
that users are really excited about,
and then just like an example
of what people can build with very, very quickly, so.
Jon, please complement that, yeah, so.
>> Jon, weigh in here.
Because you're watching open source,
there's innovation happening,
what are you seeing in the landscape
that supports this need to get tinkering,
get building, get experimenting, mixing those cocktails?
I mean, you're seeing performance enhancements,
all kinds of, what are you seeing?
>> Yeah, so it's about pushing the envelope
on one of a few dimensions.
Customization to push the envelope
on either new features, new types of data,
lower cost, higher performance or privacy,
and sensitivity of data.
And when you speak to developers or CEOs or CIOs,
what you're going to hear is some combination
of those items become very, very important.
And this is not to diminish the amazing work
that OpenAI and Cohere and Anthropic and others are doing,
but there are cases where a CEO
wants to take her own fate in her own hands,
and open source makes that possible.
>> That's awesome, and I think, too,
the support that's going to come out of open source,
you know, everyone said that we'd never see another Red Hat.
Maybe there could be because you got to get in there.
Once there's momentum, you're going to want support.
Enterprise is going to want support.
Another big part of that, Luis, isn't that true?
>> Right, absolutely, yeah.
We have world-class customer success team here
and they're ready to help
and some of our early access users
already enjoy that benefit
and now we're opening to the world,
like very easy to help folks to get started
and have a world-class team of ML engineers
and solutions engineers to help folks build with it, so.
>> Well, congratulations on the big news
of platform infrastructure managed service.
Awesome, get people standing up their models,
testing them, customizing 'em, tuning them up.
You are helping them do that.
In the last 30 seconds we have left, Luis,
give the pitch for the developers watching,
for the press that are watching this video,
what's the big deal?
Why OctoML? Why this platform?
Give the pitch.
>> Yeah, so releasing the Octo AI Compute Service,
a place where you can come and build your AI applications
with freedom of choosing your model,
the model that you want for the application that you want,
highly efficient, make that run fast at low cost
and give you control.
And finally, ease of use.
We make it very easy to broaden the set of creators
to go build with this,
with a set of world-class models
ready, optimized to be used,
and then very, very clear instructions
and quick start guides here.
So that means there'll be more people building with it,
more innovation, and a better life here, so.
>> Great service, helping the developers,
getting more innovation, getting that flywheel going,
faster, smaller, cheaper,
that's the way we like it in open source.
Luis, thanks for coming on. >> And easier.
>> Jon, congratulations on the great investor,
say hello to Matt for me,
and congratulations on the news, Luis.
Looking forward to following up.
>> Thank you, John.
Try it out, everyone.
Octo AI Compute Service, bye.
>> We'll be checking it out.
This is theCUBE, exclusive news from OctoML.
I'm John Furrier, host of theCUBE.
Thanks for watching.
(bright music)