In an unmarked office building in Austin, Texas.
Come on in. There are two small rooms where a handful of employees peer into microscopes, soldering irons or
tiny tweezers in hand.
They're designing two types of microchips made to power data centers and more recently, the AI boom.
When we get this, what do we do?
We test it.
First thing that we do, we test it.
But these chips aren't coming from Nvidia, AMD or any of the chip companies that have been hitting headlines
and market milestones since ChatGPT burst on the scene late last year.
I'm here inside Amazon's Austin, Texas chip lab, where it makes its own custom microchips to compete with
those from Intel, Nvidia and other giants.
And it's actually a way for them to save money and boost performance because it's one of the biggest
buyers of data center chips in the world.
CEO Adam Selipsky told CNBC that the chips that we saw here today are powering large language models
and more for the AI boom.
The entire world would like more chips for doing generative AI and whether
that's GPUs or whether that's Amazon's own chips that we're designing, I think that we're in
a better position than anybody else on Earth to supply the capacity that our customers collectively are going
to want.
Amazon Web Services is the world's biggest cloud computing provider and the most profitable arm of the
retail giant with an operating income of $5.4 billion in Q2.
Although that number has been down year over year for three quarters in a row, AWS still accounted for
70% of Amazon's overall 7.7 billion Q2 operating profit, giving it the cash it
needs for the huge undertaking that is custom silicon, and a growing portfolio of developer tools that could
eventually propel Amazon to the center of all the buzz.
Many of our AWS customers have terabytes or petabytes or exabytes of data already stored on
AWS, and they know that that data is going to help them customize the
models that they're using to power their generative AI applications.
And yet others have acted faster and invested more to capture business from the generative AI boom.
Think Microsoft's reported $13 billion investment in ChatGPT maker OpenAI and
Google's chatbot Bard, followed by its $300 million investment in OpenAI rival Anthropic.
AWS's profit margins have historically been far higher than those at Google Cloud, but those margins have been
narrowing. And although AWS's growth is still impressive, that's happening at a slower pace too.
Amazon is not used to chasing markets.
Amazon is used to creating markets and I think for the first time in a long time they are finding themselves
on the back foot and they are they are working to play catch up.
CNBC sat down with top AWS executives and analysts to ask about custom chips and how it
plans to make strides in generative AI to catch Google and Microsoft and perhaps give a needed boost to AWS,
too.
We end up with a package part like this.
And this is an actual machine learning accelerator that was designed and you can see
Annapurna Labs on it.
In 2015, Amazon bought Israeli startup Annapurna Labs to accelerate its dive into the chip business.
In July, we went to AWS's Annapurna location in Austin for an exclusive look at the chip design process
with lab director Rami Sinno.
AWS also designs chips in Silicon Valley, Canada and at a larger lab in Israel, then sends them off to be
made by chip manufacturers like TSMC in Taiwan.
AWS quietly started production of custom Silicon back in 2013 with a piece of specialized hardware called
Nitro, now the highest volume chip with more than 20 million in use in every AWS
server. Then at AWS's big annual customer conference re:Invent in 2018, Amazon launched
its ARM-based server chip Graviton to rival x86 CPUs from giants like AMD and Intel.
That's probably high single digit to maybe 10% of total server sales are ARM, and a good chunk
of those are going to be Amazon.
So on the CPU side, they've done quite well.
We're into our third generation of our graviton chip that provides acceleration in terms of
speed and cost efficiency and power for very general web-based workloads.
After announcing Graviton in 2018, announced its first AI focused chips, VP of
Product Matt Wood showed us the two AI chips it has today.
This big one here is called Trainium, and this small one here.
It's called Inferentia.
Inferentia. Amazon's first AI chip launched in 2019.
Which we're on our second generation of which allows customers to deliver very, very low cost, high
throughput, low latency machine learning inference, which is all the predictions of the when you type in
a prompt into your generative AI model, that's where all that gets processed to give you the response.
With Inferentia, you can get about four times more throughput and ten times lower
latency by using Inferentia than anything else available on AWS today.
Trainium came on the market in 2021.
All right. So this is a packaged part.
And then let me show you the other side.
What you see here is all the interfaces.
Machine learning breaks down into these two different stages.
So you train the machine learning models and then you run inference against those trained models.
And so we see a lot of customers that are interested in training their own machine learning models and their
own generative AI models.
And so that's where Trainium really, really helps.
Trainium provides about 50% improvement in terms of price
performance relative to any other way of training machine learning models on AWS.
But for now, Nvidia's GPUs are still king when it comes to training.
AWS itself just launched new AI acceleration hardware powered by Nvidia
H100s.
Accelerating performance by up to 6x and reducing training costs by up to
40% as compared to EC2 P4 instances.
Nvidia chips have a massive software ecosystem that's been built up around them over the last 15 years that
nobody else has. The big winner from AI right now is Nvidia.
That seems clear.
Still, Amazon is not the only non chip giant getting into custom silicon.
Apple has its M series of chips and a couple years before Amazon had AI chips, Google launched its own
cloud tensor processing units or TPUs.
Nobody's at the same scale as Google.
Google has been deploying this for like eight years.
My assumption is all of the hyperscalers, whether they've announced it or not, are all working on their
own accelerators and many are also working on their own CPUs as well.
But when it comes to custom chips, Microsoft is lagging behind Amazon and Google.
Microsoft has yet to announce the Athena AI chip it's been working on, reportedly in partnership with AMD.
I think the true differentiation is the technical capabilities that they're bringing to bear because
guess what? Microsoft does not have Trainium or Inferentia.
Generative AI is the current craze, but Amazon was building out a broader AI infrastructure for machine
learning with dozens of services, long before it made chips or used them to train LLMs.
Late 1990s, we were the first one to actually leverage machine learning-based technologies
to reinvent recommendation engines, and we leveraged machine learning to do things like
better product search and then automating leveraging robotics and computer vision in our Amazon
FCs or fulfillment centers to help products ship faster to actually
reinventing completely new customer experiences with things like Amazon Alexa.
But when OpenAI launched ChatGPT in November 2022, Microsoft was suddenly dominating the
AI headlines, followed by Google's Bard in February.
Two months later, Amazon announced its own large language model called Titan and Bedrock, a cloud
service to help developers enhance software using generative AI.
I think ChatGPT and Microsoft rollout of their initiatives were so fast, so aggressive, so quick, it
caught a lot of the market participants flat footed.
Amazon is trying to educate the market in order to close the gap, but frankly speaking, it's going to take
a couple of months.
Let's rewind the clock even before ChatGPT.
It's not like after that happened.
Suddenly we hurried and came up with a plan because you can't engineer a chip in that
quick of time. If anything, it actually accelerated some of the customer conversation and their keenness to
actually move forward with generative AI deployments.
Meta also recently announced its own LLM, Llama 2.
The open source ChatGPT rival is available on Microsoft's Azure Cloud platform.
Now a leaked internal email shows Amazon CEO Andy Jassy is directly overseeing a new central
team that's building out expansive large language models, but so far, AWS has focused on tools
instead of building a ChatGPT competitor.
So if you look at the Bedrock strategy that they are trying to focus on, they are betting the farm on the
fact that enterprises might not necessarily be building out their own GPT models.
Bedrock gives AWS customers access to LLMs made by Anthropic, Stability AI,
AI21 and Amazon's own Titan.
Titan is actually a family of foundational models.
We have text-based models which are great for generative text, so creating marketing copy and
advertising, chat bots, those sorts of things.
And then we have an embedding model which is great for personalization and ranking those sorts of use cases.
Amazon says its AI products are being used by numerous customers like Philips, 3M, Old Mutual and
HSBC. In the Q2 earnings call, it said a very significant amount of AWS business is now driven by
AI and the 20+ machine learning services it offers.
We don't believe that one model is going to rule the world.
We want our customers to have the state of the art models from multiple providers because
they are going to pick the right tool for the right job.
One of Amazon's new AI offerings is AWS HealthScribe, a service unveiled in July to help doctors
automatically draft patient visit summaries and more.
Another big tool in the AWS AI stack is CodeWhisperer.
CodeWhisperer generates code recommendations from natural language prompts based on contextual
information. Participants who use CodeWhisperer were 27% more
likely to complete their tasks successfully, and they did it 57% faster on
average.
Last year, Microsoft also reported productivity boosts from its coding companion GitHub Copilot.
And then there's SageMaker, Amazon's machine learning hub that offers algorithms, models and more.
Autodesk, for instance.
They were able to leverage generative foundational models to design a bulkhead of an aircraft.
In one example, it was 45% lighter for a particular carrier.
In June, AWS also announced a $100 million generative AI innovation center.
We have so many customers who are saying, I want to do generative AI, but they don't necessarily know what
that means for them in the context of their own businesses.
And so we're going to bring in solutions architects and engineers and strategists
and data scientists to work with them one on one.
When companies are choosing between Amazon, Google and Microsoft for their generative AI needs, some may
choose Bedrock because they're already familiar with AWS, where they run other applications and store a ton
of data.
If you took the data that we have in Amazon S3 that's stored on devices like this, and you stack
them up one on top of another, it would take you all the way to the
International Space Station and almost all the way back.
And that is a lot of data.
At the end of the day, Amazon does not need to win headlines.
Amazon already has a really strong cloud install base.
All they need to do is to figure out how to enable their existing customers to expand into value
creation motions using generative AI.
So how many AWS customers are actually using it for machine learning?
We have over 100,000 customers today that are using machine learning on AWS, many of which
have standardized on our machine learning service, which is called SageMaker, to build, train and deploy
their own custom models.
But in reality, that's not a big percentage of AWS's millions of customers.
Although most aren't tapping into it for AI yet, that could change.
What we are not seeing is enterprises saying, 'Oh wait a minute, Microsoft is so ahead in generative AI,
let's just go out and let's switch our infrastructure strategies, migrate everything to Microsoft.' That is
not happening because at the end of the day, even if you're trying to create a chatbot, if you're already an
Amazon customer, chances are you're likely going to explore Amazon ecosystems quite extensively.
How quickly can these companies move to develop these generative AI applications
is driven by starting first on the data they have in AWS and
using compute and machine learning tools that we provide.
Imagine you're cooking dinner and you're using a new recipe.
It is a lot faster to start with ingredients that you already know that have been cut and
prepared to go ahead and put together the recipe than if you have to research all the ingredients, get
familiar with them and then learn how you're going to put them together and cook with them.
Right? That's what AWS customers are doing.
They have all the different ingredients that they are familiar with and they know how to use.
And whether that's storage or compute or it's a machine learning tools like Amazon
SageMaker and Amazon Bedrock, and they're putting it together that much faster.
And as generative AI continues to accelerate, all the big players are scrambling to establish how to use
these tools responsibly and securely.
I can't tell you how many Fortune 500 companies I've talked to who have banned ChatGPT.
So with our approach to generative AI and our Bedrock service,
anything you do, any model you use through Bedrock will be in your own isolated
virtual private cloud environment.
It'll be encrypted, it'll have the same AWS access controls.
Selipsky joined six other AI players at the White House in July to sign pledges to ensure that AI tools are
secure.
There are open problems that still need to be solved, especially when you're trying to deal with highly
regulated industries in financial services, health care and beyond.
We still do not have any well thought out regulatory guardrails around data
protection, private information protection and responsible AI capabilities in this space.
There's also national security concerns.
The Biden administration has proposed new rules that would require U.S.
cloud providers like Amazon and Microsoft to seek government permission before providing China with cloud
computing services using AI chips.
But for now, there's no slowdown in sight for the development of new generative AI applications or the
chips needed to power them.
And that race is just beginning.
So let's say that we're three steps into a race and we start asking, 'Well, who's ahead?
Who's behind? How do the runners look?' But then you look up and you realize that it's a 10k race
and then you realize it's the wrong question to ask, who's where three steps into the race?
The real question is what's going to happen at the end of the 10k race?