- Hi, I am Mark Russinovich,
chief technology officer
and technical fellow of Microsoft Azure,
and what I'm going to talk to you about
is a major step forward in confidential computing
that we've taken together with our partner NVIDIA.
We're realizing together a vision of confidential AI.
Now first, what is confidential computing?
Those of you familiar with computer security
know that we try to protect data
throughout its entire lifecycle,
and we do that through encryption.
When we store data on media like a disk,
we encrypt it, and this is called encryption at rest,
and that protects the data from anybody
that has physical access to the media
or logical access to that disk
without going through the application.
If they try to read the data,
it's encrypted and not in clear.
We do the same thing with data
when we send it between two different servers,
we encrypt it on the wire
and that means that somebody that is sniffing the network
just sees the encrypted data and not the clear text data.
What's been missing up until now
is protecting that data
in the third pillar of its lifecycle,
which is while it's in use.
When the CPU loads that encrypted data from disk
or gets that encrypted data off the network,
it must decrypt it to process it,
and at that point it's vulnerable, it's in the clear,
and that means that the hypervisor
might be able to have access to it.
The operating system that the code is running on
might have access to it.
Admins servicing the virtual machine or the server
might have access to it,
and physical personnel doing maintenance on the server
might be able to get access to it.
Confidential computing completes
that third pillar of protection
by creating a trusted execution environment, or TEE,
that surrounds the code and the data
that is processing that data.
When the TEE is created, it's loaded with code.
That code then can get an attestation report
from the hardware, from the CPU,
present it to somebody external,
a remote party through something called remote attestation.
That remote party can verify that attestation report,
make sure that the code is the code that it trusts
and that it's being protected in a TEE
by hardware that it trusts.
And at that point,
release a key that it encrypts to that TEE
that allows that code then
to decrypt the data that it might read off disk
or read from the network and perform processing on it.
Now it's in the clear inside that TEE,
but because it's inside the TEE,
it's protected from the hypervisor,
from the host operating system,
from the admin of the server,
and from physical access as well.
This major step forward we're taking with NVIDIA
is to introduce confidential GPUs.
Working deeply with them, we've co-designed
the Hopper H100 confidential computing GPU.
This GPU creates a trusted execution environment
such that the model of an AI model
will be protected inside that trusted execution environment
and so will the data that that model processes.
In combination with a confidential CPU,
that allows us to have confidential AI
where the data might be loaded into a CPU,
decrypted inside that CPU TEE,
sent over to the GPU
where it is also then decrypting that data
that the CPU encrypted as it hands it to it
and then performing computation on it.
That protects the data through its entire lifecycle
in an AI scenario.
And the AI scenarios that confidential computing protect
include protecting the model itself.
A lot of these models that are being created these days
have IP inherent in them.
They must be protected from unauthorized access and usage.
Confidential computing with Hopper GPUs
protects those models from outside access.
It also protects the data that those models have access to.
And there's a few different ways that models use data.
One of them is in the training of the model itself,
in that pre-training phase where the model
is processing data as it learns that data.
With confidential computing, that data can now be protected
through that pre-training phase,
and that data might have very sensitive information in it
that others shouldn't have access to.
Another type of data that can be protected
with confidential computing
is fine-tuning data, where you take a pre-trained model
and then continue some training on it
with custom data that is for a particular scenario.
This is common for enterprises, for example,
where they're bringing their own private data
and having a model learn the specifics of their own data.
That data can be very sensitive
and so it can be protected through that fine-tuning process
with confidential computing.
And then a third type of data that AI models process
are the prompts and the responses that they generate.
For example, when you send a query
into a large language model,
that's called the prompt,
and you want that protected
maybe because it has very sensitive information in it.
Similarly, as the model processes that data
and it generates a response,
that might also have sensitive data in it.
With confidential GPUs, you can protect that data
all the way from its origination into the processing
and the response on the way back.
A third scenario for confidential AI that's really exciting
is the confidential multi-party AI scenario.
That's where multiple parties each have their own data sets
that through regulatory reasons
or maybe data privacy reasons
might not be able to share with one another.
But combining them, they might get really deep insights
they can't on their own.
Confidential multi-party AI
allows them to combine their data by trusting only the TEE
and the code that has access to that data,
and then they can get the insights back from that data.
And maybe not all of them are allowed to get the insights,
maybe only one of them is.
The scenarios in verticals are endless.
For example, in healthcare
where you've got different parties
that have their own health data sets they want to combine
to get deep insights from it is just one scenario.
In the financial sector.
Having auditors looking for fraud
that might combine data sets they have access to
with the transaction data sets
that a financial entity might have is another example.
And the list goes on and on.
Now, what I'm going to do to demonstrate
in concrete terms what confidential AI looks like
is with a scenario
that we call retrieval-augmented generation.
I mentioned fine-tuning a little bit earlier,
where you take a pre-trained model
and then you have it learn about a custom dataset,
an enterprise dataset.
But a more common scenario
for having a model know about custom data
is through retrieval-augmented generation.
In retrieval-augmented generation, or RAG,
you take your dataset, you chunk it up,
and you create vectors that are embeddings
of those particular chunks,
and you store those embeddings in a database.
What that allows you to do is to later take a query,
create an embedding of that query,
and then do a nearest search in that database
to find relevant snippets of code or text or documents,
pull that out,
and then hand that relevant information
as part of the prompt to the large language model.
And that allows it to go reason
over that relevant information
and provide a response that is very tailored
towards that custom data
with the insights that the large language model provides.
In the demo setup I've got, I've got a web browser
that is going to connect to a service.
That service is running in a confidential virtual machine
that is connected to a confidential H100 GPU.
That confidential virtual machine
has a vector database as part of it,
and that's going to allow us
to do retrieval-augmented generation
on a document that we upload.
Through the entire process, I'm going to show you
my data's going to be protected end to end.
So let's go take a look at that demo.
Now here you can see I've got the website.
It's the first time that I've visited that website,
and so my extension that is enlightened
to understand attestations from the CPU and GPU
realizes that we don't have a policy set for that site.
Now, in a real scenario, the enterprise administrators
would've configured my extension
to know that site with the policy.
But here, what I'm going to do
is add that site to my trusted websites
by authorizing a policy that matches the attestations.
And those attestations include the CPU versions,
the TEE versions, the firmware,
the software that's running
in the confidential virtual machine,
as well as the same for the H100 GPU,
including its driver, firmware information.
Now that I've got a trusted policy set up,
I'm going to go and send in a prompt.
This prompt is going to be encrypted on my end,
sent in encrypted form
into that confidential virtual machine,
given to the confidential GPU.
That confidential GPU is going to perform inference on it
and provide a response.
In this case, I've asked it what confidential computing is.
In this particular model
that I've got running inside the GPU,
it's not familiar with that term.
This is where the power of retrieval-augmented
or RAG is going to come in.
I'm going to upload a PDF
that includes information about confidential computing.
That document's going to be chunked,
vectorized with embeddings,
and then that database is going to be augmented.
And now when I enter the prompt again,
the large language model is going to be given
the relevant snippets of text from that document
and it's able to provide an accurate answer to my question.
The process of uploading that document
was protected end to end with confidential computing.
The vectorization was protected, that database is protected.
That data that it might be very sensitive
is not accessible to anything
but the confidential virtual machine and the GPU.
Now I'm going to do one last thing
and that is to go to my policy
and change the driver version for the GPU.
That's going to mean that when I go visit that website
and it gets an attestation,
the browser extension is going to see
that it's not matching the policy that I've set,
and it's going to show me with this warning in red here
that I should not trust the website
given the policy I've got configured.
That's going to prevent me
from inadvertently submitting prompts
or uploading documents for RAG processing
that might have sensitive data,
but then at that point might be available to others
that are unauthorized
because it's not being protected in the way that I want
with confidential computing.
So I've shared with you a major milestone
in the journey that Microsoft Azure is on
towards ubiquitous confidential computing,
filling in a missing piece with the introduction
of confidential GPUs in deep partnership with NVIDIA.
I recommend you check out our confidential GPU offering
as well as the other confidential computing offerings
on Microsoft Azure.
Thank you.
(soft music)