-Recently another vulnerability has come to light.
L1TF, or L1 Terminal Fault, happens
for similar reasons to the Meltdown
vulnerability that affected Intel processors
earlier in 2018.
In Meltdown, the processor is busily reordering
instructions, it tracks these in an internal
structure known as the reorder buffer.
When it performs a data access,
it will do the permission check
and tag an entry in the reorder buffer.
Then, when the instruction's complete or retire,
it will check at the last second
to see if data access was supposed to occur.
If it was supposed to occur, everything's great.
If it's not supposed to occur,
what will happen is the processor will undo
the speculatively executed instructions
from the reorder buffer.
Like Meltdown, the L1 Terminal Fault vulnerability
also occurs as a result of attempts to
speed up and parallelize this permission checking activity.
During execution of programs, processors need to access
many different parts of system memory.
Applications have a view of system memory
known as virtual memory in which the addresses
that they are using for individual memory locations
are actually translated using a CPU structure
known as the Memory Management Unit.
The Memory Management Unit translates
virtual addresses as used by applications
into the physical addresses located in system memory chips
using a series of software-managed structures
in memory known as page tables.
Page tables form a hierarchical structure
in which different parts of a virtual address
are translated into a physical address
by walking through the individual entries of the page table.
This is performed by special hardware
inside the Memory Management Unit
known as a page table walker.
During the page table walk,
the processor will perform the translation,
but it will also check other attributes
of the page table entries.
One of the attributes that it will check
is whether the page table entry is valid or present.
Non-present page table entries can exist
for many different reasons.
A classical example of this comes as a result
of an abstraction called swapping
in which applications have the illusion
of having much more memory than
is physically present in the machine.
Memory that hasn't been used recently
is swapped or moved out to disk
and the underlying page table entry
is marked as not present.
Then, when a later access occurs to that memory location,
a trap or a fault will occur
and a software fix-up routine will run
that will look at this page table entry,
read some bits from that entry to work out
where the memory is actually located on disk
and pull it back into the processor,
changing the page table entry to present
and then resuming the application.
The application is not aware
the swapping activity has happened
other than perhaps a small delay to its memory access
and everything continues as before.
So, having not present page table entries
isn't in of itself an unusual thing,
especially on a very busy and active system.
In L1TF, the processor will see a page table entry
that's marked as not present,
but because of an aggressive performance optimization,
it will take this knowledge, tag the reorder buffer
to say that at some point in the near future,
an exception has to be raised to the operating system
and meanwhile, it will continue to speculate forward.
During this small window of speculation,
it will take the address bits of this page table entry
and forward it to the internal processor data cache.
Then, if a suitable entry that matches the physical address
from that page table entry happens to already be loaded
into the data cache, it may be forwarded
speculatively to further operations within the program.
What this means is that a malicious attacker
can cause the creation of not-present page table entries,
pointing to physical addresses of interest.
These physical addresses could point to secrets,
cryptographic keys, and other sensitive information
within the machine to which the attacker
should not have access.
There are several different variants
of the L1 Terminal Fault attack.
The first concerns bare metal operating systems.
It's possible for a malicious application
to use standard Linux software interfaces
to cause the creation of not-present page table entries
and attack the host kernel on a bare metal machine.
This in of itself is of course quite bad,
but we can mitigate against the bare metal attack
by making some relatively straightforward changes
to help Linux generate some not-present page table entries.
We can make sure that when we generate
these not-present page table entries,
we do so in a way that they are never vulnerable
to the L1 Terminal Fault attack.
We do this by making sure that certain bits
in the not-present entries are always set.
This is through a very simple change to the operating system
that has no measurable performance impact.
The bad news is that a third variant
of the L1 Terminal Fault vulnerability
impacts virtualization.
Intel processors incorporate a technology
known as Extended Page Tables or EPT.
In EPT, a guest virtual machine is able to co-manage
its page tables in cooperation with the hypervisor.
The guest provides one level of page tables,
the hypervisor provides another
and every access to memory inside a guest virtual machine
is actually translated twice.
First, using the guest page tables
and then using the host page tables.
By having these two stages of translation,
it allows a virtual machine to have the illusion
that it has access to all of its own physical memory,
but its physical memory is actually virtual memory
within the host machine that is translated again.
A malicious attacker can create specially modified
guest virtual machine images in which the kernel
will generate Extended Page Table entries
that contain physical addresses of interest on the host.
Ordinarily what should happen is that memory accesses
through these guest Extended Page Tables
should be translated twice.
First, by the guest virtual machine page tables
and then by the host page tables,
but in L1TF, the processor will begin
to walk through the guest page tables.
Notice that a page table entry is marked as not present
and tag the offending data access to generate an exception.
During a small window of time
prior to that exception being raised,
the physical address inside that page table entry
will be speculatively forwarded
into the processor's data cache.
The address that's forwarded, however,
is an untranslated host address.
So now, we're only performing one level of translation,
just the guest translation.
These translations are fully under the ownership
of the malicious virtual machine.
Any physical address in that offending page table entry
will be forwarded without the translation at the host
and without any confinement by the hypervisor.
Therefore, if there is any sensitive data
at the physical address of interest in the host memory,
the malicious guest is able to speculatively retrieve that
using an attack similar to Meltdown.
The offending data access by the malicious virtual machine
will eventually generate a measurable fault
and we are working on means to detect and monitor this,
but unfortunately that speculation window
during which the physical address of interest is forwarded
means that the virtual machine has already read
the data that it wants prior to us
being able to trap and analyze this.
As with other vulnerabilities, we can't change
existing field deployed hardware.
All we can do is look for ways to mitigate
against the impact of these vulnerabilities.
We can't prevent L1 Terminal Fault attacks,
but we can render them less useful
by making sure that there are no secrets of interest
to a malicious virtual machine
as it tries to perform an L1TF attack.
To do this, we perform a series of modifications
to our hypervisor in kernel.
First, on entry into virtual machines,
we will flush the level one data cache of any secrets.
This means that any L1 Terminal Fault attack
won't be able to pull anything of interest out of the cache,
because it's been flushed.
This flushing activity of course has a performance impact
that is measurable, anything between a few percent
and up to 15% overhead.
But, it's not a devastating performance impact
and it will serve to keep customer data safe.
Secondly, we have to make some changes
to how we handle a technology
in Intel processors called Hyper-Threading.
Hyper-Threading is a technology that allows
more efficient use of CPU resources,
recognizing that today's workloads
are often heavily threaded in nature.
Hyper-Threading introduces an additional nuance
to the mitigation of L1 Terminal Fault.
Because hyperthreads very tightly share resources
with one another, it's possible that one hyperthread
may be running a trusted virtual machine
while another hyperthread is simultaneously
running a malicious virtual machine.
Hypervisor has been modified to flush data caches
whenever it begins to execute virtual machines.
Unfortunately, in the case that two peer hyperthreads
are running different trusted
and untrusted virtual machines,
it's possible for the malicious actor
on the untrusted virtual machine
to be trying to read from the data cache
while simultaneously the trusted virtual machine
is trying to flush that same data cache.
Erase condition exists and there's a small window of time
during which the malicious actor
may still be able to read data speculatively
from that data cache.
L1 Terminal Fault is a complex vulnerability
that will affect each environment differently.
In some cases, it may be necessary
to disable Intel Hyper-Threading.
In other cases, especially those running
only trusted host and guest operating systems,
this may not be necessary.
We recommend that you consult
Red Hat's Use Case-Driven documentation
and take all steps as appropriate
to your local environment and security policy.
To learn more about this and other vulnerabilities,
as well as recommended mitigations and remediations,
visit redhat.com and thank you.
Achieving a complete mitigation
against the L1 Terminal Fault vulnerability
may in many cases require
that the user disable Intel Hyper-Threading.
Our analysis of the problem
in collaboration with our industry peers
has conclusively shown that this is the only way
to affect a true remediation when running
a general purpose Linux operating system on the hypervisor.
In some cases, our industry cloud partners
have been able to leverage special purpose hardware
unique to their environment
in order to run safely with Intel Hyper-Threading enabled.
Alas, this is not more broadly applicable
to a general data center environment.
To learn more about this and other vulnerabilities,
as well as recommended mitigations and remediations,
visit redhat.com and thank you.