Non-uniform memory access is a computer memory design used in multiprocessing, where the
memory access time depends on the memory location relative to the processor.
Under NUMA, a processor can access its own local memory faster than non-local memory.
The benefits of NUMA are limited to particular workloads, notably on servers where the data
are often associated strongly with certain tasks or users.
NUMA architectures logically follow in scaling from symmetric multiprocessing architectures.
They were developed commercially during the 1990s by Burroughs, Convex Computer, Honeywell
Information Systems Italy, Silicon Graphics, Sequent Computer Systems, Data General, and
Digital.
Techniques developed by these companies later featured in a variety of Unix-like operating
systems, and to an extent in Windows NT.
The first commercial implementation of a NUMA-based Unix system was the Symmetrical Multi Processing
XPS-100 family of servers, designed by Dan Gielan of VAST Corporation for Honeywell Information
Systems Italy.
Basic concept
Modern CPUs operate considerably faster than the main memory they use.
In the early days of computing and data processing, the CPU generally ran slower than its own
memory.
The performance lines of processors and memory crossed in the 1960s with the advent of the
first supercomputers.
Since then, CPUs increasingly have found themselves "starved for data" and having to stall while
waiting for data to arrive from memory.
Many supercomputer designs of the 1980s and 1990s focused on providing high-speed memory
access as opposed to faster processors, allowing the computers to work on large data sets at
speeds other systems could not approach.
Limiting the number of memory accesses provided the key to extracting high performance from
a modern computer.
For commodity processors, this meant installing an ever-increasing amount of high-speed cache
memory and using increasingly sophisticated algorithms to avoid cache misses.
But the dramatic increase in size of the operating systems and of the applications run on them
has generally overwhelmed these cache-processing improvements.
Multi-processor systems without NUMA make the problem considerably worse.
Now a system can starve several processors at the same time, notably because only one
processor can access the computer's memory at a time.
NUMA attempts to address this problem by providing separate memory for each processor, avoiding
the performance hit when several processors attempt to address the same memory.
For problems involving spread data, NUMA can improve the performance over a single shared
memory by a factor of roughly the number of processors.
Another approach to addressing this problem, utilized mainly by non-NUMA systems, is the
multi-channel memory architecture; multiple memory channels are increasing the number
of simultaneous memory accesses.
Of course, not all data ends up confined to a single task, which means that more than
one processor may require the same data.
To handle these cases, NUMA systems include additional hardware or software to move data
between memory banks.
This operation slows the processors attached to those banks, so the overall speed increase
due to NUMA depends heavily on the nature of the running tasks.
Intel announced NUMA compatibility for its x86 and Itanium servers in late 2007 with
its Nehalem and Tukwila CPUs.
Both CPU families share a common chipset; the interconnection is called Intel Quick
Path Interconnect.
AMD implemented NUMA with its Opteron processor, using HyperTransport.
Freescale's NUMA for PowerPC is called CoreNet.
Cache coherent NUMA
Nearly all CPU architectures use a small amount of very fast non-shared memory known as cache
to exploit locality of reference in memory accesses.
With NUMA, maintaining cache coherence across shared memory has a significant overhead.
Although simpler to design and build, non-cache-coherent NUMA systems become prohibitively complex
to program in the standard von Neumann architecture programming model.
Typically, ccNUMA uses inter-processor communication between cache controllers to keep a consistent
memory image when more than one cache stores the same memory location.
For this reason, ccNUMA may perform poorly when multiple processors attempt to access
the same memory area in rapid succession.
Support for NUMA in operating systems attempts to reduce the frequency of this kind of access
by allocating processors and memory in NUMA-friendly ways and by avoiding scheduling and locking
algorithms that make NUMA-unfriendly accesses necessary.
Alternatively, cache coherency protocols such as the MESIF protocol attempt to reduce the
communication required to maintain cache coherency.
Scalable Coherent Interface is an IEEE standard defining a directory-based cache coherency
protocol to avoid scalability limitations found in earlier multiprocessor systems.
For example, SCI is used as the basis for the NumaConnect technology.
As of 2011, ccNUMA systems are multiprocessor systems based on the AMD Opteron processor,
which can be implemented without external logic, and the Intel Itanium processor, which
requires the chipset to support NUMA.
Examples of ccNUMA-enabled chipsets are the SGI Shub, the Intel E8870, the HP sx2000,
and those found in NEC Itanium-based systems.
Earlier ccNUMA systems such as those from Silicon Graphics were based on MIPS processors
and the DEC Alpha 21364 processor.
NUMA vs. cluster computing One can view NUMA as a tightly coupled form
of cluster computing.
The addition of virtual memory paging to a cluster architecture can allow the implementation
of NUMA entirely in software.
However, the inter-node latency of software-based NUMA remains several orders of magnitude greater
than that of hardware-based NUMA.
Software support Since NUMA largely influences memory access
performance, certain software optimizations are needed to allow scheduling threads and
processes close to their data.
Microsoft Windows 7 and Windows Server 2008 R2 add support for NUMA architecture over
64 logical cores.
Java 7 added support for NUMA-aware memory allocator and garbage collector.
The Linux kernel 2.5 already had basic support built-in, which was further extended in subsequent
releases.
Linux kernel version 3.8 brought a new NUMA foundation which allowed more efficient NUMA
policies to be built in the next kernel releases.
Linux kernel version 3.13 brought numerous policies that attempt to put a process near
its memory, together with handling of cases such as shared pages between processes, or
transparent huge pages; new sysctl settings are allowing NUMA balancing to be enabled
or disabled, as well as various NUMA memory balancing parameters to be configured.
OpenSolaris models NUMA architecture with lgroups.
See also
References
This article is based on material taken from the Free On-line Dictionary of Computing prior
to 1 November 2008 and incorporated under the "relicensing" terms of the GFDL, version
1.3 or later.
External links