A memory-mapped file is a segment of virtual memory which has been assigned a
direct byte-for-byte correlation with some portion of a file or file-like
resource. This resource is typically a file that is physically present on-disk,
but can also be a device, shared memory object, or other resource that the
operating system can reference through a file descriptor. Once present, this
correlation between the file and the memory space permits applications to
treat the mapped portion as if it were primary memory.
Benefits The primary benefit of memory mapping a
file is increasing I/O performance, especially when used on large files. For
small files, memory-mapped files can result in a waste of slack space as
memory maps are always aligned to the page size, which is mostly 4 KiB.
Therefore a 5 KiB file will allocate 8 KiB and thus 3 KiB are wasted. Accessing
memory mapped files is faster than using direct read and write operations for two
reasons. Firstly, a system call is orders of magnitude slower than a simple
change to a program's local memory. Secondly, in most operating systems the
memory region mapped actually is the kernel's page cache, meaning that no
copies need to be created in user space. Certain application-level memory-mapped
file operations also perform better than their physical file counterparts.
Applications can access and update data in the file directly and in-place, as
opposed to seeking from the start of the file or rewriting the entire edited
contents to a temporary location. Since the memory-mapped file is handled
internally in pages, linear file access requires disk access only when a new
page boundary is crossed, and can write larger sections of the file to disk in a
single operation. A possible benefit of memory-mapped
files is a "lazy loading", thus using small amounts of RAM even for a very
large file. Trying to load the entire contents of a file that is significantly
larger than the amount of memory available can cause severe thrashing as
the operating system reads from disk into memory and simultaneously writes
pages from memory back to disk. Memory-mapping may not only bypass the
page file completely, but the system only needs to load the smaller
page-sized sections as data is being edited, similarly to demand paging
scheme used for programs. The memory mapping process is handled by
the virtual memory manager, which is the same subsystem responsible for dealing
with the page file. Memory mapped files are loaded into memory one entire page
at a time. The page size is selected by the operating system for maximum
performance. Since page file management is one of the most critical elements of
a virtual memory system, loading page sized sections of a file into physical
memory is typically a very highly optimized system function.
Drawbacks The major reason to choose memory mapped
file I/O is performance. Nevertheless, there can be tradeoffs. The standard I/O
approach is costly due to system call overhead and memory copying. The memory
mapped approach has its cost in minor page faults - when a block of data is
loaded in page cache, but is not yet mapped into the process's virtual memory
space. In some circumstances, memory mapped file I/O can be substantially
slower than standard file I/O. Another drawback of memory mapped files
relates to a given architecture's address space: a file larger than the
addressable space can have only portions mapped at a time, complicating reading
it. For example, a 32-bit architecture such as Intel's IA-32 can only directly
address 4 GiB or smaller portions of files. An even smaller amount of
addressable space is available to individual programs -- typically in the
range of 2 to 3 GiB, depending on the operating system kernel.
Common uses Perhaps the most common use for a
memory-mapped file is the process loader in most modern operating systems When a
process is started, the operating system uses a memory mapped file to bring the
executable file, along with any loadable modules, into memory for execution. Most
memory-mapping systems use a technique called demand paging, where the file is
loaded into physical memory in subsets, and only when that page is actually
referenced. In the specific case of executable files, this permits the OS to
selectively load only those portions of a process image that actually need to
execute. Another common use for memory-mapped
files is to share memory between multiple processes. In modern protected
mode operating systems, processes are generally not permitted to access memory
space that is allocated for use by another process. There are a number of
techniques available to safely share memory, and memory-mapped file I/O is
one of the most popular. Two or more applications can simultaneously map a
single physical file into memory and access this memory. For example, the
Microsoft Windows operating system provides a mechanism for applications to
memory-map a shared segment of the system's page file itself and share data
via this section. Platform support
Most modern operating systems or runtime environments support some form of
memory-mapped file access. The function mmap(), which creates a mapping of a
file given a file descriptor, starting location in the file, and a length, is
part of the POSIX specification, so the wide variety of POSIX-compliant systems,
such as UNIX, Linux, Mac OS X or OpenVMS, support a common mechanism for
memory mapping files. The Microsoft Windows operating systems also support a
group of API functions for this purpose, such as CreateFileMapping().
Some free portable implementations of memory-mapped files for Microsoft
Windows and POSIX-compliant platforms are:
Boost C++ Libraries Fmstream
Cpp-mmf The Java programming language provides
classes and methods to access memory mapped files, such as FileChannel.
The D programming language supports memory mapped files in its standard
library. Ruby has a gem called Mmap, which
implements memory-mapped file objects. Since version 1.6, Python has included a
mmap module in its Standard Library. Details of the module vary according to
whether the host platform is Windows or Unix-like.
For Perl there are a several modules available for memory mapping files on
the CPAN, such as Sys::Mmap and File::Map.
In the Microsoft .NET runtime, P/Invoke can be used to use memory mapped files
directly through the Windows API. Managed access to memory mapped files
was introduced in version 4 of the runtime. For previous versions, there
are third-party libraries which provide managed API's.
PHP supported memory-mapping techniques in a number of native file access
functions such as file_get_contents() but has removed this in 5.3.
For the R programming language there exists a library on CRAN called
bigmemory which uses the Boost library and provides memory-mapped backed arrays
directly in R. The J programming language has supported
memory mapped files since at least 2005. It includes support for boxed array
data, and single datatype files. Support can be loaded from 'data/jmf' J's Jdb
and JD database engines use memory mapped files for column stores.
References hed.comaUsing-mmap-for-Advanced-File-IO/
^ , "What Do Memory-Mapped Files Have to Offer?".
freebsd.orgfreebsd-questions050371.html, read vs. mmap by Matthew Dillon
^ "Demand Paging" ^ Memory Mapped Files
^ Apple - Mac OS X Leopard - Technology - UNIX
^ CreateFileMapping Function ^ "Memory-Mapped Files". Boost.org. 2
February 2008. Retrieved 4 December 2011.
^ "Memory Mapped Files for Windows and POSIX systems". SourceForge.
^ "cpp-mmf". GitHub. ^ "std.mmfile - D Programming Language".
Digital Mars. Retrieved 4 December 2011. ^ "New Modules in 1.6". Archived from
the original on 30 December 2006. Retrieved 23 December 2008.
^ "Sys::Mmap Perl Module". ^ "File::Map Perl Module".
^ DotNet External links