Object storage is a storage architecture that manages data as objects, as opposed
to other storage architectures like file systems which manage data as a file
hierarchy and block storage which manages data as blocks within sectors
and tracks. Each object typically includes the data itself, a variable
amount of metadata, and a globally unique identifier. Object storage can be
implemented at multiple levels, including the device level, the system
level, and the interface level. In each case, object storage seeks to enable
capabilities not addressed by other storage architectures, like interfaces
that can be directly programmable by the application, a namespace that can span
multiple instances of physical hardware, and data management functions like data
replication and data distribution at object-level granularity.
Object storage systems allow relatively inexpensive, scalable and self-healing
retention of massive amounts of unstructured data. Object storage is
used for diverse purposes such as storing photos on Facebook, songs on
Spotify, or files in online collaboration services, such as Dropbox.
History = Origins =
Object storage was first proposed at Carnegie Mellon University's Parallel
Data Lab as a research project in 1996. Research by Garth Gibson, et al. on
Network Attached Secure Disks first promoted the concept of splitting less
common operations, like namespace manipulations, from common operations,
like reads and writes, to optimize the performance and scale of both. Another
key concept was abstracting the writes and reads of data to more flexible data
containers. Fine grained access control through object storage architecture was
further described by one of the NASD team, Howard Gobioff, who later was one
of the inventors of the Google File System. Other related work includes the
Coda filesystem project at Carnegie Mellon, which started in 1987, and
spawned the Lustre file system. There is also the OceanStore project at UC
Berkeley, which started in 1999. One of the earliest and best-known object
storage products, EMC's Centera, debuted in 2002. However, development of
Centera's technology started even earlier in 1995 at a company in Belgium
called FilePool. = Development =
Overall industry investment in object storage technology has been sustained
for over a decade. From 1999 to 2013, there has been at least $300 million of
venture financing related to object storage, including vendors like
Amplidata, Bycast, Caringo, Cleversafe, Nirvanix, and Scality. This doesn't
include millions of dollars of private engineering from systems vendors like
DataDirect Networks, EMC, HDS, HP, IBM, NetApp, Redhat GlusterFS and Keeper
Technology, cloud services vendors like Amazon, Microsoft and Google, or the
many man years of open source development at Lustre, OpenStack,
MogileFS, Ceph and Skylable SX. Architecture
= Abstraction of storage = One of the design principles of object
storage is to abstract some of the lower layers of storage away from the
administrators and applications. Thus, data is exposed and managed as objects
instead of files or blocks. Objects contain additional descriptive
properties which can be used for better indexing or management. Administrators
do not have to perform lower level storage functions like constructing and
managing logical volumes to utilize disk capacity or setting RAID levels to deal
with disk failure. Object storage also allows the
addressing and identification of individual objects by more than just
file name and file path. Object storage adds a unique identifier within a
bucket, or across the entire system, to support much larger namespaces and
eliminate name collisions. = Inclusion of rich custom metadata
within the object = Object storage explicitly separates file
metadata from data to support additional capabilities: As opposed to fixed
metadata in file systems, object storage provides for full function, custom,
object-level metadata in order to: Capture application-specific or
user-specific information for better indexing purposes
Support data management policies Centralize management of storage across
many individual nodes and clusters Optimize metadata storage and
caching/indexing independently from the data storage
Additionally, in some object-based file system implementations:
The file system clients only contact metadata servers once when the file is
opened and then get content directly via object storage servers
Data objects can be configured on a per-file basis to allow adaptive stripe
width, even across multiple object storage servers, supporting
optimizations in bandwidth and I/O Object-based storage devices as well as
some software implementations manage metadata and data at the storage device
level: Instead of providing a block-oriented
interface that reads and writes fixed sized blocks of data, data is organized
into flexible-sized data containers, called objects
Each object has both data and metadata; physically encapsulating both together
benefits recoverability. The command interface includes commands
to create and delete objects, write bytes and read bytes to and from
individual objects, and to set and get attributes on objects
Security mechanisms provide per-object and per-command access control
= Programmatic data management = Object storage provides programmatic
interfaces to allow applications to manipulate data. At the base level, this
includes CRUD functions for basic read, write and delete operations. Some object
storage implementations go further, supporting additional functionality like
object versioning, object replication, and movement of objects between
different tiers and types of storage. Most API implementations are ReST-based,
allowing the use of many standard HTTP calls.
Implementation = Object-based storage devices =
Object storage at the protocol and device layer was proposed 20 years ago
and approved for the SCSI command set nearly 10 years ago as "Object-based
Storage Device Commands", but has not been productized until the development
of the Seagate Kinetic Open Storage platform. The SCSI command set for
Object Storage Devices was developed by a working group of the Storage
Networking Industry Association for the T10 committee of the International
Committee for Information Technology Standards. T10 is responsible for all
SCSI standards. = Object-based file systems =
Some high-performance distributed file systems use an object-based
architecture, where file metadata is stored in metadata servers and file data
is stored in object storage servers. File system client software interacts
with the distinct servers, and abstracts them to present a full file system to
users and applications. Lustre is an example of this type of object storage.
= Archive storage = Some early incarnations of object
storage were used for archiving, as implementations were optimized for data
services like immutability, not performance. EMC Centera and Hitachi HCP
are two commonly cited object storage products for archiving. Another example
is Quantum Lattus Object Storage Platform.
= Cloud storage = The vast majority of cloud storage
available in the market leverages an object storage architecture. Two notable
examples are Amazon Web Services S3, which debuted in 2005, and Rackspace
Files. Other major cloud storage services include Microsoft Azure ,
Google Cloud Storage and Oracle Elastic Storage Service.
= "Captive" object storage = Some large internet companies developed
their own software when object storage products were not commercially available
or use cases were very specific. Facebook famously invented their own
object storage software, code-named Haystack, to address their particular
massive scale photo management needs efficiently.
= Hybrid storage = A few object storage systems, such as
Ceph, GlusterFS, and Scality RING support Unified File and Object storage,
allowing some clients to store objects on a storage system while simultaneously
other clients store files on the same storage system. While "hybrid storage"
is not a widely accepted term for this concept, interoperable interfaces to the
same set of data is becoming available in some object storage products.
= Object storage systems = More general purpose object storage
systems came to market around 2008. Lured by the incredible growth of
"captive" storage systems within web applications like Yahoo Mail and the
early success of cloud storage, object storage systems promised the scale and
capabilities of cloud storage, with the ability to deploy the system within an
enterprise, or at an aspiring cloud storage service provider. Notable
examples of object storage systems include EMC Atmos, OpenStack Swift,
Scality RING and Caringo Swarm. Market adoption
One of the first object storage products, Lustre, is used in 70% of the
Top 100 supercomputers and ~50% of the Top 500. As of June 16, 2013, this
includes 7 of the top 10, including the current fastest system on the list -
China's Tianhe-2 and the second fastest, the Titan supercomputer at Oak Ridge
National Laboratory. Object storage systems had good adoption
in the early 2000s as an archive platform, particularly in the wake of
compliance laws like Sarbanes-Oxley. After five years in the market, EMC's
Centera product claimed over 3,500 customers and 150 petabytes shipped by
2007. Hitachi's HCP product also claims many petabyte-scale customers. Newer
object storage systems have also gotten some traction, particularly around very
large custom applications like eBay's auction site, where EMC Atmos is used to
manage over 500 million objects a day. As of March 3, 2014, EMC claims to have
sold over 1.5 exabytes of Atmos storage. On July 1, 2014, Los Alamos National Lab
chose the Scality RING as the basis for a 500 petabyte storage environment,
which would be among the largest ever. "Captive" object storage systems like
Facebook's Haystack have scaled impressively. In April 2009, Haystack
was managing 60 billion photos and 1.5 petabytes of storage, adding 220 million
photos and 25 terabytes a week. Facebook more recently stated that they were
adding 350 million photos a day and were storing 240 billion photos. This could
equal as much as 357 petabytes. Cloud storage has become pervasive as
many new web and mobile applications choose it as a common way to store
binary data. As the storage backend to many popular applications like Smugmug
and Dropbox, AWS S3 has grown to massive scale, citing over 2 trillion objects
stored in April 2013. Two months later, Microsoft claimed that they stored even
more objects in Azure at 8.5 trillion. By April 2014, Azure claimed over 20
trillion objects stored. Windows Azure Storage manages Blobs, Tables, and
Queues and counts them all as objects. Market analysis
IDC has begun to assess the object-based storage market annually using its
MarketScape methodology. IDC describes the MarketScape as: "...a quantitative
and qualitative assessment of the characteristics that assess a vendor's
current and future success in the said market or market segment and provide a
measure of their ascendancy to become a Leader or maintain a leadership. IDC
MarketScape assessments are particularly helpful in emerging markets that are
often fragmented, have several players, and lack clear leaders."
In 2013, IDC rated Cleversafe, Scality, DataDirect Networks, Amplidata, and EMC
as leaders. In 2014, it rated Scality, Cleversafe, DataDirect Networks, Hitachi
Data Systems, Amplidata, EMC, and Cloudian as leaders.
Standards = Object-based storage device standards
OSD Version 1 In the first version of the OSD
standard, objects are specified with a 64-bit partition ID and a 64-bit object
ID. Partitions are created and deleted within an OSD, and objects are created
and deleted within partitions. There are no fixed sizes associated with
partitions or objects; they are allowed to grow subject to physical size
limitations of the device or logical quota constraints on a partition.
An extensible set of attributes describe objects. Some attributes are implemented
directly by the OSD, such as the number of bytes in an object and the modify
time of an object. There is a special policy tag attribute that is part of the
security mechanism. Other attributes are uninterpreted by the OSD. These are set
on objects by the higher-level storage systems that use the OSD for persistent
storage. For example, attributes might be used to classify objects, or to
capture relationships among different objects stored on different OSDs.
A list command returns a list of identifiers for objects within a
partition, optionally filtered by matches against their attribute values.
A list command can also return selected attributes of the listed objects.
Read and write commands can be combined, or piggy-backed, with commands to get
and set attributes. This ability reduces the number of times a high-level storage
system has to cross the interface to the OSD, which can improve overall
efficiency. OSD Version 2
A second generation of the SCSI command set, "Object-Based Storage Devices - 2"
added support for snapshots, collections of objects, and improved error handling.
A snapshot is a point in time copy of all the objects in a partition into a
new partition. The OSD can implement a space-efficient copy using copy-on-write
techniques so that the two partitions share objects that are unchanged between
the snapshots, or the OSD might physically copy the data to the new
partition. The standard defines clones, which are writeable, and snapshots,
which are read-only. A collection is a special kind of object
that contains the identifiers of other objects. There are operations to add and
delete from collections, and there are operations to get or set attributes for
all the objects in a collection. Collections are also used for error
reporting. If an object becomes damaged by the occurrence of a media defect or
by a software error within the OSD implementation, its identifier is put
into a special error collection. The higher-level storage system that uses
the OSD can query this collection and take corrective action as necessary.
Differences between Key-Value and Object Stores
Let’s first clarify what a key/value store and an object store are. Using the
traditional block storage interface, one has a series of fixed size blocks which
are numbered starting at 0. Data must be that exact fixed size and can be stored
in a particular block which is identified by its logical block number.
Later, one can retrieve that block of data by specifying its unique LBN.
With a key/value store, data is identified by a key rather than a LBN. A
key might be "cat" or "olive" or "42". It can be an arbitrary sequence of bytes
of arbitrary length. Data does not need to be a fixed size and also can be an
arbitrary sequence of bytes of arbitrary length. One stores data by presenting
the key and data to the data store and can later retrieve the data by
presenting the key. You’ve seen this concept before in programming languages.
Python calls them dictionaries, Perl calls them hashes, Java and C++ call
them maps, etc. Several data stores also implement key/value stores such as
Memcached, Redis and CouchDB. Object stores are similar to key/value
stores except that the key must be a positive integer like a LBN. However,
unlike a LBN, the key can be any positive integer; it does not have to
map to an existing logical block number. In practice, it is usually limited to 64
bits. More like a key/value store than the traditional block storage interface,
data is not limited to a fixed size block but may be an arbitrary size.
Object stores also allow one to associate a limited set of attributes
with each piece of data. The key, value and set of attributes is referred to as
an object. To add more confusion, sometimes key/value stores are loosely
referred to as object stores but technically there is a difference.
See also Cloud storage
Clustered file system Object access method
References External links
AWS S3 API Documentation Google Cloud Storage API Documentation
Openstack Swift API Documentation Seagate Kinetic Open Storage
Documentation Windows Azure Storage API Documentation