BigTable is a compressed, high performance, and proprietary data storage system built
on Google File System, Chubby Lock Service, SSTable and a few other Google technologies.
It is not distributed outside Google, although it underlies Google Datastore, which is available
as a part of Google Cloud Platform
History BigTable development began in 2004 and is
now used by a number of Google applications, such as web indexing, MapReduce, which is
often used for generating and modifying data stored in BigTable, Google Maps, Google Book
Search, "My Search History", Google Earth, Blogger.com, Google Code hosting, Orkut, YouTube,
and Gmail.
Google's reasons for developing its own database include scalability and better control of
performance characteristics.
Google's Spanner RDBMS is layered on an implementation of BigTable with a Paxos group for two-phase
commits to each table.
Google F1 was built using Spanner to replace an implementation based on MySQL.
Design BigTable maps two arbitrary string values
and timestamp into an associated arbitrary byte array.
It is not a relational database and can be better defined as a sparse, distributed multi-dimensional
sorted map.
BigTable is designed to scale into the petabyte range across "hundreds or thousands of machines,
and to make it easy to add more machines [to] the system and automatically start taking
advantage of those resources without any reconfiguration".
Each table has multiple dimensions.
Tables are optimized for Google File System by being split into multiple tablets – segments
of the table are split along a row chosen such that the tablet will be ~200 megabytes
in size.
When sizes threaten to grow beyond a specified limit, the tablets are compressed using the
algorithm BMDiff and the Zippy compression algorithm publicly known and open-sourced
as Snappy, which is a less space-optimal variation of LZ77 but more efficient in terms of computing
time.
The locations in the GFS of tablets are recorded as database entries in multiple special tablets,
which are called "META1" tablets.
META1 tablets are found by querying the single "META0" tablet, which typically resides on
a server of its own since it is often queried by clients as to the location of the "META1"
tablet which itself has the answer to the question of where the actual data is located.
Like GFS's master server, the META0 server is not generally a bottleneck since the processor
time and bandwidth necessary to discover and transmit META1 locations is minimal and clients
aggressively cache locations to minimize queries.
Other similar software Apache Accumulo — built on top of Hadoop,
ZooKeeper, and Thrift.
Has cell-level access labels and a server-side programming mechanism.
Written in Java.
Apache Cassandra — brings together Dynamo's fully distributed design and BigTable's data
model.
Written in Java.
Apache HBase — Provides BigTable-like support on the Hadoop Core.
Has cell-level access labels and a server-side programming mechanism too.
Written in Java.
Hypertable — Hypertable is designed to manage the storage and processing of information
on a large cluster of commodity servers.
Written in C++.
"KDI", Bluefish, GitHub — Kosmix attempt to make a BigTable clone.
Written in C++.
LevelDB — Google's embedded key/value store that uses similar design concepts as the BigTable
Tablet.
See also Amazon SimpleDB
Big data Distributed data stores, an overview
Dynamo Column-oriented DBMS
Hadoop References
Bibliography Chang, Fay; Dean, Jeffrey; Ghemawat, Sanjay;
Hsieh, Wilson C; Wallach, Deborah A; Burrows, Michael ‘Mike’; Chandra, Tushar; Fikes,
Andrew; Gruber, Robert E, "Bigtable: A Distributed Storage System for Structured Data", Research,
Google . External links
BigTable: A Distributed Structured Storage System, Washington . Video, Google .
UWTV . Witchcock, Andrew, Google's BigTable .
Carr, David F, "How Google Works", Baseline . "Is the Relational Database Doomed?", Read-write
web .