MALE SPEAKER: This is the first of our
two datastore sessions.
You've already learned about the App Engine, so we're now
discussing this important peripheral service both in
this introductory session, followed by an advanced one.
For this introductory course session, it's very basic, like
what is a datastore as well as some internals.
We'll talk about how the underlying BigTable provides
the scalability and reliability
issue of the data store.
And finally, we'll go beyond theory by covering more
practical operations about how to use the datastore.
There are many definitions of the datastore but very simply
and easy to understand, it's a persistent storage for App
Engine data.
Why does App Engine need to store the data?
You've already learned that App Engine is the framework to
run the web application.
Its nature is very scalable.
Basically, the scalability means that if your application
is a surge on the load, it automatically increases the
number of instances.
So we need some central place to save the data, otherwise
this instance and that instance would return
different results for the same request.
This is not an App Engine specific problem.
If you've worked on other traditional web applications
such as CGI to the more recent PHP, relational databases and
datastores for App Engine is a counterpart of the relational
database or RDB.
So the question is, why don't we use an RTB?
Why does Google provide datastore instead of using
some relational database?
The answer is really easy--
scalability.
So if the App Engine is highly scalable, this datastore must
also match its scalability.
Let's look into database scalability to explain why
Google uses datastore as database for App Engine.
So let's first compare the datastore
with relational databases.
If I look clearly, an RDB supports the following
functionality.
It includes a table JOIN, flexible filtering, and
subqueries.
The datastore supports some of these, but its functionality
is limited to the simple filtering and sorting.
On the other hand, datastore is highly scalable and
reliable without sacrificing performance.
RDBs, however, are really difficult to scale.
First, let's look at a server configuration of databases.
The server can be a single instance.
This is a simple configuration, but here the
performance of the system is limited by hardware resources
of the machine.
And it's going to be the single point of failure.
If we have replication or copies, it increases the
reliability.
But here arises the issue of consistency among all server
instances, We can also shard.
Now, shard means split the server or data into multiple
machines, so that the multiple machines altogether work as
one machine.
If we shard the database, we often must look for
transactions which will become more complicated.
Let's talk about consistency, which I'm going to use a lot.
Consistency can have two levels--
strong or eventual.
Strong consistency means that the data is always consistent,
whatever happens into whatever server to query.
When write operation is finished, data is already
consistent among all servers.
Or if a server fails in the middle of a write operation or
in the middle of whatever operation, all servers return
the same result.
Eventual consistency indicates that data will eventually be
consistent.
But we have a certain timeframe where data is not
consistent.
DNS or domain name services is a good and easy to understand
example of eventual consistency.
DNS is thought of as a worldwide
distributed database system.
When you update a configuration on your domain,
it's not reflected to all DNS immediately.
For a certain period of time, some DNS servers return old
configurations and others return the new configuration.
But eventually, all DNS servers return the new
configuration.
A traditional RDB ensures strong consistency.
This is the reason why it's hard to ensure scalability
without sacrificing performance in RGB and the
reason why Google pursued another solution, datastore,
for App Engine.
Datastore is based on BigTable, which is a super
high scalability underlying technology
used throughout Google.
And BigTable is a key for datastore to secure
scalability.
So we'll spend some time to look into BigTable and how it
achieves it.
So this is the BigTable.
It's a mechanism to offer scalability to the datastore.
So what is a BigTable?
BigTable is a scalable, distributed, highly available
and structured storage.
So the key thing here is a BigTable is not
a database by itself.
It doesn't support the query.
So it's storage of the data.
It has strong consistency for a single row.
And I think the best part is, it's being used throughout
Google for more than seven years.
And it's used by web search, YouTube, analytics, and many
other projects throughout.
This is a pretty picture about how the BigTable scales and is
really the key of scalability.
Take a close look at one shard of data in the BigTable
tablet, which is just the name of the server.
So this shard, this tablet server,
has many write requests.
It has so many write requests that it needs to multiply.
And then we have two sets of tablets servers.
So then the load is distributed into two servers,
two tablets.
And this happens automatically.
So you don't really need to care about whether you need to
increase the number of servers or tablets.
This is the BigTable data model.
Big data is data storage, and it's a key value data storage.
So here in BigTable, we call a value as columns.
And internally in those, the data is
sorted by a key order--
key and lexical order.
And the facts are sorted by key to enable the range query
by the application.
Application means the user of the BigTable.
Here is one BigTable support.
BigTable supports CRUD.
Now that's create, read, update, and delete operations
on our own.
And it's strong consistency but not multiple rows.
And the support scans by range of keys.
So in this example, we're retrieving the rows with
prefix B. You want B to be three.
Or in the second example, you can get rows with keys ranging
from B to D. So this is what BigTable can do.
But what BigTable can't do is search by column values.
So you can't get column or rows where the column has
certain values.
This is a limitation of BigTable.
Yes, this is a Google solution for the datastore.
It's actually how we secure with scalability, reliability,
and performance.
So the scalability is based on BigTable automated sharding.
BigTable applications are reliable.
It ensures access to your data if the server is down.
And Megastore supports transactions, which offer you
the strong consistency.
Finally, performance--
reducing the lock granularity and allows
co-location of the data.
So during the transaction, the data you need can
be safe in one shard.
So that drastically reduces the necessity of the
communication be with the shard.
This is a big difference from the relational database.
This is how you know Google achieves these three.
This is the internals of the datastore.
Let's move on to the actual usage part--
the practical part.
The datastore and RDB use different terms for the same
corresponding concepts.
For example, the category of the object, which type of the
object, is a kind in datastore versus a table in RDB.
And the one entry of the data is an entity instead of a row.
And the unique identifier or a file is a key in both.
The individual data in the entity is called the property
instead of the field in RDB.
Perhaps this real example will be easier to understand.
Here's an example application of a
blog with social features.
We have two kinds of data here--
a blog entry and a user.
Each key is unique with one blog entry and two users.
These are called entities.
The property is the actual data you want to
store in the datastore.
So in this example, the blog entry, the name, message and
date are the properties.
Note user Joe has two followers.
And the property has multiple values here.
It's different from an RDB structure where we use two
tables, one for user and another for
relations of users.
Also, Joe has no followers, which is also valid.
Datastore allows us to have multiple
data for one property.
And it also allows the empty data like this.
So this is the other big difference from RDB.
This is a more practical example about how we create an
entry or entity in a Java low-level API.
The point is we first create an entity Java object.
Then we set properties.
And finally, we put this Java object into the datastore with
this put function, which actually communicates with the
datastore and it saves the data.
This same pattern is used in Python.
First create a Python object.
Then set properties, either by giving values to constructor
or by setting values afterwards.
And lastly, put saves the data into the datastore.
These are only some examples of the low-level APIs in both
Java and Python.
We actually have various kinds of APIs and also some third
party frameworks for Java you can choose from.
You might want one with a standard API providing you
with more portability or one with more
functionality, like caching.
Or another to use more high level APIs.
We might introduce new APIs.
So some API might be deprecated.
It can change from time to time.
The most important thing to remember from this lesson is
the datastore is a highly scalable and highly reliable
database that Google offers for App Engine.
Finally, here are some more documents you may want to
refer to for more details.