So now, I'd like to discuss App Engine
Architecture and Services.
So before I get started, I'd like to warn you one thing.
Google App Engine can be dangerous for developers
because it can change your life.
I found that I no longer was interested in ordinary,
traditional web application technology anymore.
After the App Engine project, I had to build another typical
MySQL kind of web server with applications.
And I was so tired of doing that.
So this job is very dangerous and will change the way you
approach web application development.
In this lesson, I'd like to discuss about three things.
One is the scalability and reliability that's provided by
Google App Engine.
And I'd like to compare that between the traditional way of
designing web applications and the App Engine design.
I'd also like to discuss two major components that are
running inside the App Engine stack.
One is the Front End and how the Front End works.
And another is the App Server and how it works.
The first topic, "Designing for Scale and Reliability." So
here's an excellent example of an App Engine application
that's called Real time Earthquake Monitor.
And it's built by the NIED, developed by the
public sector in Japan.
This is the actual app running.
You can see that there are many blinking dots, blue dots
or green dots, on the map of Japan.
Each dot represents the acceleration of the ground
surface right now in Japan.
So if you're having an earthquake right now, you'll
be seeing yellow or red dots will be spreading out all over
that region.
And I have another demonstration that shows you
how it worked in the 3/11 disaster we had last year.
So you'll be able to see red and yellow dots spreading out
from the northern part of Japan.
Yes, that's what happened on the 3/11 disaster we had.
So let's discuss how the app is designed for
this kind of system.
So this system is built with the App Engine as well as the
data center which is hosted by the NIED.
It has many sensors on the surface of the
ground all over Japan.
And all the metrics will be collected by them at the data
centers for us where they can create an image file.
They ping and publish it in the Google App Engine every
one second.
And then the client, the browsers, the users, are
polling that image every two seconds.
So that means at peak rate, since there are about 20,000
people, the App Engine will be getting about 10,000 requests
per second at peak.
So what are the important requirements for this app?
We can summarize these as scalability, reliability, and
cost efficiency.
So scalability because this site will be getting a spike
of traffic in the event of the earthquake.
So it has to provide more scalability than
your ordinary website.
And also, it has to provide these so high of a reliability
because the people will be using this site only when they
have the disaster.
If you're connecting to this site and it's down because of
the disaster, then it doesn't mean anything.
And also, it has to be more efficient on cost.
If you're preparing for the old hardware resources that
will be required to provide the scalability, then it means
that almost all the time that hardware
resource will be resting.
Nobody's connecting using that.
So you have to be scalable enough to provide enough
server resources, but you don't have that much money to
buy all the hardware resources itself.
So how do we design a web service like that that's
scalable and reliable?
This is how you do it in a traditional way of web
application design.
Because it should be reliable and scalable, you may have
multiple instances of Apache with a
server or MySQL instances.
But if you have a single hardware failure on the
server, that should failover instantly.
It means you also have to be prepared for load balances or
reverse proxies.
And we'd need to buy two or three of them.
And if you have a traffic spike, then the load should be
balanced among the multiple instances of the server.
And if you are handling big data and if it's growing
rapidly, then it means you have to have multiple MySQL.
And you have to design a sharded MySQL instance.
And this can be very hard.
If you have experience developing this, then you know
you have to configure and test everything,
every possible pattern.
What if you have a failure on the reverse proxy?
What if you have multiple failures on your web service?
You have to configure and test them all.
And don't forget that you have to hire the people who have
this experience.
It all costs so much.
So App Engine provides them all, allows you to build that
kind of cluster and encapsulate it in an App
Engine application, all running on
Google's data centers.
You don't need to administer that anymore, like if you have
a hardware failure on one server, your application will
not stop at all.
If you have a traffic spike, the app service automatically
will add more instances on demand based on the daily
number of the request.
If you're having so much big data that it's growing
rapidly, automatically, Datastores will be splitting
your data into multiple shards of Datastore servers.
And you can start using App Engine without
any money up front.
You don't need to hire expensive administrators to
operate this massive cluster.
After migrating onto App Engine, you don't have to have
a dedicated experienced Linux administrator anymore.
And let them focus on the more business
essential thing to do.
To appreciate this App Engine design more fully, you have to
understand what is the design goal behind
the App Engine stack.
It's not just the hosting environment.
And it's not just the data center.
Google App Engine encourages you to use Google's best
practices to design your applications.
So if you have experience in producing a project that is
using App Engine, you may have noticed that there are certain
restrictions, rules, and practices to follow.
But these are important best practices to make your
application design in the Googly way; scalable and
reliable, like the non-relational data model.
When you're using Datastore--
not Cloud SQL or MySQL instances--
you have to design your data model that can be sharded.
And sometimes, you may want to use the techniques, like in
data normalization.
That may sound like a downside, but if you're an
experienced web application developer, you probably
realize that this is actually a best practice to make any
scalable and reliable website.
So the most important point is that you have to make your
application portable and fine-grained.
By designing your application in this way, you can use the
whole data center as the computer.
You are not writing your code on a specific server anymore.
You are writing code for the data center or the group of
data centers.
You may notice when you're using these APIs, like
Datastore APIs or Memcache with App Engine, that these
APIs are abstracting a whole bunch of data centers.
It's not just an API running on a single server.
So in that way, you can utilize the server resources
at a very high utilization rate.
That ultimately means you get significantly lower Total Cost
of Ownership.
You can leverage the scale of economy because your App
Engine is sharing the exactly the same Google infrastructure
that's used by other Google services, such as Google
Search or Maps.
Google's SREs or site reliability engineers will be
watching your systems 24 by 7.
And actually, Google SREs are one of the premium roles in
Google's entire company.
It's very hard to hire these kind of people.
So to utilize and realize benefits--
getting benefits from those goals are design principles
behind the App Engine stack--
you have to understand how things are working inside the
Google App Engine stack.
Now, I'd like to go deeper and dive into the two major
components.
One is the Front End.
And the other is the App Server.
These parts, right here.
So now, to the second topic, "How the Front End works."
Actually, I really like this diagram because it tells so
much to me about what's happening inside the App
Engine stack.
So we'll refer back to it throughout this lesson.
What benefits would the Front End provide?
So here, you have the actual graph that I have taken from
the Earthquake Real time Monitor applications.
And you can see that the number of requests per second
on the site is getting most of the time, it's around 1,000
requests per second if you're not having an
earthquake or event.
So it's still a very big number of requests.
And you can see the blue line.
This is actually the number of requests routed to the
application instance.
It's like 1/10 or 1/12.
There's a huge difference.
And it means that you don't have to pay for this
difference.
You can just pay for the blue request.
So how do they do that?
Earthquake site broadcasts the image to thousands of users
without handling all the requests by the app.
How can you do that?
So the secret is in the Front End.
In the Front End component, there are actually two
different components.
One is called the Google Front End.
And another thing is called the App Engine Front End.
And Google Front End resides in a data center which is
closest to the user.
So if you have global users, then all the data centers
worldwide we have in each region will be accepting
requests from those users because we have so many close
relationship pairings--
our relationship with the major ISPs--
most of the requests users will be sending to us will be
accepted, received by the closest data center.
And Google's Front End is running
inside that data center.
And that's shared by most of the other Google services,
like Google Search or Google Maps.
And it also has the edge caching capabilities.
This is the key technology.
And afterwards, because it's received by Google, that
request will be routed to other data centers that's
running the actual App Engine services.
We have Google's fibre backbone, so it
will be routed fast.
And that second data center is running another Front End
called our App Engine Front End that will provide the load
balancing among the multiple instances of the app servers
as well as the static servers.
So here, the key points are edge
caching and load balancing.
So how do you enable the edge caching feature?
There are two ways of using edge caching.
One is setting the cache-control header on your
HTTP request.
Then another one is using the Static content configuration.
You can just set these kinds of headers and set the
expiration period, like whatever amount you want, to
tell Google's Front End to cache the content.
And also, there's another way that you can
define a Static content.
So if you have a static HTML file or an image file on your
applications, those are all served by the App Server, from
an App Server to the browsers.
So it costs you.
Instead, you can define a Static content, like this, in
the configuration file so that all the content will be served
by the static servers.
So the last topic is, "How the App Server works." Now, let's
look at the App Server in detail.
What if the App Server can dynamically increase the
number of your app instances based on the load you get?
And the App Engine developer needs to know how you can
control it and get the best cost-performance efficiency.
The App Server is our server that provides our runtime
environment for your application instances.
Just like the virtual machines, like when you're
deploying a new application on App Engine, the App Master
will initially ask App Servers to host that particular
application.
And if your application is getting many requests from
users, then the App Master directs App Servers to have
more application instances to scale out.
And also, App Master tells the App Engine Front End the
location of the App Server where your
application is running.
So the App Master orchestrates the whole load balancing or
scaling out; whereas, the App Server provides runtime
environment for the application instance.
Application instances in App Engine is very similar to
virtual machines.
And also, just like in virtual machines, has a dedicated
memory for your application.
But the biggest difference is it's fully managed.
You don't have to bear the burden of managing the whole
operating system or application stack.
You don't have to care about what version of the device
driver you're using or what kind of security patches
you've applied to the OS or what versions of the libraries
or application services you're using.
You don't have to queue them all up.
Everything's taken care of by Google.
So that's the huge difference.
And also importantly, if you're running always inside
the virtual machine, there is a certain big overhead for
that because you're running different OS images on
different virtual machine instances.
It is a certain overhead, like it can be an overhead in terms
of the CPU hours, for example.
So application instances in App Engine
doesn't have that overhead.
And also, it's fully managed.
You have to understand there are two different types of
application instances.
One is a Frontend instance.
And another is a Backend instance.
In Frontend instance, it's suitable for processing
short-lived requests, like when you click a button on
your web application.
And your request should be handled within a few seconds.
Those kinds of requests should be handled by
the Frontend, generally.
Instead Backends should be used for long-lived requests,
like batch processing, that should be
running during the night.
Or if the process would take 10 minutes or 10's of minutes.
And Frontend instances will be
dynamically created and deleted.
It means you can minimize the costs of the CPU hours where
the Backend instances will be statically--
not totally statically.
There are some dynamic characteristics of the Backend
instance, too.
But it can be a higher cost value Frontend instance.
Frontend instances, as you may know, there's a big
limitation--
the 60-second limitation on that.
So you have to design your systems to process everything,
to process every request within 60 seconds.
And also, you want to make your application in a
stateless design; whereas, the Backend instance, there's no
limit for a response time.
So you can spend like 10's of minutes to
handle a single request.
And you can rely on the statefulness
of the server instance.
But currently, we are having the separation between them.
And I'd like to show you how the Frontend instance will be
scaling out.
App Engine are watching the status of the Pending Requests
Queues that's held by each application instance.
Each application instance has a queue that will be holding
the request from the users.
So if you're getting a spike of traffic, that means the
queue will be filled up with many requests.
In that case, the App Engine detects it and is trying to
add more and more Frontend instances.
Once you have processed all the requests from the queue,
then App Engine tries to reduce the number of Frontend
instances--
so dynamically scaling out and scaling in.
And the important thing to notice is that there are two
important parameters.
One is the Pending Latency.
And the other is Idle Instances.
What is Pending Latency?
Pending Latency is the time to hold the largest in the
pending queue before creating a new instance.
It means if you are setting a very short time period to
these parameters, then App Engine will be creating more
and more instances instantly without waiting for a certain
amount of time.
If you're setting this amount to be a very big amount, like
15 seconds, then your
application can respond slowly.
Plus, you can save your cost for the instance hours.
So you can have a balance between the trade-offs,
between the cost and the responsiveness.
Also, there's another important parameter called
Idle Instances, which is the number of the instances you
will be having even if your application is not getting any
requests from users.
If you set this number to something, like 10 instances,
10 idle instances, then it means even if you're getting
no requests from users, there will be 10 application
instances running all the time.
It will cost you higher, but it can respond to the sudden
spike of traffic much more quickly.
How long will it take for creating a new instance?
Whenever a server tries to create a new application
instance, the App Server will actually be reading the .class
file or the .py file of the application loading it, so the
instance can do the initialization.
Normally, it would be under one second.
But if you're using so many libraries throughout, then
sometimes, like using the Spring Framework for example,
it sometimes takes 30 seconds or 60 seconds.
And you'll be seeing exceptions in that case.
So every App Engine developer has to be careful about the
loading request and what latency you will get when your
application will be instantiated.
Key Takeaways.
App Engine encapsulates all the cost and the effort to
build scalable and available applications on its cluster.
It's not just a hosting platform.
It's a platform that encourages users to design
your system in Google's best practices for scalability and
reliability.
The Front End provides the Edge
Caching and Load Balancing.
These are very important technologies to utilize.
And the App Server controls the scalability of the
Frontend instance and the Backend instances.
Pending Latencies and Idle Instances are key parameters
to use for controlling that behavior.