[MUSIC PLAYING]
WESLEY CHUN: Hi, and welcome to "Serverless Migration Station."
I'm Wesley from Google, joined by Martin to familiarize you
with modernizing your apps on one
of our serverless compute platforms.
This time, our little friend Porter
will take us on the final leg of our migration to Cloud Tasks,
doing a mixture of optional migrations.
MARTIN OMANDER: More optional migrations, Wes?
Why are we even making this video, then?
WESLEY CHUN: Well, when modernizing your apps,
there are going to be some steps which
are less critical than others.
But they may be necessary at some point, so we're here
to show our friends out there how
to do them when the time comes.
MARTIN OMANDER: I suppose you're right.
How many of these optional migrations are we looking at
and what are they?
WESLEY CHUN: Aha.
Glad you're a good sport.
We're doing three migrations today,
all optional but recommended.
First, we're porting from Python 2 to 3.
You probably should move to Python 3,
since Python 2 has been sunset and that's
where the community has moved to, right?
Honestly, this app is already compatible with both Python 2
and 3, so there's really not much work to be done here.
Next, moving to Cloud Tasks is also pretty much a no-op--
meaning you don't really need to change anything
beyond the version numbers.
However, this may be different in future versions.
For now, it suffices to say that development on Cloud Tasks
has froze for Python 2.
Hence why the client library is on version 1,
whereas Python 3's is on version 2.
Finally, we know from modules 2 and 3
that Cloud NDB is just a migration tool for Python 2 App
Engine developers.
From modules 6 and 10, we know that Cloud Firestore
is the latest NoSQL database solution from Google Cloud.
By going straight there, we get to jump over the Cloud
Datastore migration.
Moving directly to Firestore does
require a completely different project,
so this endeavor is optional, but more
like building a prototype to see what these migrations entail
and for you to consider before doing these to your own apps.
MARTIN OMANDER: OK, those all make sense.
It's been a while since I've done one of these migrations.
Before we do the migration today,
can I get a quick review of how we got here?
WESLEY CHUN: Yep.
Sure, Martin.
Previously in module 8, we migrated from App Engine task
queues to Cloud Tasks.
Because migrating to Cloud Firestore
requires a brand new project, you should get more details
from module 6.
Links to the videos and codelabs are below so you can catch up,
as we're going to pick up where module 8 left off
and merge in part of that module 6 migration.
MARTIN OMANDER: Sounds good, Wes.
You know I'm here to learn all about the ways
to modernize my apps and get hands-on experience doing so.
Can you put me to code, as well as the codelab,
so I can do this migration on my own time after watching you?
WESLEY CHUN: Of course, Martin.
Same drill as before for all of you.
Use your module 8 solution or clone the repo
or download the zip file if you don't have it.
We'll start with the module 8 code
and finish with what's in the module 9 repo folder.
Links to the repo and codelab are down below, so pause here
if you need to go get anything.
Ready?
Let's dive into these migrations.
Since we're doing several migrations with a bunch of code
changes, you have to confirm you're
starting with the working module 8 code, whether yours or ours.
If you're on Python 2, double-check your lib folder
or blow it away and run the pip install command
to reinstall those third-party packages into the lib folder.
Then use gcloud app deploy and confirm your App Engine
app outputs the same most recent visit list and possibly
older visits to delete with a push task.
Now, if things look good, copy the entire folder,
as we're going to make major changes to it,
as we will deploy to a completely different project.
First up, app.yaml, where modern App Engine
doesn't require much.
For simple apps like this one, the most critical part
is switching from Python 2 to 3, and as a result,
moving from the first generation to second-generation service.
One big change is that web frameworks
must do their own routing.
This means that all app handlers have to switch to auto.
But if everything is auto, you really
don't need them at all, especially
since the framework does it.
So yes, you can remove all app handlers, like what we did.
But if you've got static files, leave them here,
because that does not go through the framework.
Next, the section 4 built-in third party libraries
is no longer needed.
They now go into requirements.txt.
Similarly, there's no more copying or self-bundling
of non-built-in libraries.
You just put them into requirements.txt and that's it.
This is covered in a bit more detail in the module 2 video,
linked below.
For our simple app, the net result
is that you can delete all lines except for the runtime, where
we're sending it to python38 here.
Since all third-party management is in requirements.txt,
that means no more appengine_config file,
so delete that, too.
For requirements.txt, we know that all third-party packages
go in here now.
Version numbers do need a bump, because every library
has a final version that works on Python 2.
For us, this was Cloud Tasks 1.5.
Today, the version for Python 3 is 2.3,
but that too will change.
Similarly, we're switching from Cloud NDB to Cloud Firestore,
so this line has to be completely swapped.
In Python 2, we self-bundled third-party libraries with pip
install.
But in Python 3, don't do that.
Seriously, Google takes care of this
when deploying so you don't have to lift a finger.
Are we liking Python 3 yet?
Anyway, that's it for the config files.
Let's turn to the real work, and that's in main.py.
At the top, there are two changes.
One is that logging in Python 3 is easier.
You don't need to use Python logging, just print to stdout.
So remove that import.
Next, we're switching from Cloud NDB to Cloud Firestore,
so swap that client library import.
Finally, instantiate the correct API clients.
Next, all of the constants for task queues stay the same,
but we have to add a new one.
The QUEUE_PATH variable is the fully-qualified queue path
name.
One difference between App Engine task queues
and Cloud Tasks we didn't cover last time in module 8
is that when using App Engine push queues,
the default queue was automatically
created by App Engine.
But that's not the case with Cloud Tasks, which
is independent of App Engine.
With Cloud Tasks, you need to create all of your push queues,
including the default queue if you want one.
And to do so, you need the path name up to, but not including,
the queue name.
So that's what the PATH_PREFIX is.
For store_visit(), the changes you see are the differences
between Cloud NDB and Cloud Firestore.
Cloud Firestore doesn't have a data model,
nor does it require context managers and with statements.
In NDB, we refer to data records as entities of a certain kind,
while Firestore has documents belonging to collections.
Regardless of all this, know that store_visit() has one job.
It creates a new data record with the connecting client IP,
as well as their web client or user agent.
One last note.
NDB automatically adds a timestamp
when an object is created, but it must be done explicitly
in Firestore.
Since App Engine task queues create a default queue
by default, we didn't need a function like
_create_queue_if() in our module 8 main.py.
Before creating a task, this function
is used to check if we need to create
the queue, which is only done if it doesn't already exist.
Next is fetch_visits().
At the top, querying for the most recent visits
has to change from Cloud NDB to Cloud Firestore.
Again, the functionality is identical.
The differences are just between how
NDB does it versus Firestore.
For the next section, fortunately, Cloud Tasks
doesn't change between version 1 and version 2 of the client
library, so no real updates to your code
are needed as to how the task is created.
Now the logging call has to change to a print
and we have to add that check to make sure the queue exists
before adding the task.
You may ask why we didn't do this before
with Cloud Tasks in module 8.
And again, it's because the module 7 code used App Engine
task queues, where the default queue was automatically
created.
Now, it wasn't until this app where
we had to use Cloud Firestore, requiring us to create
a completely brand new project, that we discovered that we had
to make our own default queue.
Let's skip over this new function, _delete_docs(),
for just a moment and dive into the trim() task handler.
We saw just a moment ago from fetch_visits() how querying is
identical, but NDB and Firestore do it slightly differently.
The same applies to this query, looking for visits older
than the oldest one shown.
One subtle difference is that NDB has convenient batching,
like a keys-only query since I don't need to look at the data,
plus a delete multi-function for batch delete.
In Firestore, you would issue a regular query, then loop
through and delete them one at a time or use a batch delete.
Since I want to log all the objects deleted,
I moved the deletion loop into _delete_docs,
a generator that yields the ID for each document deleted so I
can count how many were deleted and log that back in trim().
Otherwise, I logged that there weren't any old records
to delete.
The main GET handler doesn't change at all,
so we're skipping that.
So overall, that was probably one
of the more intense migrations so far in this series
if you look at how much code was deleted,
updated, or new code was added.
Simple migrations are great for teaching,
but generally not as realistic.
So the goal was to try and make it more lifelike.
We actually forgot one last migration
of porting this app from Python 2 to 3.
But don't worry, I'm not going to keep
this video running too long.
I deliberately crafted all of the sample
apps to run on both Python 2 and 3 without any modifications
or compatibility libraries, so the porting part
is already done.
Now that everyone's on Python 3, you've got no more pip install,
so just run gcloud app deploy to confirm it
works as advertised, making sure that you
change to your new project before doing so.
Because we're doing more migrations in this module,
the chances of doing something wrong
is higher, so let the logs help you or you can check out
our solution in the module 9 repo folder
to compare your answers with ours.
All right.
Let's go back to the main presentation.
MARTIN OMANDER: Wow, lots of different migrations and things
to consider.
If I wanted to get more information about migrating
my App Engine app from Python 2 to 3, where should I look?
WESLEY CHUN: Yeah.
The best place is the Python 2 to 3
App Engine migration guide.
It links to all the individual pages
for migrating to Cloud Datastore, Cloud Tasks, and all
the other standalone services.
MARTIN OMANDER: OK.
And same for Cloud NDB and Cloud Firestore?
I'd like to review both to get a better idea of the differences
between them.
Do you have any links for them?
WESLEY CHUN: Yep, sure thing.
While we don't have a formal Cloud NDB product
page, nor an NDB to Firestore migration guide,
your best resources are the client library docs
for both, as well as the general Cloud Firestore docs.
MARTIN OMANDER: Ah.
Yeah, those look really useful.
Also, now that we're checking for and creating our own task
queues, I need to read up on those?
WESLEY CHUN: Gotcha.
Well, all the links mentioned here, plus the Cloud Task
client library and general docs, are down below for your reading
pleasure.
MARTIN OMANDER: Great.
Thanks for those, Wes.
Are we doing any more task queue migrations?
WESLEY CHUN: Nope.
Well, this is the last of the task queue miniseries.
Summarizing today's adventure, we ported our app from Python 2
to 3.
We moved from Cloud NDB to Cloud Firestore,
combining modules 3 and 6.
And we revved up to the latest Cloud Tasks client library,
including adding new code to create the default
queue, something we didn't need before since we were still
running on our original project where the App Engine task
queues system automatically created that default queue.
MARTIN OMANDER: Cool.
What other migrations should I consider for my App Engine app?
WESLEY CHUN: Well, since we've already
done Cloud NDB, Cloud Datastore, and Firestore,
you're done with all the database stuff.
Now if you're thinking about containerizing your App Engine
apps to run on Cloud Run, well, check out modules 4 and 5.
Finally, if you've got a tiny app that's
just one function or a giant app you
want to break up into microservices,
those sound like really good use cases for Cloud Functions,
and that migration is covered in module 11.
There are other migrations coming,
so look out for blog posts, as well as checking
the repo from time to time for more migration content.
MARTIN OMANDER: Thanks, Wes.
I'll check those out.
WESLEY CHUN: And thank all of you
for sticking with us for all these different migrations.
This is Wesley, on behalf of Martin and Porter,
and we'll see you at the next "Migration Station"
or on another "Serverless Expedition" soon.
[MUSIC PLAYING]