[MUSIC PLAYING]
WESLEY CHUN: Hi, and welcome to "Serverless Migration Station."
I'm Wesley from Google, joined by my colleague, Martin,
to help you navigate the world of migrating from one product
to another, usually as it relates
to one of our serverless compute platforms.
Today, Porter will show us the optional migration from Cloud
Datastore to Cloud Firestore.
MARTIN OMANDER: Oh, really?
Another option on migration?
What's wrong with you people?
WESLEY CHUN: [LAUGHS] I know, right?
To be fair, we did let developers
know in the module 2 and 3 videos
that Python 2 App Engine developers really
only need to move from the original NDB Library
to Cloud NDB.
Doing so makes it easier to port applications to Python 3
and also get them on Cloud Datastore,
the standalone, NoSQL solution that
grew out of the original App Engine Datastore.
Since the Cloud NDB client library works with Python 3 App
Engine, users would only further migrate to Cloud Datastore
or Cloud Firestore client libraries
if they had a really good reason to do so.
MARTIN OMANDER: Oh, yeah.
I think I remember, but can you remind me?
WESLEY CHUN: Sure, Martin.
Both the Cloud NDB and Cloud Datastore client libraries
access Cloud Datastore, the underlying database.
Cloud NDB was created specifically for Python 2 App
Engine developers as a migration tool,
while Python 3 and non-App Engine users
are directed to Cloud Datastore immediately.
So developers should only migrate
from Cloud NDB to the Cloud Datastore client library
if they already had code using it,
meaning it's where you want to consolidate
to using just one client library to talk to Datastore.
MARTIN OMANDER: Ah, that's right.
Thanks for the reminder.
But now what about today's migration to Cloud Firestore?
WESLEY CHUN: Glad you asked, Martin.
Well, there's good news and then there's bad news.
MARTIN OMANDER: Uh-oh.
Well, I'm an optimist.
What's the good news, Wes?
WESLEY CHUN: Well, Cloud Datastore
became its own product in 2013.
But the following year, Google acquired
Firebase for its amazing realtime database.
Both teams then worked together and launched
the next-generation scalable NoSQL database in 2017.
To show that Datastore joined forces with Firebase,
the product was rebranded as Cloud Firestore.
MARTIN OMANDER: Oh, yeah, and you
told me to wait until this video to learn more.
So here we are.
You also mentioned that Firestore
is fully compatible with Datastore,
and that all Datastore databases will be converted
to Firestore under the covers.
Is that right?
WESLEY CHUN: Yeah, Martin.
Once converted, they will be operating
in Datastore Compatibility Mode, or Firestore in Datastore mode,
for short.
This mode, however, does not give you access
to Firestore's native features, only Datastore's.
In order to access all of Firestore,
you need to be running in Firestore Native mode.
Similarly, when prompted to create a new NoSQL Datastore,
you get to choose between Firestore in Datastore mode
or Firestore in Native mode.
Check out the chart in the docs to see
the differences between them.
MARTIN OMANDER: Oh, I see.
So all users need to do is switch from Firestore
in Datastore mode to Native mode, and they're good, right?
But I think I better ask now.
What's the bad news you mentioned?
WESLEY CHUN: Yeah.
I think you know what I'm going to say.
You can't switch between Datastore and Native Firestore
modes.
Once the mode is set and data is stored, the mode is locked.
Also, to compound matters, Datastore and Firestore
are mutually exclusive in GCP projects,
meaning that once the mode is locked,
not only can you not change it.
But in order to use the other mode,
you need to actually create a brand new project.
MARTIN OMANDER: So what you're saying
is if I've got data in Datastore,
even if I migrate to the Cloud NDB or Cloud Datastore client
libraries, I'm not able to use Cloud Firestore in Native mode
unless I create a new project with a new app,
select Firestore in Native mode in that new project,
dump the Datastore data out of my current project,
and load it into Firestore in the new project?
WESLEY CHUN: Yep, you got it.
And now you know why this is even more
optional than moving from Cloud NDB to Cloud Datastore.
At least with that migration, you
don't have to create a brand new project.
But this one does require that.
And it doesn't matter if it's App Engine app, Cloud Run,
or even a non-Cloud app using Datastore.
You have to have a new project.
So once again, you really need to have a good reason
to make this migration.
And if accessing those Firebase features is compelling enough,
you got to do what you got to do, right?
MARTIN OMANDER: Yeah, that's right.
Well, I do want to take advantage of Firestore's
native features, and I'm willing to create a new project
to make it happen.
How do I get started?
WESLEY CHUN: Well, the first thing
is you must recognize that there are two pieces to this puzzle.
First, your data must be transferred
from Datastore in your current project
to Firestore in the new project.
That's part one.
Secondly, it probably goes without saying
that you need an app in that new project that uses the Firestore
client library.
MARTIN OMANDER: OK, I got it.
But before we do that, I remember from,
was it, module 3 that the Cloud Datastore client
library can be used by both App Engine and non-App Engine apps.
Is that true for Cloud Firestore, as well?
WESLEY CHUN: Yeah, definitely Martin.
While we're going to demo migrating an App Engine
app to Cloud Firestore, it could be any app, really.
The only real requirement is that you
have to have a new project.
And since you mentioned module 3,
that's where we created our Cloud Datastore app.
So we're going to migrate that one to Cloud Firestore.
Specifically the Python 3 version, as Python 2 apps
are more likely to be using Cloud NDB instead.
So if you did the codelab, use your code,
or grab ours from the module 3b folder
by either cloning the repo or downloading the zip file.
Pause here if you need to.
But once you're ready, let's go to the computer now.
OK, before we start, we need another Cloud project
for Firestore.
So make a new one in the Cloud Console,
or reuse an existing one where neither Datastore nor Firestore
are used.
Grab that project's ID--
not the project name, nor number, the ID.
Then enable the Cloud Firestore API.
If you prefer the command line, use gcloud projects create,
then change to that new project with gcloud config set,
and finally, turn on Firestore with gcloud services enable.
Of course, you'll be asked to enable billing, as well.
Since we always suggest you start with working code,
deploy the module 3 Python 3 app,
whether you're using yours or ours.
But do this from your old project,
the one using Datastore.
Use gcloud config list to see your current project,
and use gcloud config set, if necessary.
Once you're on your Datastore project,
run gcloud app deploy and confirm the app works
like before.
Now switch to your new project with gcloud config set project
one more time, as we'll be here the rest of this migration,
starting with the configuration files.
The only config you need to change
is to switch from Cloud Datastore, or Cloud NDB
if you're using that, to Cloud Firestore.
There should be a one-line swap in requirements.txt.
All other changes are in main.py.
Replace the Cloud Datastore client library imports
and API client instantiation at the top with the same for Cloud
Firestore.
Groups of like entities in Datastore
are identified with a common parent key.
Entities in Datastore are referred
to as documents in Firestore.
Entities are of a certain kind in Datastore,
while Firestore documents are grouped as collections.
When storing a new visit in Datastore,
a new entity is created with the visit parent key.
Likewise on the Firestore side, we
set our collection to the same name as a Datastore parent key.
The Datastore entity gets its data key-value pairs
in the same way that a Firestore document does,
but the latter gets added to the collection
immediately, while the Datastore entity has
to be written explicitly.
Next are the Datastore queries where
you query for visit entity kinds, the desired ordering,
then fetch and return.
For Firestore, query the visit collection,
specify the ordering, fetch and convert
to Python-friendly dictionaries, then
return as a memory-friendly iterable.
Believe it or not, that wraps the changes
necessary to switch from Cloud Datastore to Cloud Firestore.
The real work is in having to create a new project.
So what do you think of that migration, Martin?
MARTIN OMANDER: That migration wasn't too bad at all.
But earlier, you said any app can use Cloud Firestore.
If I'm willing to create a new project to access Firestore
and take the effort to migrate my data over,
does this mean I can rewrite my Python 2 Datastore
app as a Node.js Firestore app?
WESLEY CHUN: Absolutely, Martin.
Your Datastore app could be App Engine, Cloud Run,
or even non-Cloud.
It's not like the old App Engine days,
where if you're using Datastore, it had to be an App Engine app.
That's certainly not the case anymore.
So the same applies to your new project,
meaning you can have any app using
Firestore as its database.
So if your company is doing more work in Node.js today,
there's nothing wrong with rebuilding or refactoring
as a Node 16 Cloud Run app for your new Firestore project.
That's what we mean by App Engine, and really
all of Google Cloud as a whole, being
more open than ever before, for both your data
and for your apps.
MARTIN OMANDER: Hm.
Well, that's really interesting.
I've been thinking about doing exactly
that, as my team is doing more with Node.js these days.
Anyway, now that I know how to migrate my app to Firestore,
what about migrating my data?
WESLEY CHUN: Definitely, Martin.
We're going to tackle that in module 10.
While you're awaiting for the data migration sequel,
here are some additional resources to consider.
Earlier, we linked you to a chart differentiating Datastore
and Firestore Native modes.
Above that chart is our guidance on which of those modes
to choose for your apps, so check that page in the docs
and just look over all of the Firestore docs,
anyway, to become familiar with your brand new NoSQL database.
Also, check out this other video from one of our colleagues
that dives deeper into the migration and the differences
between Datastore and Firestore.
MARTIN OMANDER: Cool.
I'll check those out.
Now are there other migrations I should
consider for my App Engine app?
WESLEY CHUN: Well, first, you can see from this list,
we've already migrated from our old App Engine
Datastore and NDB to Cloud NDB, Cloud Datastore, and now
Cloud Firestore.
However, if you're on Cloud NDB, you
can migrate directly to Cloud Firestore.
There's no need to migrate first to Cloud Datastore
and then go from Datastore to Firestore.
just combine module 3 with this module 6.
As far as other migrations go, if you're
thinking about containerizing your App Engine
apps to run on Cloud Run, go and check out modules 4 and 5.
Modules 7, 8, and 9 help you learn
about how to migrate from App Engine Task Queue
to Cloud Tasks.
Finally, if you've got a small app that's just one function,
or a big app that you want to break up
into multiple microservices, consider
migrating to Cloud Functions.
You'll find that in module 11.
MARTIN OMANDER: Ah.
Thanks for the pointers.
There are so many ways to upgrade.
WESLEY CHUN: Well, that's definitely true.
We want to give our users lots of options.
And thank you, viewers, for staying
with us on these journeys modernizing your serverless
apps.
This is Wesley from Google Cloud on behalf of Martin and Porter,
hoping you'll join us soon at the next "Migration Station"
or on a different "Serverless Expedition" soon.
Please like and subscribe to see more content like this.
And until next time, happy travels.
[MUSIC PLAYING]