[MUSIC PLAYING]
ANDREA WU: Hi.
Welcome back to the second video of the series,
where we are building an Expense Tracker web app using
Firebase, React, and Next.
If you did not watch the first one, don't worry.
You can jump right into this one without needing
to go back to the first one.
As a recap, we built functionality for a user
to create an account and log in as well as add, view, edit,
and delete expenses.
We did so using Firebase Authentication, Storage,
and Firestore.
And after building out all of that,
we deployed our great web app to Firebase Hosting
so we can share it with our friends to use.
If you'd like to go through that video,
the link is in the description.
Just as a side note, we will be using a bit of Firestore
in this video as well.
So if you need a refresher, feel free to check out
the first video of the series or our "Getting Started
with Firestore" video linked in the description.
[MUSIC PLAYING]
So what are we going to do today?
Well, sometimes when we accumulate a lot of receipts
for whatever reason, maybe we just
went on a long vacation or a work trip,
had to buy a lot of items to decorate a new home,
or we just felt particularly spendy,
it'd be really slow and inconvenient to have
to manually add each receipt to our expense tracking app.
What if we could upload images to our web app,
and it would automatically extract
the relevant information for us to confirm?
That's probably a lot more efficient,
and I won't hate having to go through this huge pile
of receipts.
With the help of Google Cloud's Vision API and Firebase Cloud
Functions, we're going to implement Optical Character
Recognition or OCR so our users can do exactly
that, upload images and get back the relevant information
without needing to type in every single field themselves.
All they'll need to do is upload the receipts
and then confirm the contents of the receipt.
Text recognition will automatically do the rest.
Being a bit more technically specific,
we're going to write a Firebase Cloud
Function that gets triggered when a new image gets
uploaded to storage.
We'll get the newly uploaded image
and pass it to the Google Cloud Vision
API, which will return the text on the receipt.
We then have to parse it, which we're
going to bypass here because it's complicated to parse.
And there are libraries out there to do this sort of thing.
After we extract the information we need from the receipt,
we'll add it to Firestore with a Boolean
for whether it still needs user confirmation.
And we'll show it in a particular section on the UI
to let the user know that that receipt needs confirmation.
Once a user confirms the receipt,
we'll flip the Boolean in Firestore
and then show the expense in the Confirmed Expenses section
on the UI.
As a side note, if you came from the first video of this series,
aside from adding a Boolean for whether the receipt needs user
confirmation, I have removed allowing users
to update the image once it's added because it doesn't really
make any sense for users to upload an entirely
new receipt when the information all
needs to get extracted again.
They might as well delete the old receipt
and upload a new one.
Without further ado, let's start building this feature.
You can get the starter code by git cloning the GitHub
repo linked in the description.
If you came straight from the first video,
you'll probably still want to git clone this repo
because some of the component details
changed to align with the addition of confirming
receipts.
And just as a reminder, while we are
using React and Next to build this app,
I want to focus on building the functionality with Firebase
and Google Cloud and not the components themselves.
So let's get into it.
[MUSIC PLAYING]
At a high level, when a user adds a receipt,
we're going to want to automatically parse that data.
You might be thinking to yourself,
that sounds like back end.
But I don't really want to make a back end
system because then I'd have to manage
servers and infrastructure.
We probably don't want to do all of the processing
on the client side, either, for a few reasons.
First, once we write our top secret code
to process the receipts, we don't want anyone
to just be able to take it along with the data.
Requests coming in from users' phones
are considered an untrusted environment.
Someone could modify the code just
for that one page or device.
So we need a trusted environment to help us
process the receipts securely.
Next, this might not be super relevant to us today,
but imagine we also built an iOS and Android
app for this Expense Tracker so a user on any device
could use it.
If that were the case, and we put the processing
code in the client, we'll have to duplicate the code
to process the receipts three times, one for each platform.
It'd be so much better if we could just
write it once and have the code run in one place instead
of writing it once for iOS and deploying to the App Store
and writing it once for Android to deploy to the Play Store
and writing it once for web to deploy to hosting.
Also, if we deployed the app separately,
iOS users may get this cool feature first,
and Android or web users might get mad.
And we definitely don't want that to happen.
Lastly, we're looking to process our receipts.
And to do, so we'll likely need to run text recognition
on the receipts, which we can imagine might be CPU-intensive.
The last thing we want is to run highly intensive code
on our users' devices, especially if it
might drain a user's battery.
So what should we do?
Well, Firebase Cloud Functions enables
you to run code in the Cloud.
It's in a secure environment, and you can write it once
to be used for any platform.
To run the code, you can trigger it
based on platforming events or HTTP requests.
And since it's running on the Cloud off of the user's device,
this won't be hard on their devices.
Best of all, you can basically run back end code
without managing any servers.
In addition, Cloud Functions automatically scales
up and down to meet user demand, which
means that even if the app is an overnight success,
Firebase can handle that.
And you'll only pay for what you use,
which means if you take your app down because of some bugs,
and there are no users, you won't have to pay for anything.
If you do have users, then how much you pay
depends on how many times these functions get triggered.
You do have to be on a pay as you go plan, though,
to use functions.
How do we turn on these functions?
Cloud Functions works with a variety of Firebase products,
including Firestore, Storage, and Authentication.
In our case when an image gets added to Storage,
we're going to want to get that image and process it.
And in general for Storage, you can
trigger a function in response to uploading, updating,
or deleting files and folders in Storage.
[MUSIC PLAYING]
Before we can implement the code to do this,
we're going to initialize Cloud Functions.
You may have noticed that nowhere in the starter code
is any mention or folder or file about Cloud Functions,
but we're about to change that.
There are a few steps we need to take to accomplish this.
First, if you don't already have a Firebase project
from the first video, feel free to follow
the link in the description on how to do so.
If you do already have a Firebase project,
please go to a Firebase Console, copy your project configuration
by going to Project Settings, and paste
that into the Firebase Config variable in Firebase.js.
Next, copy the storage bucket URL
and open Storage.js to set the bucket_URL variable
as what you just copied.
This is going to make sure we upload
images to the correct storage bucket for your project.
We'll also have to do one more step in Firebase Console, which
is to upgrade to the Blaze Plan if you haven't already.
Remember, we cannot use Cloud Functions otherwise.
We can go to Usage and Billing, then Details and Settings
and click on Modify Plan.
Select the pay as you go plan, and we
can set a billing budget, which means
you'll get an email notification whenever
the bill goes over whatever amount you put in here.
I'll put in a dollar since I don't really
want to spend money on this right
now, and then click Purchase.
Next, we'll need to install the Firebase
CLI by doing npm install -g firebase-tools in the command
line.
After that finishes, let's run Firebase login,
which will have us log in via the browser.
Choose the appropriate email address, the one with which you
created your Firebase project, and allow Firebase CLI
to access your Google account.
Upon success, the UI will say so, and so
will the command line.
We'll then do firebase init functions which will take us
through some project setups.
We'll use an existing project, so scroll through
to find the one you're looking for.
After doing so, we then choose JavaScript
for language, no to ESLint, and yes to installing dependencies.
With that, our Firebase initialization is complete.
And we see here that there is a functions folder
and an index.jls file now.
That's where we're going to write
all of our functions code.
So let's go ahead and write some code.
We'll create a new function called readReceiptDetails,
and we'll use functions.storage.object
to listen for object changes on the default Cloud Storage
bucket.
We can use the onFinalize event which
is sent when a new object or a new generation of an existing
object is successfully created in the bucket.
This includes copying or rewriting an existing object,
and a failed upload does not trigger this event,
meaning if the receipt fail to upload or update Storage,
those events will not be triggered.
Once that is triggered, we get back a Cloud Storage object
which contains a variety of attributes, including
the storage bucket that contains the file, the file
path, and the bucket content type and file size.
We'll need the bucket and name for OCR and checking Firestore.
So now we're ready to do some image processing
on our receipt.
[MUSIC PLAYING]
Before we do so, we're going to enable the Google Cloud Vision
API.
We'll go to Google Cloud's Console, which
is console.cloud.google.com.
And let's make sure we pick the correct project.
Once we do, we can go to the search bar and type in vision.
And we see this Cloud Vision API under the Marketplace section.
After clicking on that, we can enable the API.
All right, so now we can go back to our code and use this API.
We'll need to npm install @google-cloud/vision first
in the functions folder.
After that's done, we can open index.js and write some code.
We'll first get from the vision library
an ImageAnnotatorClient, which is a service that
performs Google Cloud Vision API detection
tasks over client images.
With the service, we'll use the text detection function
from the client and pass in the image bucket.
And then we'll get the text annotations
from that and the description from the annotation.
We can log this in Cloud Functions using
logger.log with the text to see that we did parse
this information properly.
[MUSIC PLAYING]
Let's then deploy the function so we
can see what happens when we add an image to Storage.
On the command line, we can do firebase deploy --only
functions.
And this takes a bit of time to upload.
Whoops, we see this error that tells us we
can't use import outside of a module.
We'll open package.json inside of our functions folder.
And at the very bottom, we're going to add type, module.
All right, let's try again to deploy functions.
And hopefully it will be successful this time.
Well, this is taking a while, isn't it?
Let's fast forward time a bit to when it's done.
Great.
We can see functions successfully deployed,
which means it's now running on the Cloud.
And we'll be able to execute the function.
We'll also be able to see this function here
in Firebase Console.
We can run our web app using npm run dev.
And when we open our app, we get this error
about needing to create an index for our Firestore query.
You might be thinking, hey, wait.
We didn't really do anything related
to Firestore yet, so why do we need to create an index?
Well, I also secretly already wrote
the needed Firestore code.
And it's basically the same as the first video's Firestore
code, except we added an isConfirmed field.
So we need to create a new index to include that.
All right, I hoped by the time I finish saying all of that,
the index would be done creating, but evidently not.
Seems like we're doing a lot of waiting in this video,
aren't we?
First, waiting for functions to upload,
and now we're waiting for this index to build.
OK, still not done.
Let's fast forward time a bit again.
Okey dokes, now on the Console we
see that our index is finally enabled.
Let's go back to our web app, and we can check the page now.
Let's log in, then add an expense
to check whether our Cloud Function gets triggered
and that the uploaded receipt gets processed.
First, let's check Storage and Firebase Console
to see that a new image has been added.
We'll go to this folder, and we see the image is there.
Great.
Next, let's go back to the Cloud Console
to see whether the information got parsed correctly.
If we go back to the Console home page
and, again, make sure that the project is the right one,
we can type in logs into the search bar.
On the top right here, the Jump to Now button
has been useful because sometimes the timeline
is a bit off.
Here, we can see everything that got logged,
and it contains information from our image.
Seems like it's working.
Next, we'll want to parse this.
But since parsing isn't trivial, and covering the cost
to parse the receipt will likely take up more time
than talking about implementing OCR with Firebase and Google
Cloud, we don't have time today to properly parse the text.
So for now we'll just hardcode the results.
With that, our function is almost done,
and we just need to add the new receipt
information to Firestore.
Before we can do so, we'll have to initialize Firebase Admin,
which is a set of server libraries that lets us interact
with Firebase from privileged environments
to perform actions, including accessing Cloud Storage buckets
and Cloud Firestore databases.
As a side note, only use the Firebase Admin SDK
from a trusted environment, like your own servers or Firebase
functions, because it bypasses security rules.
From part one of the series, we stored the address amount,
date, image bucket, items, location name,
and user ID of the receipt into Firestore.
We'll still have these, but we'll
need to add one field for this video with OCR.
Since now we want users to confirm the receipt,
we'll have a new field called isConfirmed
so that we can show unconfirmed receipts in one
section on the dashboard and confirmed receipts in another.
This way, we can ask our users to confirm their receipts
on the UI for any fields that were processed using the Google
Cloud Vision API.
And we can allow them to adjust any information that
was incorrect.
We'll need to get the user ID to put into Firestore
for our new receipt.
Since the name of the image will be the user ID/the timestamp
of when the user uploaded the image,
we'll just use some regular expressions
to extract that user ID.
We'll then create our receipt object
with all the contents of the receipt, hardcoded here
because we didn't write parser code for the text that
came back from the Google Cloud Vision API.
We'll set isConfirmed to false since we
want the user to confirm the contents.
With that, we'll add the receipt to Firestore.
Let's deploy this function again, and just like last time,
it'll take a few moments.
Once it's done, we'll go back to our app and add a new receipt.
[MUSIC PLAYING]
We'll add a new receipt, and it'll
take some time to process.
Once it's done, we can refresh the page,
and we'll see the uploaded receipt in our Need
Confirmation section.
We can click on the check mark here,
which will open this pop-up, just in case any information
needs to be adjusted.
And we can confirm once everything looks good.
Afterward, it'll be moved to the Expenses section,
and we can still edit and delete the receipt, as needed.
So there we have it.
Thanks for joining me in building
an OCR where the receipt images get automatically
read by the Google Cloud Vision API.
And we used Firebase Cloud Functions
to trigger all of this.
I hope this was helpful.
And see you next time, where we'll
see how to roll out new features safely
and confidently using Firebase Remote Config and Google
Analytics.
In the meantime, happy Firebasing.
[MUSIC PLAYING]