MARTIN OMANDER: Hi, Karolina.
Welcome to "Serverless Expeditions."
KAROLINA NETOLICKÁ: Thank you, Martin.
MARTIN OMANDER: You are a Cloud Run product manager,
and I hear you've worked in a way
to make Cloud Run even more useful?
KAROLINA NETOLICKÁ: That's right.
We've released Cloud Run Jobs, which makes it easier
to use Cloud Run for unattended batch jobs,
like database migrations, report generation, and so on.
MARTIN OMANDER: Can't wait to hear more about it.
[MUSIC PLAYING]
KAROLINA NETOLICKÁ: Developers like Cloud Run because it lets
you run containers in a serverless way.
Containers are good because they make
it easy to package up your application and the libraries
it needs.
MARTIN OMANDER: And serverless is
good because it means your cloud provider worries about scaling
up and down, so you don't have to, right?
KAROLINA NETOLICKÁ: Right.
So far, you've been able to trigger Cloud Run executions
on HTTP calls or events, like a file
being added to Cloud Storage.
But there are some workloads that
aren't well-suited for that.
MARTIN OMANDER: What kind of workloads?
KAROLINA NETOLICKÁ: Well, you might have a database migration
or cleanup work, an export or import process,
reports to generate, and so on.
These jobs need to run unattended without an HTTP
request from a user.
Also, they should run to completion.
MARTIN OMANDER: Oh, right.
Back in my sysadmin days, I had many things
that weren't web servers, like maintenance scripts.
KAROLINA NETOLICKÁ: Exactly.
These jobs often take longer than it
takes to process an HTTP request or an event, typically
minutes instead of seconds.
MARTIN OMANDER: Right.
I remember setting up a virtual machine
to run some of those scripts.
KAROLINA NETOLICKÁ: Many organizations do that,
but it's inefficient.
You have to manage and configure the virtual machine,
and you have to remember to take it down once your job finishes
or pay for it 24/7.
Also, it can't easily scale up to run really large jobs.
MARTIN OMANDER: Yeah, I only used a virtual machine
for scripts like that because there was no serverless way
of doing it back then.
KAROLINA NETOLICKÁ: Well, now there is.
It's called Cloud Run Jobs.
It's still serverless.
It's still running containers.
But it works better for unattended workloads that
aren't triggered by HTTP calls.
MARTIN OMANDER: That sounds good.
But I suspect it's still not completely clear to everyone
how it works.
We are developers, and we like to see code and a demo.
KAROLINA NETOLICKÁ: Well, let's hope the demo gods are with me.
Imagine a company receives invoices from its vendors
as PDF files.
Here are a few examples of what they might look like.
Every night, they want to extract the vendor names
and totals from those PDFs so their finance team can review
and approve them the next day.
MARTIN OMANDER: Got it.
I've built nightly jobs like that before.
KAROLINA NETOLICKÁ: Right.
Now here's the code.
This part reads the invoice files
from Google Cloud Storage.
This part calls the Document AI API to extract vendor names
and totals from each invoice.
This part writes the invoice data to Firestore.
And, finally, this part moves the processed invoices
to another folder so they won't get picked up
next time the job runs.
MARTIN OMANDER: And, once the job runs,
how will we see the parsed invoice data?
KAROLINA NETOLICKÁ: We also have this simple web page that reads
the parsed invoice data from Firestore and displays it
to the user.
MARTIN OMANDER: Got it.
Would we deploy the invoice parsing job
like a regular Cloud Run service?
KAROLINA NETOLICKÁ: It's similar.
You first build your container with Google Cloud Build
and then create your job with gcloud run jobs create.
Let me create it right now.
MARTIN OMANDER: OK.
This code you showed me does not listen to HTTP calls.
So how do we get it to run?
KAROLINA NETOLICKÁ: Well, you can run it from the Google
Cloud Console or the command line.
It's similar to running a shell script on your computer,
but it runs in the cloud and has access to all
your Google Cloud resources.
Let me do that now.
There it goes.
MARTIN OMANDER: Mhm.
Looks like something ran.
Can we see the output of the job?
KAROLINA NETOLICKÁ: Let's have a look.
I'll go to the simple web page that
displays the parsed details from the invoices.
Looks like it captured the invoice data.
MARTIN OMANDER: Nice.
But, Karolina, what if I want to run a nightly job when I'm
not at my keyboard?
KAROLINA NETOLICKÁ: Anything that can call the Cloud Run
admin API can start the job, including Cloud Scheduler.
Let's use Cloud Scheduler to make our job run every night.
I go to Cloud Scheduler.
Create a new job.
Give it a name and a region to run in.
Set a schedule for it to run using crontab notation
and a time zone.
Click Continue.
Target type, HTTP.
Enter the Cloud Run admin API endpoint to run your job.
You can find that in the documentation.
And keep HTTP method as POST.
Now we have to give Cloud Scheduler
permissions to start the job.
I click here to pick OAuth token.
And then I select which Service account Scheduler
should use to start the job.
I know this one has sufficient permission.
You need a service account that has the Cloud Run invoker
permission on the job.
Then I click Create.
Now the job will be run nightly.
You won't have to be at your keyboard at midnight, Martin.
MARTIN OMANDER: That's a relief.
The code you showed me processed one invoice at a time
in series.
If we have a thousand invoices, that would take a long time.
Can we speed it up somehow?
KAROLINA NETOLICKÁ: Yes.
You can tell Could Run Jobs to start several parallel tasks.
Let's say you have a thousand invoices
and you start 10 parallel tasks.
Your code can detect what number task it is--
for example, 4 out of 10.
Then your code can process its part of the total work.
For task number 4, that would be invoices 301 to 400.
MARTIN OMANDER: Sounds very useful for a large jobs,
like big old data migration jobs that need
to process millions of records.
Do you have a code example for that?
KAROLINA NETOLICKÁ: Let's save that for another video, Martin.
Also, it's all in the documentation.
The link is in the description.
MARTIN OMANDER: Very good.
Thanks for showing us, Karolina.
And thank you, everyone, for watching.
If you have questions for Karolina or me,
let us know in the comments.
Also let us know if there are other serverless topics you'd
like to hear about.
Until next time.
[MUSIC PLAYING]