I want to run an unattended, long running batch job in the cloud
without having to configure servers.
You should try out Cloud Run Jobs!
So, Karolina, you are a product manager for Cloud Run?
That's right.
Recently, I've been focusing on making it easier to run unattended
jobs on Cloud Run without having to set up or manage servers.
Perfect.
I have code here that should run every night.
It reads a large text file and it does some lookups
for each line and then inserts the values in a database.
But it takes several hours to run.
Can I use Cloud Run for that? Yes, you can.
We recently increased the maximum length of jobs
so they can run for up to 24 hours.
And it's simple to deploy jobs because you don't have to set up any servers.
Also, you're only built for the time that the job actually runs.
Sounds great, Carolina.
Now I've created a Google Cloud project already.
Would you help me deploy my code as a Cloud Run job?
Sure.
So first run “gcloud run jobs deploy”.
Set “task-timeout” to 24 hours and specify where your source code is.
If you're happy with the default
timeout of 10 minutes, you can skip the “task-timeout” parameter.
Okay.
The job is being deployed and now
looks like it's done.
And if I go to the Cloud Console and then Cloud
Run, then Jobs, I see my newly deployed job
and I can even click here to run it.
Very good.
So, Karolina, what if I want to schedule this to run
unattended in the middle of the night?
Then you'd click Triggers and “Add scheduler trigger”.
Very good.
I want my job to run every midnight UTC
so I enter the crontab expression for that here.
Then I click Create.
All done.
So this is great.
I can run my job that takes many hours.
What if my job takes more than 24 hours?
Well, you could parallelize your job so that it completes faster.
By default, your job runs as a single unit.
But if you set the tasks parameter to 100 for example,
it'll start hundred parallel instances of your code.
So my code would run 100 parallel tasks, each doing 1% of the work.
How would each task figure out
which records, which 1%, each would process?
To help with that, each task is aware
of its task index and the total number of tasks.
And based on that, your code can calculate
which database records each task should process.
Excellent, Karolina! Let's wrap this up.
What are the takeaways?
You can run your unattended jobs like data conversions
or clean ups in Cloud Run jobs.
You won't need to set up or maintain any servers
and you only pay for the time your job is actually running.
And now your jobs may run for up to 24 hours.
But just as before, you can easily run your code as multiple parallel tasks.
Thanks for showing me this, Karolina!
Thanks for having me, Martin.
And thank you, everyone, for watching.
If you have any questions for Karolina or me, please post them below.
Also, let me know if there are any other serverless topics
you'd like to hear about in future episodes.
I read every single comment.
Until next time!