MARTIN OMANDER: Hi, Tianzi.
TIANZI CAI: Hi, Martin.
What's going on?
MARTIN OMANDER: I'm writing a ride hailing
app that matches cars with riders in real time
using Pub/Sub.
I noticed some glitches.
And I wondered if you could help me.
TIANZI CAI: A ride hailing app, that sounds super cool.
Pub/Sub has released several new features this year.
I wonder if some of them could help.
Could you tell me more about your use cases?
MARTIN OMANDER: Of course.
[MUSIC PLAYING]
MARTIN OMANDER: When a rider requests a ride,
the rider's mobile app sends an HTTP post to a Cloud Run
service called Rider Handler.
That service publishes a message to Pub/Sub.
The message is consumed by the Dispatcher service.
The Dispatcher service calls the Fleet Manager service
to find nearby drivers.
It then sends a push notification
to the nearest driver via Firebase Cloud Messaging.
When a driver accepts or declines a ride,
their mobile app calls a cloud run
service called Driver Handler.
That service publishes a message to Pub/Sub,
which is consumed by the Fleet Manager service.
Pub/Sub lets me decouple Rider Handler
from Dispatcher and Driver Handler from Fleet Manager.
It also lets me add more message subscribers in the future,
for example, for reporting.
TIANZI CAI: Thank you for showing me how the app works.
I see how Pub/Sub fits in now.
Let's see the issues you face.
MARTIN OMANDER: Yeah, the first issue I'm facing
is caused by malformed GPS data from riders' phones.
It's still a little bit of a mystery to me how this happens,
but when a dispatcher receives Pub/Sub messages with bad GPS
data, it throws an exception and doesn't
acknowledge the message.
When there are many such bad messages,
new ride requests don't make it to the dispatcher.
TIANZI CAI: Oh, no.
It sounds like you've got a growing
backlog of undeliverable messages in your Pub/Sub topic.
Could you show me the graphs for unacked message
count and oldest unacked message?
MARTIN OMANDER: Yeah.
I went into the Pub/Sub area of Cloud Console
and I took these screenshots.
TIANZI CAI: Ah.
This confirms my suspicion.
Let me explain.
A small backlog is harmless.
But a huge backlog could cause all your messages
to be stuck because Pub/Sub always
tries to deliver the oldest messages first.
You never want a situation where the number
of undelivered messages grows with the age of the oldest
message.
It looks like you have that kind of backup.
And it's causing new messages to be
stuck behind the old messages.
MARTIN OMANDER: Oh.
I see.
What should I do?
TIANZI CAI: You should use a [INAUDIBLE] cue.
You could tell Pub/Sub to forward
the undeliverable messages to a different topic,
also known as a dead letter topic,
after a certain number of failed delivery attempts.
MARTIN OMANDER: Oh, really?
Can I update an existing subscription
to send bad messages to a dead letter queue?
TIANZI CAI: Absolutely.
Let's go to Cloud Console to do that.
MARTIN OMANDER: OK.
TIANZI CAI: Let's go to your subscription dispatcher-sub.
MARTIN OMANDER: All right.
TIANZI CAI: Edit it.
MARTIN OMANDER: Edit.
Yep.
TIANZI CAI: We want to scroll down to Dead Lettering
and enable Dead Lettering.
MARTIN OMANDER: This checkbox here?
TIANZI CAI: That's right.
MARTIN OMANDER: Dead Letter Topic.
TIANZI CAI: You want to create--
MARTIN OMANDER: You said--
TIANZI CAI: Yes.
Create a Dead Letter topic to forward the messages.
MARTIN OMANDER: All right.
So I'll call that riders-dead-letter-topic.
TIANZI CAI: You also want to attach a default subscription
to your dead letter topic.
MARTIN OMANDER: OK.
I'll call that riders-dead-letter-subscription.
TIANZI CAI: Sounds great.
How many times do you want Pub/Sub to try before it
forwards the bad messages?
MARTIN OMANDER: So this is how many times
it will have to fail in dispatcher before it
comes into this subscription, is that right?
TIANZI CAI: That's correct.
MARTIN OMANDER: OK.
Maybe 10 times.
If it doesn't work after 10, it will never work.
TIANZI CAI: Now you can click Update.
MARTIN OMANDER: All right.
TIANZI CAI: And you want to go to Dead Lettering.
MARTIN OMANDER: Oh, yeah.
There's an orange triangle there.
Oh.
TIANZI CAI: This is to grant Pub/Sub the publisher
and subscriber role so that it could forward
the messages on your behalf.
MARTIN OMANDER: Oh, I see.
So I just granted these roles.
TIANZI CAI: Yes.
You will expect messages that exceeds
the maximum number of delivery attempts
to be forwarded to your Dead Letter topic.
MARTIN OMANDER: Excellent.
I should ask for someone who knows more about GPS data
to write a service that subscribes to that Dead Letter
subscription and do a full diagnosis.
TIANZI CAI: They will have all the bad data in one place.
I think they will appreciate that.
MARTIN OMANDER: The second issue I have is mistakenly
sending cars to pick up a rider who has canceled.
First, the rider sends a request ride message.
Next, the rider sends a canceled ride message
if they want to cancel the ride.
And for some reason, the canceled ride message sometimes
arrives first.
When it does, the dispatcher doesn't know about this ride
yet, so it can't cancel it.
And then later the request ride message comes in.
And the dispatcher then creates the rides
and finds a driver for this canceled ride.
TIANZI CAI: This sounds like messages were definitely
arriving out of order.
But Pub/Sub Message Ordering could help.
MARTIN OMANDER: Oh, excellent.
Please tell me more.
TIANZI CAI: I imagine requests matter per rider.
When these requests are published to Pub/Sub,
if you attach the rider's ID as an ordering key,
then these messages will be delivered in the same order
they are received by Pub/Sub.
The request ride message will arrive in a dispatcher service
first, followed by the cancel ride message.
MARTIN OMANDER: Oh, and this would make sure
that the dispatcher is always using
the latest, correct information about each ride, right?
TIANZI CAI: Exactly.
MARTIN OMANDER: I want to give it a try.
How do I set it up, Tianzi?
TIANZI CAI: To set it up, let's go to Cloud Console again.
MARTIN OMANDER: All right.
I'm here.
I go into Subscriptions.
TIANZI CAI: Let's recreate a dispatcher sub
so that it has ordering enabled.
MARTIN OMANDER: So I start by deleting it.
Is that right?
TIANZI CAI: That's right.
MARTIN OMANDER: All right.
Delete.
Then I'll create a new or replacement for it.
So I'll call it dispatcher-sub-ordered
because--
TIANZI CAI: It sounds great.
MARTIN OMANDER: Yeah.
And it's the rider's topic.
It's push because we're using Cloud Run on the other side.
And this is my Cloud Run end point.
And I'm just recreating what I'd done
on the previous subscription.
So I enabled authentication for it.
I set up an account for that.
And then there was something I should do differently.
Ordering?
TIANZI CAI: Yes.
MARTIN OMANDER: Is this it?
TIANZI CAI: Yes.
You want to enable message ordering.
MARTIN OMANDER: Very good.
And that's it?
TIANZI CAI: That's it.
You can click Create.
MARTIN OMANDER: All right.
TIANZI CAI: And in the meantime, I'm
going to publish a request ride message
from Bobcat followed by a cancel ride message from Bobcat.
MARTIN OMANDER: All right.
I'm selecting the log for dispatcher service here.
And there I see a request ride from Bobcat.
And let's see what is this.
TIANZI CAI: Yeah.
MARTIN OMANDER: Yeah.
And the cancel ride from Bobcat.
Our drivers will appreciate this.
They came in in the right order.
TIANZI CAI: That's great to hear.
MARTIN OMANDER: Cool.
So with Dead Letter queues and message ordering enabled,
I'm eager to test out a new version
of the algorithm in the dispatcher that
matches riders and drivers.
TIANZI CAI: Definitely.
It will be cool to compare the algorithms side by side.
I'm thinking, if you can replay the messages,
you can compare the new algorithm
with the old algorithm.
MARTIN OMANDER: Cool.
That sounds fantastic.
But, Tia, can you explain what replay means?
TIANZI CAI: To replay messages, you
need to create a Snapshot on a given subscription.
Another way to think about a snapshot
is to consider it a recording that you start.
MARTIN OMANDER: OK.
I have a subscription which hits the production dispatcher.
We just saw it.
If I want to replay messages to my test dispatcher
and compare it with the one in production, can I do that?
TIANZI CAI: Absolutely.
Let's head over to Cloud Console to do that.
MARTIN OMANDER: All right.
Let me just first grab the address
for the test dispatcher.
All right.
So I grabbed the URL for the test dispatcher.
And I'm going to Pub/Sub subscriptions now.
TIANZI CAI: Yes.
Let's go to dispatcher-sub-ordered.
MARTIN OMANDER: All right.
Dispatcher-sub-ordered.
I'm going into that one.
TIANZI CAI: Let's go into the subscription
and to create a snapshot.
MARTIN OMANDER: OK.
TIANZI CAI: Yep.
MARTIN OMANDER: Great snapshot.
All right.
TIANZI CAI: Let's give it a name.
MARTIN OMANDER: All right.
Maybe I'll call it begin-test.
All right.
TIANZI CAI: And now we want to go ahead and create
a new subscription using this Snapshot.
So go to Subscriptions.
MARTIN OMANDER: OK.
Create new subscription you said?
TIANZI CAI: Yes.
MARTIN OMANDER: All right.
I think I might call this test-dispatcher-subscription.
And this will be the same topic as the previous one.
Still push because I'm pushing to a Cloud Run service.
Here's the URL for my test to dispatcher.
I've set up a service account.
They can publish.
It's allowed to publish.
And allowed to call the Cloud Run service.
And, sorry, what more should I do here?
TIANZI CAI: You want it to retain acknowledged messages.
MARTIN OMANDER: Oh, there it is.
Retain acknowledged messages.
Yeah.
TIANZI CAI: And go ahead and create the subscription.
MARTIN OMANDER: All right.
TIANZI CAI: Very quickly, Martin, could we
go back to dispatcher-sub-ordered
and also retain acknowledged messages
in dispatcher-sub-ordered as well.
MARTIN OMANDER: Oh.
Yes, of course.
Good, good thinking.
Sorry, was it retain acknowledged messages?
TIANZI CAI: That's right.
MARTIN OMANDER: OK.
TIANZI CAI: I'm going to go ahead and publish
some messages.
In the meanwhile, you can go to your test subscription
and seek to the snapshot that you just created.
MARTIN OMANDER: Oh.
OK.
Test dispatcher sub.
And replay messages.
And Seek to a snapshot, you said.
TIANZI CAI: Yes.
MARTIN OMANDER: Picking a test.
OK.
Should I hit seek?
TIANZI CAI: Yes.
MARTIN OMANDER: Very good.
Replaying messages.
TIANZI CAI: And now we can look at your logs
to see some messages that are just published.
MARTIN OMANDER: All right.
I'm in the logs.
I'm going to filter so that I only see the test dispatcher.
Only see the messages from the test dispatcher.
Let's see here.
So test dispatcher received a request ride from Starfish.
And let's see if there were any more.
I'll run this again.
Test dispatcher received a Pub/Sub message request ride
from Penguin.
A request ride from Gopher.
And finally, there was a request ride from Starfish.
Cool.
TIANZI CAI: That's awesome.
MARTIN OMANDER: So any point, I can
go back and replay these and send them over and over
to the test dispatcher?
TIANZI CAI: The snapshot will be there for seven days.
So you can go back and replay as many times as you want.
MARTIN OMANDER: Cool.
So I see that I can seek to a snapshot.
And the other option I see here is to seek to a point in time.
This is a really powerful feeling,
being able to jump around in time.
TIANZI CAI: Right?
It's empowering.
But we should probably also want to do with caution
by reading the docs carefully.
MARTIN OMANDER: Yeah, I can't agree more.
We time travelers know that we must always read
the documentation carefully.
I feel like I've learned a lot today, Tianzi, dead lettering,
ordering, snapshot, and seek.
Thank you for showing me these Pub/Sub features.
TIANZI CAI: My pleasure, Martin.
Let me know how your ride hailing app does.
[MUSIC PLAYING]