PRIYANKA: If you are ingesting large amount of data
for analysis or looking to simplify
the development of event driven microservices,
then you need to check out Cloud Pub/Sub.
[MUSIC PLAYING]
Welcome to Pub/Sub Made Easy.
I am Priyanka.
And in today's episode, we will dive deeper into Publishers
and see some configuration and application design choices
you will need to make when building a real world Publisher
application.
A Publisher application creates and sends messages
to a desired topic on Cloud Pub/Sub.
Cloud Pub/Sub offers at least once message delivery
and best effort ordering to existing subscribers.
While publishing messages to a topic
when using JSON over REST, the message data
must be base64 encoded.
The entire request, including one or many messages,
must be smaller than 10 MB after decoding.
The message payload must not be empty.
It must contain either a nonempty data field or at least
one attribute.
What can you include in the message itself?
Well, here's the message schema.
You can embed custom attributes as metadata in the messages.
Attributes can be text strings or byte strings.
A server generated ID unique within the topic
is returned on the successful publication of a message.
If you would like to see an example of publishing
a message, check out the previous episode.
All client libraries offer an asynchronous
API where your application does not
have to wait for the actual published request
to be complete, although synchronous or blocking
mode of operation is supported as well.
Check out the link in the description below for the API
reference documentation for your choice of programming
language and its support.
The first decision you have to make
is how to batch your messages.
You can configure the client library
to send multiple messages per request
to the service based on the size of the buffer
or time the client waits for the buffer to fill.
Batching has the obvious advantage.
The larger batch size increases message throughput,
which is the rate of messages sent per CPU.
Asynchronous client libraries try
to publish multiple batches in parallel for higher throughput
but it also comes at a cost.
It introduces latency for individual messages,
which are cued in memory until their corresponding batch is
filled and ready to be sent over the network.
To minimize latency, batching should be turned off.
This is particularly important for applications
that publish a single message as part of the request response
sequence.
However, the additional network and computing overhead
may make small batches preferable.
You'll have to tune your system.
Messages can be batched based on request size in bytes,
number of messages, and time.
You can override the default settings by defining your own.
Here's an example code in Python where
we are defining batch settings for maximum bytes, latency,
and messages.
A caveat here is that once you override one setting,
you should override them all explicitly.
Now here's how this works.
On the first message that you send,
a new batch is automatically created with a countdown.
For every subsequent message, if there is already
a valid batch that is still accepting messages,
then that batch is used to drop the new message.
The batch is published once sufficient time has elapsed,
maximum messages has reached, or maximum bytes is met.
By default, the batch setting is set 0.05 seconds.
Now what happens if publishing fails?
Publishing failures are automatically
retried in some client libraries.
Check out the linked documentation
to check the support for the language of your choice.
Because we are dealing with requests over a network,
request failure is a normal part of life and understanding
retries is essential.
Client libraries support custom retry settings.
Here's a snippet with retry error code
received from the publisher.
Here's a snippet with retried parameters
that include timeouts and delays with the default settings.
The main two parameters to understand
are the total timeout, which is how long the application will
keep retrying before giving up, and the initial timeout,
which is how soon it will start retrying.
The total time out depends on the typical availability
of your network, the memory available to your machines,
and application specific considerations.
All right.
So today we learned how to publish messages
in Cloud Pub/Sub, different options including
synchronous/asynchronous messages,
and setting up batches and custom attributes.
Join us next time to learn more about setting up Subscriber
to receive messages.
If you liked this video and would
like to see more such content, don't
forget to like and subscribe to our channel.
[MUSIC PLAYING]