Welcome to another episode of 'Back to Basics'.
I am Sid, and today we will talk about
using fan out patterns in distributed microservice architectures.
Let's start by talking about what this term fan out means.
Fan-out is a messaging pattern
where a piece of information or a message is distributed
or 'fanned out' to multiple destinations in parallel.
The idea is that each of these destinations can work
and process this message in parallel.
One way to implement the messaging pattern
is to use publisher/subscriber, or the pub/sub model.
In the pub/sub model, we define a topic
which is a logical access point enabling message communication.
A publisher simply sends the message to the topic.
This message is then immediately fanned out
or pushed out to all the subscribers of this topic.
This message communication is completely decoupled and asynchronous.
Each component can operate and scale individually
without having any strict dependencies with any other components.
The publisher doesn't need to know who is using the information that it is broadcasting.
And the subscribers don't need to know where the message comes from.
If you were to model this using stateful point-to-point communication,
every publisher will have to establish and track connections with every other subscriber.
Additionally, failure handling and retry logic will have to be handled by you in your code.
Now, imagine doing this at scale with tens or hundreds of microservices.
This would make your application extremely coupled and convoluted,
and this is exactly what you don't want to do when building modern applications.
Now, the best way to build pub/sub fan out messaging on AWS
is to use Amazon Simple Notification Service (Amazon SNS).
Amazon SNS is a fully managed reliable and secure pub/sub messaging service.
You can send any number of messages at any time to SNS.
Additionally, failure handling and retry logic are built into the service.
So, all you need to do in your code is send the message to SNS,
and SNS takes care of all the complexity involved with sending this message
at scale to all your subscribers.
Now, let's look at an end-to-end architecture
that leverages this pub/sub fan out pattern.
Our set-up consists of a distributed application that handles order processing.
First, a customer, using a web or mobile application,
places a successful order.
The web app sends this request to an API Gateway endpoint in AWS.
This endpoint is the “front door” of our application.
API Gateway handles all the tasks involved in accepting and processing
up to hundreds of thousands of concurrent API requests
including traffic management, authorization, access control,
throttling, and monitoring.
API Gateway sends the order request to the first microservice,
the acknowledgment microservice.
This microservice leverages AWS Lambda for compute.
This microservice does three things.
First, it verifies the request, generates the confirmation ID,
and posts that in a durable database like DynamoDB.
Given the order detail has been securely and durably stored,
Lambda sends a confirmation message back to API Gateway.
Finally, it creates a message which will be sent downstream to different microservices.
We have the notification service to send email and SMS notifications to customers.
The shipment processing microservice to initiate the shipping workflows.
And Data Lake ingest microservice to push the order details
in a data lake for analytics and machine learning.
One thing I've seen people try to do,
is to use SQS queues to send messages between microservices.
That's a totally valid pattern, and allows for efficient batch processing
where the microservices can consume messages from the queues in batch
at a pace that works for them.
But in a fan out pattern, the same message
has to be consumed by multiple microservices simultaneously,
and that is not possible with queues.
So, what you can do is build a hyper design, leveraging both SNS and SQS.
You can put individual SQS queues in front of microservices,
and use SNS to fan out messages to these queues.
Once the data reaches the queues,
AWS Lambda service automatically polls the queues,
extracts the messages in batch,
and invokes Lambda functions to process them.
Now, a distributed system not every message is required to be sent to every microservice.
Often there are scenarios where you conditionally want to forward a message
based on an attribute in the message.
You can achieve this using the SNS filter policy feature.
Let's say you had a different plan for processing digital orders.
Using SNS filter policy, you can route
the right order type to the right processing pipeline.
Offloading this message filtering capability to SNS
lets you keep your application code simple,
while letting Amazon SNS do all the heavy lifting.
In this episode, we explored how the fan out patterns with Amazon SNS
can enable asynchronous message communication
when building distributed microservice architectures.
Check out the links below for more details.
See you next time.