Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service) are two powerful AWS services that can be used together to create scalable, reliable, and efficient messaging systems.
In this article, we will explore how to connect SNS and SQS to create a robust messaging system. We will walk through the steps required to configure and integrate these services.
After reading this article, you will have a comprehensive understanding of how to effectively utilize SNS and SQS in tandem to develop a messaging system that can distribute messages to numerous destinations, all while maintaining message persistence for adaptable, asynchronous consumption.
Introduction
AWS offers two messaging services, Amazon SNS and SQS, which enable the decoupling and scaling of microservices, distributed systems, and serverless applications.
Before we start practicing, let's take a quick look at both messaging services offered and understand which one to use for what purpose.
Brief Overview of SNS and SQS
Amazon SNS is a messaging service that is fully managed and uses a pub/sub model. It allows you to send messages to multiple subscribers or recipients all at once.
Messages are delivered to topics and clients can subscribe to those topics to receive the published messages. SNS topics can be subscribed to by a variety of services and clients, including:
Email messages (through Amazon SES)
SMS text messages
Mobile push notifications
SQS queues
AWS Lambda functions
HTTP/HTTPS endpoints
SNS is built to be scalable and highly available, making it an excellent choice for use cases such as mobile messaging, event-driven computing, and application alerts.
On the flip side, Amazon SQS is a message queuing service that is fully managed. It allows you to break apart and expand microservices and distributed systems.
The service offers a dependable, extremely available, and scalable foundation for storing and retrieving messages between distributed software components and applications. SQS is purposely created to make certain that messages are delivered with dependability and in sequence (if using FIFO queues), even if there are issues with component failures or network disruptions.
Key Differences between SNS and SQS
One of the main distinctions between Amazon SNS and Amazon SQS lies in their messaging models. SNS functions as a pub/sub messaging service that facilitates many-to-many messaging, whereas SQS operates as a queue-based messaging service intended for the decoupling and scaling of microservices and distributed systems.
What Should Be Used When?
Both services play a crucial role in constructing resilient systems, however, they offer distinct advantages.
Use SNS for the following cases:
Alerting and monitoring - you can easily connect CloudWatch alerts with an SNS topic to send real-time notifications in the event of threshold breaches for application monitoring.
Notifying customers in real-time - you can use SNS to send emails, SMS messages, and push notifications, which are compatible with iOS, Android, Fire OS, and Windows devices.
Fan-out messages to a large number of subscribers - with SNS, you can notify a large number of subscribers.
Use SQS for these cases:
Decoupling distributed systems: Using SQS queues to separate the various components of your distributed system can minimize their interdependence and enhance the overall system's dependability.
Scaling microservices: With SQS, you can easily scale your microservices by processing messages asynchronously and in parallel, without any concern for the underlying infrastructure.
Managing spikes in traffic: SQS can assist in managing sudden increases in traffic by enabling you to hold requests in a queue and handle them at a pace that your system can handle, without any requests being lost.
Ensuring reliability: SQS guarantees that your messages are received sequentially and without any loss, even if some parts fail or the network experiences disruptions.
Reasons for Connecting Both Services
By utilizing Amazon SNS and Amazon SQS in tandem, applications can receive prompt alerts for events while also storing messages in an Amazon SQS queue for future processing by other applications.
The service combination can also be used to ensure that messages are delivered in the correct order. By creating an SNS topic that's connected to a FIFO SQS queue, messages will be consumed by consumers in the same order they were published to the SNS topic.
Setting up the Environment
Let's go over the essential steps for establishing a connection between an SNS topic and an SQS queue. We will cover this process using both the AWS console and Infrastructure as Code (IaC). For IaC, we will be using Terraform as it is a declarative language that is easy to understand and execute.
If you haven't set up Terraform yet but would like to follow the article, please read our introduction article.
Creating an SNS Topic
To create a topic on the AWS console, we need to first go to the SNS service. Then, we can select Topics from the menu on the left-hand side and click on Create topic.
We'll establish a conventional subject, and message sequence is not a concern in our example.
With Terraform, let's create a new main.tf file with the following code:
provider "aws" {
region = "us-east-1"
}
resource "aws_sns_topic" "sns_to_sqs" {
name = "sns-to-sqs"
display_name = "SNS to SQS"
}
By running terraform init and afterward terraform apply, we'll create our topic.
Setting up an SQS Queue
In the second step, we'll set up our SQS queue. Let's jump to the SQS console and click on the Create queue button. We can go with the standard settings and finish the creation.
For Terraform, we'll extend our main.tf with the following code:
resource "aws_sqs_queue" "sns_to_sqs" {
name = "sns-to-sqs"
}
Let's also apply the changes so our new queue gets created.
Subscribing Our Queue to Our Fanout Topic
As native English speakers, we aim to have our queue receive all published messages from the SNS topic. To achieve this, we simply need to revisit the SNS topic and establish a fresh subscription.
Click on the Create subscription button and fill out the three requirements:
Topic ARN - the ARN of our current topic.
Protocol - select
Simple Queue Service.Endpoint - the ARN of our new queue.
Using infrastructure as code makes things even simpler because we don't have to manually manipulate ARNs, which are highly susceptible to errors. Let's continue by adding to our main.tf file.
resource "aws_sns_topic_subscription" "my_subscription" {
topic_arn = aws_sns_topic.sns_to_sqs.arn
protocol = "sqs"
endpoint = aws_sqs_queue.sns_to_sqs.arn
}
As you can see, we can refer to the resources we created earlier: the ARN of our SNS topic and the ARN of our SQS queue.
Allowing SNS to queue messages in SQS
As you are aware, AWS denies all actions by default, and we must explicitly grant them. To allow the SNS topic to add messages to our SQS queue, we should attach a resource-based policy to it.
resource "aws_sqs_queue_policy" "access_policy" {
queue_url = aws_sqs_queue.sns_to_sqs.url
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "Allow-SNS-to-Publish-to-SQS"
Effect = "Allow"
Principal = {
Service = "sns.amazonaws.com"
}
Action = "sqs:SendMessage"
Resource = aws_sqs_queue.sns_to_sqs.arn
Condition = {
ArnEquals = {
"aws:SourceArn" = aws_sns_topic.sns_to_sqs.arn
}
}
}
]
})
}
We can also do this in the AWS console, by clicking on our topics edit button and adapting the access policy.
Testing our Subscription
Our setup is now fully configured. To ensure its functionality, let's publish a message to our SNS topic and then poll the message from our SQS queue.
We can easily do this via the AWS CLI via aws sns and aws sqs.
💡 Get superpowers via auto-completions in your terminal by installing aws-shell. Not only will this provide you with immediate access to the documentation, but it will also display a list of accessible commands and their necessary and optional parameters without ever exiting the terminal.
Let's publish a new message to our topic:
aws sns publish \
--topic-arn arn:aws:sns:eu-west-1:<account-id>:sns-to-sqs \
--message "Hello, world" \
--region=eu-west-1
If it succeeds, we'll receive the message id of our published message.
{
"MessageId": "db3975fc-0eef-5e23-bcc1-295a94ec413a"
}
Let's now poll the message from our SQS topic.
aws sqs receive-message \
--queue-url "https://sqs.eu-west-1.amazonaws.com/<account-id>/sns-to-sqs" \
--max-number-of-messages 1 \
--region=eu-west-1
The response will look something like this:
{
"Messages": [
{
"MessageId": "74636b57-b7c8-43fa-84a1-de321f96dee0",
"ReceiptHandle": "AQEB9kNznje37A13Bx9xa+UUwbBXjzsNGOj1MjJsRN8Kj3x4n5RPcRPr40oPKFU/R9r/YIJB2r9DqRRHmUR6mAP7V6E1VGp0SMC8jNTJdnVC4z99iITzQSWDQounfGOZDXJJfL0i42F+X2IgSqlhjdotzktduaQPxsd2rNcP+6nqU3jsr2Pl9tGMil73LPlWfGYzktD9eStD96WZaITsT7UVtjPQ8hAPUyiBsftn+V9Oh26vnzbFyFgoa8PHaG+/tz7CsEfM/vXn05iIzWefq3ott+t4Arxun6Bmh5VZ+XwoL4wSs5V3tcyP/Zju7SUjs28uJs8t9A5NNuWZmCcfjzgmX+JQYAoLlTVK44PlHGPxs8I1VzjUAe35YWWnx/bZs52V",
"MD5OfBody": "2461945ded46105cdf259d792fac3093",
"Body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"3080f69a-eff4-5df4-b88f-aa3c764c7641\",\n \"TopicArn\" : \"arn:aws:sns:eu-west-1:157088858309:sns-to-sqs\",\n \"Message\" : \"Hello, world\",\n \"Timestamp\" : \"2023-04-04T21:32:40.755Z\",\n \"SignatureVersion\" : \"1\",\n \"Signature\" : \"JXHQDzsvMEshEvSdLXItNSO/NmRGVVEkrsUq7oiJQvPu2MStV1qK61EjqPAXhc3skGpkwiV97/2KCq1nVjL89eahYVSVzbzd2peIdge960lJUbLQz+lnjYSKv6gFCduSqNWczS1q20ebH/O1nDnuxi6v2gLA1mW8Ms46A3hJGJNGOr974oR1Zinbu+gWr08Y35/lRATJ0UyKSC7wK8aOD7ycCwNbXx69TpkxqTqNh8COTy/wzFYhgsayXDWTfQt9IAfJDfjVEiI5h0kNRsvK9da4KjqTX5pWytgiC6420p157tnK0AncRxpL7M/zW4VIqd84UdOROiOrxj2LlNjpkQ==\",\n \"SigningCertURL\" : \"https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem\",\n \"UnsubscribeURL\" : \"https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:157088858309:sns-to-sqs:a6b5852e-1f4f-454c-a329-7dfad9e3cbdb\"\n}"
}
]
}
With the help of jq, we can directly parse the response and only access the message body.
aws sqs receive-message \
--queue-url "https://sqs.eu-west-1.amazonaws.com/157088858309/sns-to-sqs" \
--max-number-of-messages 1 \
--region=eu-west-1 \
| jq '.Messages[].Body | fromjson | .Message'
"Hello, world"
Great job! We were able to successfully publish a message to SNS, which was then forwarded to SQS and later consumed by us.
If you've followed this article via Terraform, you can easily clean up our created topic, queue, and subscription by running terraform destroy.
Conclusion
In conclusion, Amazon SNS and SQS are two powerful messaging services offered by AWS that enable the decoupling and scaling of microservices, distributed systems, and serverless applications.
While SNS is a pub/sub-messaging service that facilitates many-to-many messaging, SQS operates as a queue-based messaging service intended for the decoupling and scaling of microservices and distributed systems.
By combining these two services, applications can receive prompt alerts for events while also storing messages in an Amazon SQS queue for future, asynchronous processing by other applications or components.
Overall, both services play a crucial role in constructing resilient systems and offer distinct advantages.
In terms of pricing both services are completely usage-based, check out our pricing guide for SQS to get a more in-depth understanding of how the pricing works.