This article gives you an overview of the AWS CLI and which operations you can do with it for AWS's awesome Simple Queue Service (SQS in the following).
We start with some basic introductions to SQS, CLIs, and the AWS CLI. Then we'll focus on operations specifically for SQS.
Let's go 🚀
What is SQS?
Amazon SQS is short for AWS's Simple Queue Service. SQS is a message queue that can be polled from different types of consumers. Consumers typically are:
Lambda functions
EC2 instances
ECS cluster
SQS is a poll system but acts like a push system. Consumers are constantly polling for new messages, working on these messages, and deleting them afterward.
Fun Fact: SQS is the oldest service of AWS 👆🏽
What is a CLI?
A CLI is short for Command Line Interface. If you are a developer you surely have used CLIs everywhere. Normally, they are normal programs that take arguments via the command line.
CLIs can be created with different programming languages, nowadays many CLIs are written in either Node.js or Python.
CLIs are extremely powerful tools, especially in combination with existing bash programs like grep, sed, or jq.
While there are tons of CLIs out there this blog post handles only the AWS CLI.
AWS CLI
The AWS CLI is very popular among AWS users. Chance is that if you have any program on your PC that is interacting with AWS it uses the CLI under the hood.
The AWS CLI is also very powerful. You can interact with almost all services and do everything you can do in the console also in the CLI.
Popular examples are:
Creating S3 buckets
Creating Lambda Functions
Executing lambda functions
Launching an ECS Cluster
And much more. The AWS CLI is also often used in CI/CD clusters like Jenkins or GitHub Actions.
Configure AWS CLI
It is recommended to configure the AWS CLI before using it. There are different ways of configuring the CLI. The main goal is to have an access key and secret access key saved somewhere securely. Here is an official article by AWS on how to configure the CLI. I recommend doing it with names profiles and I show you quickly how you can do that.
Install AWS CLI
For installing the AWS CLI please follow AWS's official guidelines.
Configure Named Profiles for AWS CLI
Named profiles is a collection of different AWS configurations for typically several accounts. However, you can also just use it for one account.
Two files need to be created for your profile (all on Linux or Mac OS):
~/.aws/config-> All Configuration~/.aws/credentials-> All Secret Keys
Config File
~/.aws/config
[default]
region=us-west-2
output=json
[profile user1]
region=us-east-1
output=json
Credentials File
~/.aws/credentials
[default]
aws_access_key_id=XXX
aws_secret_access_key=XXX
[user1]
aws_access_key_id=ACCESS_KEY
aws_secret_access_key=SECRET_ACCESS_KEY
I recommend adding XXX as the default profile key to not accidentally push something on the wrong account.
Now you can use the profile by appending --profile to your commands or by exporting the variable AWS_PROFILE before every call. I won't include --profile in the examples so keep that in mind!
If you want to export the environment variable for one session you can simply do the following:
export AWS_PROFILE=user1
aws s3 ls
# Same effect as aws s3 ls --profile user1
You can test it for example with the command aws s3 ls --profile user1. You should now see all buckets in the account of user1.
CLI Autocomplete
One quick tip. Install AWS CLI autocomplete feature. With that, you can just hit TAB and you'll get recommendations for all services and operations you can do.
You can see the installation guide here.
Then you can hit the tab key after typing aws or after typing aws <SERVICE> to see all operations.
For example type aws sqs, hit tab and you see all operations you can do.
Even after hitting TAB after an operation, you see all available parameters. Believe me, this is an amazing help!
SQS Operations
Now we will look into SQS operations you can do with the CLI. I collected the most important commands and will show you what happens when you execute your commands.
Create a queue
aws sqs create-queue --queue-name myQueue1
It will return the following object (depending on the region):
{
"QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1"
}
If you check the AWS management console you will find the queue with its default parameter.
Important Parameters
--queue-name: The name of the queue
--attributes : Attributes like MessageRetentionPeriod, RedrivePolicy, Policy, FifoQueue, etc.
--tags : How to tag this resource
If you want to create a queue with different attributes you can do the following for example:
aws sqs create-queue --queue-name myCustomizedQueue.fifo --attributes VisibilityTimeout=100,FifoQueue=true
The input for the attributes can also be a .json file but here I am using the shorthand syntax.
{
"QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myCustomizedQueue.fifo"
}
This command created a FIFO queue with a visibility timeout of 100 seconds.
Listing Queues
You can list all queues in your account with the list-queues command.
aws sqs list-queues
It will return a JSON object with a QueueUrls array:
{
"QueueUrls": [
"https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myCustomizedQueue.fifo",
"https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1"
]
}
Important Parameters
--queue-name-prefix: In SQS you always search by prefix.
--max-items: Total number of items to return.
--next-token: Used for pagination
--starting-token: Where to start querying from. Also used for pagination.
Get Queue Attributes
You can query queue attributes with the get-queue-attributes command.
aws sqs get-queue-attributes --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1 --attribute-names All
Output:
{
"Attributes": {
"QueueArn": "arn:aws:sqs:REGION:ACCOUNT_NUMBER:myQueue1",
"ApproximateNumberOfMessages": "0",
"ApproximateNumberOfMessagesNotVisible": "0",
"ApproximateNumberOfMessagesDelayed": "0",
"CreatedTimestamp": "1650118025",
"LastModifiedTimestamp": "1650118025",
"VisibilityTimeout": "30",
"MaximumMessageSize": "262144",
"MessageRetentionPeriod": "345600",
"DelaySeconds": "0",
"ReceiveMessageWaitTimeSeconds": "0",
"SqsManagedSseEnabled": "false"
}
}
SQS works almost every time from its URL. Unlike other services that use the ARN (Amazon Resource Name) more often.
Getting the queue attributes is important to see how you configured your queue.
Important Parameters
--queue-url: Identify from which queue to get attributes from
--attribute-names: To get all attributes to define All
Get Queue URL
You will often need the queue URL. The command for that is get-queue-url. You can get the URL by its queue name.
aws sqs get-queue-url --queue-name myQueue1
This will return a JSON object with the URL
{
"QueueUrl": "https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1"
}
Send a message to the queue
Now it gets more interesting. Sending a message to a queue is one of the core operations in SQS.
Sending a message means the message will be queued and a consumer can pick it up. It depends on several parameters and the configuration of the consumers on how fast it will be picked up.
You can send a message with the send-message command.
aws sqs send-message --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1 --message-body test
It will return the MD5 hash of the message body (of the body, not attributes) and a message ID.
{
"MD5OfMessageBody": "0096e740711f375cbdc1a47ac6864b49",
"MessageId": "a104a85f-7ca4-483e-9f10-9e82a7ce7e1f"
}
Your message body can be:
XML
JSON
Plain text
Important Parameters
--queue-url: URL to send the message to
--message-body: The actual message to send
--delay-seconds: How long to delay till the message is sent. Between 0 and 900 seconds
If you have more messages to send you can also send them in batches with the send-message-batch command.
FIFO Queue
If you send messages to a FIFO (First-in, First-out) queue remember to include the following attributes:
--message-deduplication-id: What is a duplicate and what is not (necessary if ContentBasedDeduplication is false)
--message-group-id: Which message group does the message belong to
Receiving Messages from a Queue
For consumers, it is needed to poll messages. With AWS many services can be used as an event source mapping (e.g. lambda) and you don't need to take care of it.
In case you have a custom system you need to poll the messages.
With the receive-message command this is easily possible.
If we want to get our message from the send-message command we simply need to call the receive-message command with the queue URL.
aws sqs receive-message --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1
This will give us the following result:
{
"Messages": [
{
"MessageId": "a104a85f-7ca4-483e-9f10-9e82a7ce7e1f",
"ReceiptHandle": "AQEBmPvjM/hX33ri2SP8sxf8ZqmEAmuE1BgQgHi6eQ+RZn1qm0DjrimFJgg1GuMyGLKruSFuGRY9BS6u5F83yr/OujIcttjV8ywuTzhNYvVJf2TJ0Y2tmjEY3pGk14eIgjDI22cqSuj080hy7aLdx3BR8XdoEJyE2RTs+JPNcvmd6iO03CS0A8ElpZxIqIusdsP4+1DA5PHFvNgM42b2x+WoF7Ue/irr7Vol0Ut0FPMtIq3osmVlrD4iFRjzwBMaPWZuQA/sQyE9AOKfg9bcnptQeSThdraOfI+Qdfozu/vjGzKGgJjxUU0kdr6PvQg4cc0UmbYgkECJu+OL5EIKu2j9ptqkX2ezq7EA1+5tEi3HYHtohKGeeBsG3VBpEQ0BVbUzqJXAfmbDX3Qug9S9ewT1zg==",
"MD5OfBody": "0096e740711f375cbdc1a47ac6864b49",
"Body": "test"
}
]
}
Import Parameter
--queue-url: URL to get the messages from
--attribute-names: Which attributes to get additional (e.g. All)
--message-attribute-names: The maximum number of messages to get
Receive messages in a loop
If you're building a system for SQS you don't want to manually execute the receive-message command all the time. You want to have a loop to constantly get messages.
For that you can use the following command:
while sleep 1; do aws sqs receive-message --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1; done
With that, you will get the messages constantly and can start working on them.
Delete a message
After working on a message you need to delete the message. If you don't remove it it will be picked up multiple times from another consumer.
You can do that by using the delete-message command.
aws sqs delete-message --queue-url https://sqs.eu-central-1.amazonaws.com/661734112387/myQueue1 --receipt-handle AQEBmPvjM/hX33ri2SP8sxf8ZqmEAmuE1BgQgHi6eQ+RZn1qm0DjrimFJgg1GuMyGLKruSFuGRY9BS6u5F83yr/OujIcttjV8ywuTzhNYvVJf2TJ0Y2tmjEY3pGk14eIgjDI22cqSuj080hy7aLdx3BR8XdoEJyE2RTs+JPNcvmd6iO03CS0A8ElpZxIqIusdsP4+1DA5PHFvNgM42b2x+WoF7Ue/irr7Vol0Ut0FPMtIq3osmVlrD4iFRjzwBMaPWZuQA/sQyE9AOKfg9bcnptQeSThdraOfI+Qdfozu/vjGzKGgJjxUU0kdr6PvQg4cc0UmbYgkECJu+OL5EIKu2j9ptqkX2ezq7EA1+5tEi3HYHtohKGeeBsG3VBpEQ0BVbUzqJXAfmbDX3Qug9S9ewT1zg==
To remove a message you need the queue-url and the receipt-handle. You get the handle together with the message.
Important Parameters
--queue-URL: URL of the queue
--receipt-handle: Identifier of the message to delete
This will delete our sent message from earlier.
Purge a Queue
In case you want to purge a whole queue, this is also possible via the CLI. You can use the purge-queue command.
aws sqs purge-queue https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1
This will remove all messages from the queue.
Important Parameters
--queue-url: URL of the queue
Delete a Queue
Finally, we can also remove queues with the delete-queue command.
aws sqs remove-queue --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myQueue1
aws sqs remove-queue --queue-url https://sqs.REGION.amazonaws.com/ACCOUNT_NUMBER/myCustomizedQueue.fifo
After executing both of these we will see that all queues are gone.
Related Reads
If you found the AWS SQS CLI guide useful, you might also be interested in these articles: