.NET on AWS – Queuing with SQS (Part-1)

Introduction

Queuing mechanism is very common for software applications. Its very handy in offloading the tasks from main application to background processes and typically used in building decoupled distributed systems.

Amazon SQS works on a process of clients sending messages to the queue and the consumers polling messages from the queue and processing those.

Typical use cases include tracking users activity, scheduling tasks, decoupling applications, background processing and others.

Background Information

I’ve previously written following two posts

These posts resulted in a REST application, which allows CRUD operations for Note entity and uses Amazon S3 for application data storage. Following is a simple diagram of that solution

We’ll be using same .NET solution, some of AWS resource setup and application source-code, available on git repository as a baseline.

For our current application requirements, we’ll simply built on top of this baseline.

Lets see what are the message queuing requirements.

Message Queueing Requirements

Following are very simple message queuing requirements:

“Whenever a user perform an action (e.g add new, update, delete, etc. operation) on Note entity via web-api, web application (publisher) will publish a message to SQS queue containing data about the action.

A second application (.NET6 worker service/consumer) will listen to the SQS queue and process these messages. Here, application will simply log out the event data, but processing can be anything such as storing data in the database or send an email etc.”.

Here is a simple diagram of how our solution will look like

This is a very simple example, but it will help us to learn the basics of sending and receiving messages with SQS.

Ok, let learn few basics of Amazon Simple Queue Service (SQS) next.

Simple Queue Service (SQS)

Fully managed simple queuing service, useful for decoupling components.
Highly scalable.
Typical usage in microservices/distributed applications.
Easy to get started.
2 types of messages queues
- Standard queues – nearly unlimited throughput, ordering not guaranteed, 0.40$/million requests.
- FIFO queues – up to 3000/sec throughput, Ordering, 0.50$/million requests.

Use standard-queues, when throughput is important. Use FIFO-queues, when ordering is important. If you need both, then SQS may not fit your use case (Kinesis can be considered, but its story for another day).

For more information about AWS SQS features and limitations, you can check the official page.

Provisioning SQS Queue

We need a message queue, where web-api (publisher) will publish the message and worker-service (consumer) will listen for messages. For our demo, we’ll create a Standard Queue.

We can create an SQS queue from Amazon SQS Dashboard (Web Console) as shown below

As you can see that I’ve selected Standard type and give queue a name (notesapp-events).

For queue configuration section, we can keep the defaults and for now only changed one setting (Receive message wait time) as shown below

Sending message is straight forward, client just send message to queue and moves on.

But pooling (consuming) is a different story. Situations like multiple consumers, processing issues etc. AWS has solutions for these:

Delivery delay (DelaySeconds):

Amount of time to delay the visibility of an incoming message.

When a message comes into the queue, it may or may not be visible to consumers immediately, depending on the Delivery delay attribute.

This attribute will delay the visibility of a message in the queue for a given time when the message initially arrives in the queue.

Once a message is visible and a consumer reads it, another attribute comes into play i.e. Visibility timeout.

Visibility timeout

This is amount of time to make message invisible after it has been read by a consumer.

Once a message is read, it is considered in-flight. This means, the message is still in the queue, but its not visible to consumers.

If the VisibilityTimeOut runs out, then the message becomes visible to consumers again and they can read it, which start the VisibilityTimeOut again.

A consumer should delete a message once they finish processing the message.

The VisibilityTimeOut gives consumers time to process message and then delete them.

Receive message wait time (SQS Long Polling)

Another default value you can set on your queue is “how long consumer should wait when they are polling for messages“.

This value can be set up to 20 seconds.

If any messages are available or come available while the consumer is waiting, the consumer will immediately return with the new message.

Long polling can be useful, because you’re charged for SQS by each request made (i.e. read/write message).

Both the Delivery delay and Visibility timeout can have default values set on the queue, but each message added on the queue can override those values.

So, by using a balance of long pooling and how frequently you read messages, you should try to strike a good balance that makes it economically feasible to use SQS for messaging and also high performant.

by they way, you can click info link, next to these settings, to find more information about these.

There are few other sections on create queue page, such as encryption, Dead letter queue etc. but, we’ll not make changes to any of those and simply clicked Create button and queue is created as shown below

Application Permissions for the Queue

Before we can send and receive messages, we need to give our application permission to access the queue. There are multiple ways to achieve this. However, we’ll simply update the custom managed policy as follows, we setup in one of our earlier posts.

.NET Solution

As mentioned earlier that we’ll be using an existing solution. You can get the source-code from this git repository. We’ll be updating this solution to work with SQS.

Lets see again solution components for SQS part

We already have the web application which will act as a publisher of the message. However, we’ll create a .NET worker-service to work as consumer in our application.

Following screenshot shows that I’ve added a worker-service to our solution

So, we have added a service to our solution. It is not yet configured to interact with SQS, but we will be doing that in later steps.

Adding AWSSDK.SQS Nuget Package to the Solution

To interact with SQS, we will install AWSSDK.SQS nuget package to both web-api and worker-service application as shown below

Also, add Newtonsoft.Json package to the solution. It will help us to work with JSON format in our application.

At this point we have done the basic setup for our application requirements. However, it is far from done. We still have to code the publisher and consumer parts and some wiring between these application and AWS SQS queue. However, we’ll stop here and continue with those parts in upcoming posts.

Summary

In this post, we learned about Amazon SQS which provides a fully managed, highly available and robust queuing solution for building decoupled distributed applications.

We started with application requirements and created an SQS queue using amazon web console and learned few of its configurations. We also allowed access to the queue by updating the IAM policy.

We setup our .NET solution and installed AWSSDK.SQS nuget package which we will use in upcoming post to interact with SQS.

Let me know if you have some questions or comments. Till next time, happy coding.