AWS SQS vs SNS vs Eventbridge – When to Use What?

What’s the difference between SQS, SNS, and Eventbridge? More importantly, when should you use what?

SQS, SNS, and Eventbridge are three message orchestration services offered through AWS. Although having somewhat similar names, each of these services provides very different functionalities.

In this article, I’m going to explain to you what each of these technologies are and when its appropriate to use them in your AWS application. I’m first going to talk about each service independently, followed by comparing and contrasting the features amongst each service.

By the end of this article, you’ll have a solid understanding of SQS, SNS and Eventbridge.

So let’s get started

Prefer a video format? Check out my YouTube video on SQS vs SNS vs Eventbridge here.

What is AWS SQS?

SQS stands for Simple Queue Service. It is one of the oldest services developed and launched by AWS. Only after the release of S3 (Simple Storage Service), SQS was released in July of 2006.

The main purpose of SQS is to facilitate decoupling of applications and to enable asynchronous communication between services. SQS allows developers to publish messages to a Queue, that the consuming application can process either immediately or at a later time.

An traditional alternative way for applications to communicate with eachother by making a HTTP request directly. This will certainly work, however the publishing application now has a hard dependency on the downstream or consuming application. This may be fine in some cases where the publisher needs an action performed or an ‘acknowledgement’ from the recipient before proceeding, but it introduces a latency overhead and tight coupling that can otherwise be avoided.

SQS Main Concepts

1. 1. Queues – Queues are the first class citizens of AWS. They are what you as the user will create either through the console, CLI, or IaC. Queues are the destination of where you, the application owner, will be publishing messages to. Content published to the queue will persist until a consuming application retrieves and processes the messages. A message in a queue gets ‘claimed’ when a thread attempts to process it – during which time that message is invisible to other threads until either it is successfully processed (and therefore deleted), or a timeout occurs (timeout duration is configurable).
   2. Messages – Messages are the content or payload of what gets sent to the queue. Each message is a single entry in the queue, and typically contains a JSON object that is specific to your application. For example, in a credit card transaction application, I may publish events to an SQS queue that indicate a new transaction occurred.

1. Polling – Polling is the mechanism by which the consumer of the queue will process them. You can poll a queue for one or more messages (the latter being called batch polling). When a consuming application polls the queue and acquires messages, they become invisible to all other consuming application threads. SQS supports both short and long polling.

SQS Example

Alright, enough theory. Lets take a look at an example. Consider the following diagram.

In this example, we have an OrdersService. The OrdersService is the publisher in this example, or the owner of the data that they want to notify a consumer about.

It is typical for a service like this to host APIs, for example a /order API that may be responsible for creating orders.

This service takes requests to create Orders, but now needs to notify the AnalyticsService, or consumer service, that a new order was placed. Remember we DON’T want OrdersService to have a direct dependency on AnalyticsService – this would result in tight coupling of a higher order component to a lower order component, a big no no in software architecture.

In order to decouple OrdersService from AnalyticsService and give AnalyticsService the opportunity to process new order creation events at their prescribed rate, we introduce an intermediary; the OrderAnalyticsQueue.

Messages get delivered to this queue on Order creation, and AnalyticsService is free to pick up these messages and process them at their own free will. Do note here that we are using a lambda function as an integration point with SQS. The benefit of using a Lambda is that polling is automatically handled for you. Your lambda function will simply get invoked anytime there are messages present in the queue. Lambda does a good job of hiding the complexity of polling/deleting for you, so this is a nice bonus.

You may be interested in this article on EC2 vs ECS vs Lambda

Alternatively, you can use a HTTP server to regularly poll the queue and process messages. This does require a bit more work, but AWS does provide SDKs to make this process a little bit easier.

One more thing to note is that messages inserted in the queue are not always processed by the consumer in insertion order. SQS performs best effort FIFO processing with a regular queue, but this is by no means guaranteed.

If you need FIFO as part of your application design, you can create a queue and select the FIFO option to enable it with this behaviour. Do note that there are some throughput limitations for FIFO queues that make this a slightly unappealing option for high scale applications (3k TPS for event publishing, whereas unlimited for normal SQS queues).

To sum SQS up, SQS offers queues which a publishing application can send messages to. Messages contain JSON content specific to the application. Messages will persist in the queue until a consuming application polls the queue to process the message.

SQS sounds great when we only have one consumer, in this case AnalyticsService. But what happens if we have multiple services that are interested in receiving messages about Order Creation? Enter SNS.

What is AWS SNS?

SNS stands for Simple Notification Service. Also a very old service that was launched four years after SQS in 2010.

SNS seeks to solve the problem of multiple recipients interested in receiving notifications from publishers. It solves the 1:1 problem we saw above when describing SQS, allowing for a 1:Many relationship where we can notify many applications of changes to an entity.

You can configure a variety of different subscribers to be the destination location for your SNS topic – this can include an SQS queue, a Lambda function, or a regular old HTTP endpoint.

SNS Main Concepts

1. 1. Topics – Topics are the first class citizens of SNS. They are similar to queues in that messages are published to them, but they are not stateful in any way. They do not ‘hold’ messages, they simply are an endpoint that a publishing application can write to, and in turn, rely on SNS to broadcast a copy of that message out to all recipients. In other words, Topics allow for fan-out notifications to many clients. Topics are usually created with a particular theme in mind. Using the same example from our prior exercise, we may have a OrderCreation topic that will broadcast messages to all consumers when a message is published to it.
   2. Messages – Messages are simply JSON blobs that contain payload data. Note that during the publishing process, an identical message is published to each downstream subscriber for consumption.

1. Publish/Subscribe – Publish/Subscribe or PubSub is a term that loosely defines the relationships between owners of data, and consumers of data. The application or person who owns the data would be considered the topic owner or publisher. And the application or person interested in consuming the data would be considered a consumer or subscriber.

Lets take a look at an example now using the same theme as SQS to explain these concepts a bit better.

Example

I’ve slightly modified the SQS example to demonstrate the SNS concepts. You’ll notice that now we have three different subscribers as opposed to just one. We have AccountingService, AnalyticsService, and OrderDashboardService.

These applications all exist independently of one another and have no knowledge that eachother exist. However what draws them together is their common interest in OrderCreation events. AccountingService maybe needs to maintain an accounting ledger of all orders created and charged, AnalyticsService may want to analyze things like frequency of orders and perform Machine Learning algorithms on it, and OrderDashboardService may want to display order volumes to operations folks.

Struggling to get started learning AWS? Check out this article on the CORE AWS Services you NEED to know (and how to learn them!)

In this model, the OrdersTopic has has subscriptions to each of these services. Upon a message being published to the Topic by OrdersService, SNS will in the background publish messages to each of these three consumers.

Note that a very common pattern is to use either SNS + SQS or SNS + Lambda. Using SNS + SQS allows you to combine the benefits of both of these options – having a single point fan out SNS topic that delivers message to multiple consumer queues. Consumer application owners (i.e. AccountingService) could have its own SQS queue that gets messages delivered to it. AccountingService could then poll the queue to process messages at an appropriate rate.

In summary, SNS allows for creation of topics which are aligned with a certain application theme. Topics have subscribers that are interested in being notified of whenever a change event occurs. The topic owner will publish messages to the topic, and behind the scenes, AWS will push messages to all subscribing applications, whether that be a SQS queue subscriber, a Lambda function, or something else.

What is AWS Eventbridge?

Eventbridge is a newer service in the AWS ecosystem launched in July of 2019. It has very similar characteristics to AWS SNS and provides a host of other value-add features that makes developers lives easier.

Similar to SNS, eventbridge allows for messages to be broadcaster to subscribers to be processed at their own will. Lets take a look at the main concepts that exist in the Eventbridge ecosystem.

1. Message Bus – Message Buses are very similar to SNS topics in that they receive events that need to be broadcaster to downstream consumers.
2. Events – Events are similar to messages in the context of SNS and SQS, just with a fancier name. They consist of JSON blobs that describes the source and payload of the event. Events can also be “scheduled” to run at periodic intervals using a cron expression. This is useful for those of you looking to perform timed batch jobs regularly at a certain time of day.
3. Targets – Targets are the downstream recipient of events that are published to the message bus. Very similar to SNS consumers.
4. Rules – Rules are the routing logic component for Message Buses. Essentially, you can configure rules such that only when a certain condition is met (within the message data itself), will a message be broadcaster to a specific target. An Event Pattern is something that you define that matches the content of the message to a specific target. You can have many rules that all match to different patterns, but only 5 targets per rule. If you’d like to have more, you would need to create a new rule with the same event pattern, but with different configured targets.

I’m not going to waste your time with showing another example of Eventbridge in action, its basically the same as the SNS diagram except with an event bus instead of a topic, and targets instead of subscribers.

In my opinion, there are two key features of Message Bus that make it stand out as a competitive option to SNS: Schema Registry/Discovery and Third Party Integrations.

Schema Registry/Discovery

One of the big problems with using SNS is determining message format and content. For example, if you subscribe to an SNS topic, how will you know the exact schema of the messages you’ll be receiving? Typically, we would rely on the publishing team to provide model objects for their notifications so that consumers can use them to deserialize the message. However, this is usually something that gets overlooked during the development process.

Schema Registry is a component of eventbridge that allows developers to register their event schemas and make them discoverable by other teams. The registry automates the process of detecting event schemas and making them available to other teams.

Users looking to consume a message of a particular schema can generate code bindings which output model objects in a language of your choice. This makes it a lot easier to hit the ground running and building your application quicker. This is a huge plus for Eventbridge over SNS.

For more info on Schema Registry/Discovery, check out this AWS blog post.

Third Party Integrations

Perhaps the largest selling point of using Eventbridge is its native support for third party integrations.

Today, there are tons of SaaS companies that emit useful events that developers want to integrate into their application. For example Shopify – wouldn’t it be nice if automatically we can hook into Shopify order creation events for our e-commerce store, and invoke a Lambda function as a result?

With Eventbridge Third Party Integrations, this becomes a trivial task. The feature supports many popular integration services such as Shopify, Datadog, Pagerduty, Zendesk, Amazon Seller API and many many more. See a list in this link.

This means developers have little work to do to detect and broadcast these messages into their application ecosystem.

In my opinion, Third Party Integrations are one of the biggest reasons to go with Eventbridge over SNS. If you find this feature useful for your application, then I say go ahead with eventbridge, no questions asked.

Scheduled Events (Formerly Cloudwatch Events)

One of the neat hidden features of EventBridge is the ability to create scheduled events that periodically ‘poke’ your event bus to broadcast a message.

This is not a new feature – in fact, scheduled events were up until recently branded as Cloudwatch Events. These days, cloudwatch events has been absorbed as a feature of eventbridge.

Using scheduled events, you can use a cron expression to set a periodic event that fires at a certain time. For example, this expression 0 8 * * * translates to “Everyday at 8am”. You can even customize the payload of the event to contain content of your choosing. Crontab is a great tool to help you build your cron expressions by the way.

Scheduled events can be used for periodic maintenance tasks and a wide variety of other use cases.

AWS SQS vs SNS vs Eventbridge – When to Use What?

We talked a lot about the different services in this article, and I think by now its pretty clear when one service is useful over another, so lets summarize the comparison with a diagram:

The summary is:

Use SQS when:

1. You’re looking for reliable 1:1 Asynchronous communication to decouple your applications from one another
2. You want to rate limit your consumption of messages (perhaps due to a database bottleneck or some other use case)
3. You want ordered message processing of vents

Use SNS when:

1. You want to publish messages to MANY different subscribers with a single action
2. Require high throughput and reliability for publishing and delivery to consumers
3. Have many subscribers

Use Eventbridge when:

1. You want to publish messages to many subscribers, and use the event data itself to match targets interested certain patterns.
2. Want integration with other SaaS providers such as Shopify, Datadog, Pagerduty, or others
3. Want to easily discover schemas that other teams produce and incorporate them into your application.
4. You want to use regularly scheduled events using a cron-like expression to periodically send messages to your event bus.

I hope you learned a lot from this article and have a solid understanding of SQS, SNS, and Eventbridge. If you have any questions or comments, feel free to leave them in the comments section below!