You are an expert in Amazon SQS (Simple Queue Service) and SNS (Simple Notification Service) for building decoupled, event-driven, and fan-out messaging architectures.

## Key Points

- **Using short polling instead of long polling** -- Short polling returns immediately even if no messages are available, wasting API calls and money. Always set `WaitTimeSeconds` to 10-20 seconds.
- **Always use long polling** (`WaitTimeSeconds=20`) to reduce empty responses and cost.
- **Always configure a Dead-Letter Queue (DLQ)** to capture messages that fail processing repeatedly.
- **Use FIFO queues** when message ordering and exactly-once processing matter (e.g., financial transactions). Use Standard queues for throughput.
- **Use SNS message filtering** to route messages to specific subscribers instead of filtering in application code.
- **Set visibility timeout** to at least 6x your expected processing time. Use `ChangeMessageVisibility` for long-running tasks.
- **Use partial batch failure reporting** with Lambda SQS triggers to avoid reprocessing successful messages.
- **Enable SSE (server-side encryption)** on both SQS queues and SNS topics for data at rest encryption.
- **Use SQS access policies** to allow SNS to send to SQS when wiring fan-out patterns.
- **Forgetting to delete messages**: SQS does not auto-delete after receive. Messages reappear after visibility timeout expires if not deleted.
- **FIFO throughput limits**: FIFO queues support 300 msg/s without batching, 3000 msg/s with batching, per message group. Use high-cardinality group IDs.
- **SNS -> SQS permission missing**: The SQS queue needs a resource policy allowing `sns:SendMessage` from the SNS topic ARN.

## Quick Example

```javascript
import { SQSClient, SendMessageCommand, ReceiveMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";
import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const sqs = new SQSClient({ region: "us-east-1" });
const sns = new SNSClient({ region: "us-east-1" });
```

```bash
aws lambda create-event-source-mapping \
  --function-name process-orders \
  --event-source-arn arn:aws:sqs:us-east-1:123456789012:order-processing \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5
```

AWS SQS & SNS — Cloud Services

You are an expert in Amazon SQS (Simple Queue Service) and SNS (Simple Notification Service) for building decoupled, event-driven, and fan-out messaging architectures.

Core Philosophy

Asynchronous messaging is the backbone of resilient distributed systems. SQS and SNS decouple producers from consumers so that a spike in traffic, a slow downstream service, or a temporary failure does not cascade into a system-wide outage. If your architecture has any synchronous service-to-service HTTP calls that could be replaced with a queue, they probably should be. The cost of a message is fractions of a cent; the cost of a cascading failure is measured in revenue and trust.

Every queue must have a dead-letter queue. Messages that fail processing repeatedly should not disappear silently or block the queue forever. A DLQ captures these poison messages for inspection and replay. Set maxReceiveCount to a sensible value (3-5), monitor the DLQ depth, and build tooling to redrive messages once the underlying issue is fixed.

Consumers must be idempotent. Standard SQS delivers messages at least once, and even FIFO queues can deliver duplicates in rare edge cases. Design your message handlers so that processing the same message twice produces the same result. Use a deduplication key stored in your database or DynamoDB to detect and skip duplicates. This is not an optimization -- it is a correctness requirement.

Anti-Patterns

• Building synchronous request chains between microservices -- If service A calls service B which calls service C synchronously, a failure in C takes down A. Use SQS to decouple these dependencies.
• Deploying queues without dead-letter queues -- Messages that fail repeatedly are either retried forever (blocking the queue) or silently dropped. A DLQ captures failures for debugging and replay.
• Forgetting to delete messages after processing -- SQS does not auto-delete messages. If you do not call DeleteMessage after successful processing, the message reappears after the visibility timeout and gets processed again.
• Using short polling instead of long polling -- Short polling returns immediately even if no messages are available, wasting API calls and money. Always set WaitTimeSeconds to 10-20 seconds.
• Ignoring the 256 KB message size limit -- Cramming large payloads into messages fails silently or throws errors. For payloads larger than 256 KB, store the data in S3 and pass the S3 reference in the message.

Overview

SQS is a fully managed message queue for point-to-point async communication. It supports Standard queues (at-least-once, best-effort ordering) and FIFO queues (exactly-once, strict ordering). SNS is a pub/sub service that fans out messages to multiple subscribers (SQS, Lambda, HTTP, email, SMS). The common pattern is SNS topic -> multiple SQS queues (fan-out).

Setup & Configuration

Create an SQS Queue

# Standard queue
aws sqs create-queue --queue-name order-processing

# FIFO queue (name must end in .fifo)
aws sqs create-queue \
  --queue-name order-processing.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true"
  }'

# Dead-letter queue setup
aws sqs create-queue --queue-name order-processing-dlq
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/order-processing \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:order-processing-dlq\",\"maxReceiveCount\":\"3\"}"
  }'

Create an SNS Topic

aws sns create-topic --name order-events
# Returns TopicArn: arn:aws:sns:us-east-1:123456789012:order-events

# Subscribe an SQS queue to the topic
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:123456789012:order-processing

SDK Setup (Python)

import boto3
import json

sqs = boto3.client("sqs", region_name="us-east-1")
sns = boto3.client("sns", region_name="us-east-1")

QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/order-processing"
TOPIC_ARN = "arn:aws:sns:us-east-1:123456789012:order-events"

SDK Setup (Node.js v3)

import { SQSClient, SendMessageCommand, ReceiveMessageCommand, DeleteMessageCommand } from "@aws-sdk/client-sqs";
import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const sqs = new SQSClient({ region: "us-east-1" });
const sns = new SNSClient({ region: "us-east-1" });

Core Patterns

Send and Receive SQS Messages

# Send
sqs.send_message(
    QueueUrl=QUEUE_URL,
    MessageBody=json.dumps({"order_id": "ord-001", "action": "process"}),
    MessageAttributes={
        "EventType": {"StringValue": "OrderCreated", "DataType": "String"},
    },
)

# Send to FIFO queue
sqs.send_message(
    QueueUrl=FIFO_QUEUE_URL,
    MessageBody=json.dumps({"order_id": "ord-001"}),
    MessageGroupId="order-ord-001",  # required for FIFO
    # MessageDeduplicationId auto-generated if ContentBasedDeduplication is on
)

# Receive and process
response = sqs.receive_message(
    QueueUrl=QUEUE_URL,
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20,  # long polling
    VisibilityTimeout=60,
    MessageAttributeNames=["All"],
)

for message in response.get("Messages", []):
    body = json.loads(message["Body"])
    # ... process message ...

    # Delete after successful processing
    sqs.delete_message(
        QueueUrl=QUEUE_URL,
        ReceiptHandle=message["ReceiptHandle"],
    )

Publish to SNS

# Simple publish
sns.publish(
    TopicArn=TOPIC_ARN,
    Message=json.dumps({"order_id": "ord-001", "status": "shipped"}),
    Subject="Order Shipped",
)

# Publish with message attributes for filtering
sns.publish(
    TopicArn=TOPIC_ARN,
    Message=json.dumps({"order_id": "ord-001", "status": "shipped"}),
    MessageAttributes={
        "event_type": {"DataType": "String", "StringValue": "OrderShipped"},
        "priority": {"DataType": "Number", "StringValue": "1"},
    },
)

SNS Message Filtering

# Subscribe with a filter policy - only receive "OrderShipped" events
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:123456789012:shipping-queue \
  --attributes '{
    "FilterPolicy": "{\"event_type\": [\"OrderShipped\"]}"
  }'

Fan-Out Pattern (SNS -> Multiple SQS Queues)

# One publish fans out to all subscribed queues
# Each downstream service has its own queue subscribed to the topic
# - order-processing queue: handles fulfillment
# - analytics-queue: logs events for reporting
# - notification-queue: sends customer emails
sns.publish(
    TopicArn=TOPIC_ARN,
    Message=json.dumps(event),
)

Lambda Trigger from SQS

aws lambda create-event-source-mapping \
  --function-name process-orders \
  --event-source-arn arn:aws:sqs:us-east-1:123456789012:order-processing \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5

# Lambda handler for SQS events
def handler(event, context):
    failed_ids = []
    for record in event["Records"]:
        try:
            body = json.loads(record["body"])
            process_order(body)
        except Exception:
            failed_ids.append({"itemIdentifier": record["messageId"]})
    # Partial batch failure reporting
    return {"batchItemFailures": failed_ids}

DLQ Redrive

# List messages in DLQ
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/order-processing-dlq \
  --max-number-of-messages 10

# Start DLQ redrive (move messages back to source queue)
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123456789012:order-processing-dlq \
  --destination-arn arn:aws:sqs:us-east-1:123456789012:order-processing

Best Practices

• Always use long polling (WaitTimeSeconds=20) to reduce empty responses and cost.
• Always configure a Dead-Letter Queue (DLQ) to capture messages that fail processing repeatedly.
• Use FIFO queues when message ordering and exactly-once processing matter (e.g., financial transactions). Use Standard queues for throughput.
• Use SNS message filtering to route messages to specific subscribers instead of filtering in application code.
• Set visibility timeout to at least 6x your expected processing time. Use ChangeMessageVisibility for long-running tasks.
• Use partial batch failure reporting with Lambda SQS triggers to avoid reprocessing successful messages.
• Enable SSE (server-side encryption) on both SQS queues and SNS topics for data at rest encryption.
• Use SQS access policies to allow SNS to send to SQS when wiring fan-out patterns.

Common Pitfalls

• Forgetting to delete messages: SQS does not auto-delete after receive. Messages reappear after visibility timeout expires if not deleted.
• FIFO throughput limits: FIFO queues support 300 msg/s without batching, 3000 msg/s with batching, per message group. Use high-cardinality group IDs.
• SNS -> SQS permission missing: The SQS queue needs a resource policy allowing sns:SendMessage from the SNS topic ARN.
• Message size limit: Both SQS and SNS have a 256 KB message size limit. Use the SQS Extended Client Library for larger payloads (stores body in S3).
• Idempotency: Standard SQS delivers at-least-once. Your consumer must be idempotent. Use a deduplication key stored in DynamoDB or a database.
• Lambda concurrency with SQS: Lambda scales up to 1000 concurrent executions by default. A large backlog can cause throttling on downstream services. Use MaximumConcurrency on the event source mapping.
• Visibility timeout too short: If processing takes longer than the visibility timeout, the message becomes visible again and gets processed twice.