How to design an event-driven serverless platform on AWS: lessons from 90+ Lambda functions

Zayin Krige, 2024-12-29. 4 min read.

An event-driven serverless platform on AWS is built from four services: Lambda for compute, SNS for publishing events, SQS for buffering them per consumer, and DynamoDB for state. Every write publishes an event, every consumer reads from its own queue, and every read model is a projection rebuilt from the event stream. Done this way the system scales to zero when idle, scales out under load without capacity planning, and lets one team add a consumer without touching another team's code.

I architected a platform on this pattern for a health and wearables company: biometric data from consumer devices flowing through more than 90 Lambda functions, wired with SNS, SQS, DynamoDB and API Gateway, provisioned entirely through Terraform. What follows is the design, and the mistakes it took to arrive at it.

1. Fan out with SNS, buffer with SQS, never call Lambda from Lambda

The first version of most serverless systems has Lambda functions invoking each other directly. It works until one downstream function slows down and the failure cascades back to the API. The fix is one SNS topic per event type and one SQS queue per consumer subscribed to it. The publisher does not know who consumes; a new consumer is a new queue and a new subscription, nothing else changes. SQS also gives you retries, batching and back-pressure for free.

2. Make every consumer idempotent

SQS standard queues deliver at least once, which means every consumer will eventually see the same message twice. Design for it from the first line: derive an idempotency key from the event, write it to DynamoDB with a conditional put, and drop the message if the key already exists. A consumer that is not idempotent will double-count a biometric sample, double-send an email, or double-charge a card, and the bug will appear in production months later under load.

3. Use CQRS-style projections instead of joins

DynamoDB does not join. Instead of fighting that, treat every screen or API response as a read model and build a consumer that maintains it. A sleep-summary view is a DynamoDB item updated by a Lambda that consumes the raw-sample events. The write side stays simple and append-only; the read side is as denormalised as the query needs. When the read model's shape changes, replay the events into a new table and switch the reader over. This is CQRS (Command Query Responsibility Segregation) without the ceremony.

4. Put a dead-letter queue on everything

Every SQS queue gets a dead-letter queue with a maximum receive count of three to five, and every dead-letter queue gets a CloudWatch alarm on its depth. A message that fails repeatedly is a bug report with the payload attached. Without the DLQ (dead-letter queue) the message is retried until it expires and the data is silently lost.

5. Respect the Lambda limits

The limits that bite in practice, per the AWS Lambda documentation: a maximum timeout of 15 minutes, a maximum of 10,240 MB of memory, and a 6 MB synchronous payload. Anything that might exceed 15 minutes belongs in a Step Functions workflow or a container on Fargate, not in a function that is "usually fast". Anything that returns more than 6 MB should write to S3 and return a presigned URL.

6. Provision with Terraform, never the console

Ninety functions, ninety queues, ninety subscriptions and their IAM roles cannot be managed by hand. Every resource on the platform is a Terraform module: one module for "a consumer", taking the handler and the topic as inputs and producing the queue, the DLQ, the subscription, the function and the least-privilege role. Adding a consumer is a ten-line module call and a pull request. The console is read-only.

7. Keep the API thin

API Gateway in front of Lambda handles authentication and validation, writes the command to DynamoDB, publishes the event and returns. It does no downstream work. Response times stay flat regardless of how many consumers exist because none of them are on the request path. The mobile SDK, in this case a Flutter SDK with BLE device integration, retries against that thin API and never waits on processing.

8. Observe by correlation ID, not by function

A single device sync can touch twenty functions. Put a correlation ID on the originating request, carry it in every event's message attributes, and log it as a structured field in every function. CloudWatch Logs Insights can then reconstruct the whole path with one query. Per-function dashboards look impressive and answer nothing.

What it costs to run

Serverless is cheap at low and bursty volume and can be expensive at sustained high volume. The break-even depends on the workload, but the honest comparison is against the engineering time a container platform consumes: patching, scaling policies, on-call. For a small team the operational saving has, in every case I have measured, outweighed the per-request premium.

Where to start

Do not design ninety functions. Design one event type, one SNS topic, two consumers with their own queues and DLQs, and the Terraform module that creates them. Ship that. Every later feature is another instance of the same module. If you are deciding whether to bring in outside help to set this up, the article on what a fractional CTO does covers what that engagement looks like.