How to do queues in AWS Lambda

There is a second part to this post added nearly a year after this… read this first though! Part 2 here: https://medium.com/@PaulDJohnston/how-to-do-queues-for-aws-lambda-part-2-e84c71bd62d2

It’s currently ServerlessConf.io at present and it is proving to be awesome!

This is just a quick post, that I think might help people.

One of the things that I’ve had long conversations at the conference is queues. There are many different ways of doing queues and lots of services in AWS alone that can give you queues. Since we are using Lambda, then it makes sense to use an AWS service to do this… so here goes.

SQS

Don’t use it with Lambda because it doesn’t generate events to trigger Lambda from — it’s a bad idea.

SNS

However, SNS is clunky as well. It’s really more for sending messages to SMS and push notifications to mobile apps. You have a bit more of an issue with speed and delivery mechanisms, which mean that I’ve found you need to hack it a little more on receipt in the Lambda. SNS is also a little more complex to setup in the first place — the ideas of publishers and subscribers and topics are all well and good, but really, it’s quite complex for a queue.

It’s fine to use, but there is a better way!

AWS IoT

Again there is a better way.

DynamoDB Streams

The really brilliant thing about DynamoDB Tables is Streams. This is essentially a way of capturing what is going on in the table, including addition and update of data.

It is a completely time-ordered stream of the table data updates. This time ordering is important, because it allows you to setup a message queue for a specific problem and do something in time order.

And you can push that table data straight into a Lambda function.

If you set it up as is, you get multiple records sent to Lambda. However, if you set it up to ensure that only 1 record is pushed out via a stream at a time then that allows you to create a message queue into Lambda using only a DynamoDB table and a DynamoDB Stream.

Of course, go and have a look at this yourself and learn about it, but I’m convinced that (at present — until there is a better way) the right way to do simple message queues within AWS is using DynamoDB Tables and Streams.

Event processing

It also gives us the possibility of being able to audit a queue going forwards, which has a lot of value in a debugging scenario. This might seem a little thing, but it’s genuinely quite a lot more important in a Serverless scenario.

Hope this helps some people queue events up more easily. Thoughts welcome.

Note: Collected Blog Posts on Serverless can be found here

Part 2 of this post is here: https://medium.com/@PaulDJohnston/how-to-do-queues-for-aws-lambda-part-2-e84c71bd62d2

Written by

ServerlessDays CoFounder (Jeff), ex AWS Serverless Snr DA, experienced CTO/Interim, Startups, Entrepreneur, Techie, Geek and Christian

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store