How to do queues in AWS Lambda
There is a second part to this post added nearly a year after this… read this first though! Part 2 here: https://medium.com/@PaulDJohnston/how-to-do-queues-for-aws-lambda-part-2-e84c71bd62d2
It’s currently ServerlessConf.io at present and it is proving to be awesome!
This is just a quick post, that I think might help people.
One of the things that I’ve had long conversations at the conference is queues. There are many different ways of doing queues and lots of services in AWS alone that can give you queues. Since we are using Lambda, then it makes sense to use an AWS service to do this… so here goes.
SQS
There is (of course) SQS, which is the Simple Queue Service and is for message queueing. But it’s a pull scenario where you need to pull data from it when ready. The way you use it with lambda is complex because of the fact it’s pull, but it’s just clunky.
Don’t use it with Lambda because it doesn’t generate events to trigger Lambda from — it’s a bad idea.
SNS
There is SNS, which is the main way that people seem to queue on Lambda (this was the way I was told to do it when I asked several months ago). SNS has a brilliant thing which is that you can trigger a Lambda function when you receive a message. It’s a push functionality (push something in, it pushes something out the other end) which is helpful.
However, SNS is clunky as well. It’s really more for sending messages to SMS and push notifications to mobile apps. You have a bit more of an issue with speed and delivery mechanisms, which mean that I’ve found you need to hack it a little more on receipt in the Lambda. SNS is also a little more complex to setup in the first place — the ideas of publishers and subscribers and topics are all well and good, but really, it’s quite complex for a queue.
It’s fine to use, but there is a better way!
AWS IoT
Then there is AWS IoT service, which contains MQTT as a message broker solution. This is brilliant! But it’s also a little complicated again to do this. If you’re happy with MQTT then fine, but it isn’t a straight forward way of doing it either. Once setup, it should be fine, but it’s not straight forward to code and push to or receive from.
Again there is a better way.
DynamoDB Streams
This is my current favourite solution. Create a new DynamoDB table to hold whatever data you want in the queue. What the data actually is is completely up to you of course.
The really brilliant thing about DynamoDB Tables is Streams. This is essentially a way of capturing what is going on in the table, including addition and update of data.
It is a completely time-ordered stream of the table data updates. This time ordering is important, because it allows you to setup a message queue for a specific problem and do something in time order.
And you can push that table data straight into a Lambda function.
If you set it up as is, you get multiple records sent to Lambda. However, if you set it up to ensure that only 1 record is pushed out via a stream at a time then that allows you to create a message queue into Lambda using only a DynamoDB table and a DynamoDB Stream.
Of course, go and have a look at this yourself and learn about it, but I’m convinced that (at present — until there is a better way) the right way to do simple message queues within AWS is using DynamoDB Tables and Streams.
Event processing
We use this to manage a number of message queues including event processing queues over and above SQS and SNS although we have used both in the past. Because we already store data in DynamoDB then using Streams means that we don’t have to think beyond to other services (no need to learn other services). Cost is slightly different (tables are not free whereas some queues are), but it does feel more straight forward to implement and is more stable in our setup we think. It’s more testable as well in our opinion.
It also gives us the possibility of being able to audit a queue going forwards, which has a lot of value in a debugging scenario. This might seem a little thing, but it’s genuinely quite a lot more important in a Serverless scenario.
Hope this helps some people queue events up more easily. Thoughts welcome.
Note: Collected Blog Posts on Serverless can be found here
Part 2 of this post is here: https://medium.com/@PaulDJohnston/how-to-do-queues-for-aws-lambda-part-2-e84c71bd62d2
