Using SQS FIFO Queues

Published on by

Using SQS FIFO Queues image

AWS's Simple Queue Service (SQS) is a great, high-scale, and extremely inexpensive way to get reliable queues.

One minor quibble with SQS is that you can't gurantee what order jobs will be processed in.

If there are 10 jobs in a queue, you don't know which job a queue worker will be given.

For many of us, that's just fine. If you need some jobs to be done in order, you can use Laravels Job Chaining.

However, that doesn't work in all cases.

For example, in Chipper CI, we need to process builds in the order we receive them. We have to wait on incoming webhooks that can come at any time, and so we can't simply add to a Job Chain. Instead rely on SQS FIFO queues.

FIFO Queues

FIFO stands for First In, First Out. That means that the oldest job will always be the first to be given to a queue worker.

The job "behind" the oldest job will not be given to a queue worker until the job before it is complete.

This is great when you need jobs to be processed in order, but this limits scaling queue workers.

If this was all there was to it, you'd never be able to use more than one queue worker nor process more than one job at a time (without creating and somehow managing multiple FIFO queues).

Message Group IDs

Luckily FIFO queues account for this. It turns out that messages order is only guaranteed within a Group ID.

Each job (message) in FIFO queues must have a Group ID. Any jobs that have the same Group ID will be processed in order they are received.

If you assigned a random Group ID to each job, then no order would be maintained as there would be no shared Group ID.

In Chipper CI, each Team can have multiple Projects. Each Project is a git repository that gets built when someone pushes an update to their git repository.

To keep builds in order, we use FIFO queues, creating a Group ID using the Team ID and Project ID. It looks like this:

$build = App\Build::createFromWebhook($payload);
$groupId = sprintf('build-%s-%s', $build->project->team_id, $build->project_id);
(new ProcessBuild($build))

SQS FIFO Laravel Package

To help with this, we can use the shiftonelabs/laravel-sqs-fifo-queue Composer package.

This exends the SQS queue message type and adds some functionality.

Most importantly, it gives us the onMessageGroup() method, which allows us to add the message group ID when adding a job (message) to the SQS FIFO queue.

Chris Fidao photo

Teaching coding and servers at CloudCasts and Servers for Hackers. Co-founder of Chipper CI.


Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Visit Laravel Forge
Kirschbaum logo


Providing innovation and stability to ensure your web application succeeds.

Shift logo


Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Bacancy logo


Supercharge your project with a seasoned Laravel developer with 4-6 years of experience for just $2500/month. Get 160 hours of dedicated expertise & a risk-free 15-day trial. Schedule a call now!

LoadForge logo


Easy, affordable load testing and stress tests for websites, APIs and databases.

Paragraph logo


Manage your Laravel app as if it was a CMS – edit any text on any page or in any email without touching Blade or language files.

Lucky Media logo

Lucky Media

Bespoke software solutions built for your business. Partner with Lucky Media, your favorite Laravel Development Agency!

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Oh Dear logo

Oh Dear

Oh Dear is the best all-in-one monitoring tool for all your Laravel apps.

Oh Dear
Tinkerwell logo


The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.


The latest

View all →
FrankenPHP v1.0 is Here image

FrankenPHP v1.0 is Here

Read article
Self-healing URLs in Laravel image

Self-healing URLs in Laravel

Read article
Laravel 10.35 Released image

Laravel 10.35 Released

Read article
Solve n+1 queries in PHP with Scout APM image

Solve n+1 queries in PHP with Scout APM

Read article
Show Outdated Composer Dependencies in Laravel Pulse image

Show Outdated Composer Dependencies in Laravel Pulse

Read article
Announcing Laravel Pulse - A New Performance Monitoring Tool for Laravel Apps image

Announcing Laravel Pulse - A New Performance Monitoring Tool for Laravel Apps

Read article