Using SQS FIFO Queues

Published on by

Using SQS FIFO Queues image

AWS's Simple Queue Service (SQS) is a great, high-scale, and extremely inexpensive way to get reliable queues.

One minor quibble with SQS is that you can't gurantee what order jobs will be processed in.

If there are 10 jobs in a queue, you don't know which job a queue worker will be given.

For many of us, that's just fine. If you need some jobs to be done in order, you can use Laravels Job Chaining.

However, that doesn't work in all cases.

For example, in Chipper CI, we need to process builds in the order we receive them. We have to wait on incoming webhooks that can come at any time, and so we can't simply add to a Job Chain. Instead rely on SQS FIFO queues.

FIFO Queues

FIFO stands for First In, First Out. That means that the oldest job will always be the first to be given to a queue worker.

The job "behind" the oldest job will not be given to a queue worker until the job before it is complete.

This is great when you need jobs to be processed in order, but this limits scaling queue workers.

If this was all there was to it, you'd never be able to use more than one queue worker nor process more than one job at a time (without creating and somehow managing multiple FIFO queues).

Message Group IDs

Luckily FIFO queues account for this. It turns out that messages order is only guaranteed within a Group ID.

Each job (message) in FIFO queues must have a Group ID. Any jobs that have the same Group ID will be processed in order they are received.

If you assigned a random Group ID to each job, then no order would be maintained as there would be no shared Group ID.

In Chipper CI, each Team can have multiple Projects. Each Project is a git repository that gets built when someone pushes an update to their git repository.

To keep builds in order, we use FIFO queues, creating a Group ID using the Team ID and Project ID. It looks like this:

$build = App\Build::createFromWebhook($payload);
 
$groupId = sprintf('build-%s-%s', $build->project->team_id, $build->project_id);
 
dispatch(
(new ProcessBuild($build))
->onMessageGroup($groupId)
);

SQS FIFO Laravel Package

To help with this, we can use the shiftonelabs/laravel-sqs-fifo-queue Composer package.

This exends the SQS queue message type and adds some functionality.

Most importantly, it gives us the onMessageGroup() method, which allows us to add the message group ID when adding a job (message) to the SQS FIFO queue.

Chris Fidao photo

Teaching coding and servers at CloudCasts and Servers for Hackers. Co-founder of Chipper CI.

Cube

Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

image
Paragraph

Manage your Laravel app as if it was a CMS – edit any text on any page or in any email without touching Blade or language files.

Visit Paragraph
Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Tinkerwell logo

Tinkerwell

The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.

Tinkerwell
No Compromises logo

No Compromises

Joel and Aaron, the two seasoned devs from the No Compromises podcast, are now available to hire for your Laravel project. ⬧ Flat rate of $7500/mo. ⬧ No lengthy sales process. ⬧ No contracts. ⬧ 100% money back guarantee.

No Compromises
Kirschbaum logo

Kirschbaum

Providing innovation and stability to ensure your web application succeeds.

Kirschbaum
Shift logo

Shift

Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Shift
Bacancy logo

Bacancy

Supercharge your project with a seasoned Laravel developer with 4-6 years of experience for just $2500/month. Get 160 hours of dedicated expertise & a risk-free 15-day trial. Schedule a call now!

Bacancy
LoadForge logo

LoadForge

Easy, affordable load testing and stress tests for websites, APIs and databases.

LoadForge
Paragraph logo

Paragraph

Manage your Laravel app as if it was a CMS – edit any text on any page or in any email without touching Blade or language files.

Paragraph
Lucky Media logo

Lucky Media

Bespoke software solutions built for your business. We ♥ Laravel

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
DocuWriter.ai logo

DocuWriter.ai

Save hours of manually writing Code Documentation, Comments & DocBlocks, Test suites and Refactoring.

DocuWriter.ai
Rector logo

Rector

Your partner for seamless Laravel upgrades, cutting costs, and accelerating innovation for successful companies

Rector

The latest

View all →
Generate Code Coverage in Laravel With PCOV image

Generate Code Coverage in Laravel With PCOV

Read article
Non-backed Enums in Database Queries and a withSchedule() bootstrap method in Laravel 11.1 image

Non-backed Enums in Database Queries and a withSchedule() bootstrap method in Laravel 11.1

Read article
Laravel Pint --bail Flag image

Laravel Pint --bail Flag

Read article
Laravel Herd for Windows is now released! image

Laravel Herd for Windows is now released!

Read article
The Laravel Worldwide Meetup is Today image

The Laravel Worldwide Meetup is Today

Read article
Cache Routes with Cloudflare in Laravel image

Cache Routes with Cloudflare in Laravel

Read article