Using SQS FIFO Queues

Published on by

Using SQS FIFO Queues image

AWS's Simple Queue Service (SQS) is a great, high-scale, and extremely inexpensive way to get reliable queues.

One minor quibble with SQS is that you can't gurantee what order jobs will be processed in.

If there are 10 jobs in a queue, you don't know which job a queue worker will be given.

For many of us, that's just fine. If you need some jobs to be done in order, you can use Laravels Job Chaining.

However, that doesn't work in all cases.

For example, in Chipper CI, we need to process builds in the order we receive them. We have to wait on incoming webhooks that can come at any time, and so we can't simply add to a Job Chain. Instead rely on SQS FIFO queues.

FIFO Queues

FIFO stands for First In, First Out. That means that the oldest job will always be the first to be given to a queue worker.

The job "behind" the oldest job will not be given to a queue worker until the job before it is complete.

This is great when you need jobs to be processed in order, but this limits scaling queue workers.

If this was all there was to it, you'd never be able to use more than one queue worker nor process more than one job at a time (without creating and somehow managing multiple FIFO queues).

Message Group IDs

Luckily FIFO queues account for this. It turns out that messages order is only guaranteed within a Group ID.

Each job (message) in FIFO queues must have a Group ID. Any jobs that have the same Group ID will be processed in order they are received.

If you assigned a random Group ID to each job, then no order would be maintained as there would be no shared Group ID.

In Chipper CI, each Team can have multiple Projects. Each Project is a git repository that gets built when someone pushes an update to their git repository.

To keep builds in order, we use FIFO queues, creating a Group ID using the Team ID and Project ID. It looks like this:

$build = App\Build::createFromWebhook($payload);
 
$groupId = sprintf('build-%s-%s', $build->project->team_id, $build->project_id);
 
dispatch(
(new ProcessBuild($build))
->onMessageGroup($groupId)
);

SQS FIFO Laravel Package

To help with this, we can use the shiftonelabs/laravel-sqs-fifo-queue Composer package.

This exends the SQS queue message type and adds some functionality.

Most importantly, it gives us the onMessageGroup() method, which allows us to add the message group ID when adding a job (message) to the SQS FIFO queue.

Chris Fidao photo

Teaching coding and servers at CloudCasts and Servers for Hackers. Co-founder of Chipper CI.

Cube

Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Tinkerwell logo

Tinkerwell

The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.

Tinkerwell
No Compromises logo

No Compromises

Joel and Aaron, the two seasoned devs from the No Compromises podcast, are now available to hire for your Laravel project. ⬧ Flat rate of $7500/mo. ⬧ No lengthy sales process. ⬧ No contracts. ⬧ 100% money back guarantee.

No Compromises
Kirschbaum logo

Kirschbaum

Providing innovation and stability to ensure your web application succeeds.

Kirschbaum
Shift logo

Shift

Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Shift
Bacancy logo

Bacancy

Supercharge your project with a seasoned Laravel developer with 4-6 years of experience for just $2500/month. Get 160 hours of dedicated expertise & a risk-free 15-day trial. Schedule a call now!

Bacancy
Lucky Media logo

Lucky Media

Get Lucky Now - the ideal choice for Laravel Development, with over a decade of experience!

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
LaraJobs logo

LaraJobs

The official Laravel job board

LaraJobs
SaaSykit: Laravel SaaS Starter Kit logo

SaaSykit: Laravel SaaS Starter Kit

SaaSykit is a Multi-tenant Laravel SaaS Starter Kit that comes with all features required to run a modern SaaS. Payments, Beautiful Checkout, Admin Panel, User dashboard, Auth, Ready Components, Stats, Blog, Docs and more.

SaaSykit: Laravel SaaS Starter Kit
Supercharge Your SaaS Development with FilamentFlow: The Ultimate Laravel Filament Boilerplate logo

Supercharge Your SaaS Development with FilamentFlow: The Ultimate Laravel Filament Boilerplate

Build your SaaS application in hours. Out-of-the-box multi-tenancy and seamless Stripe integration. Supports subscriptions and one-time purchases, allowing you to focus on building and creating without repetitive setup tasks.

Supercharge Your SaaS Development with FilamentFlow: The Ultimate Laravel Filament Boilerplate
Rector logo

Rector

Your partner for seamless Laravel upgrades, cutting costs, and accelerating innovation for successful companies

Rector
MongoDB logo

MongoDB

Enhance your PHP applications with the powerful integration of MongoDB and Laravel, empowering developers to build applications with ease and efficiency. Support transactional, search, analytics and mobile use cases while using the familiar Eloquent APIs. Discover how MongoDB's flexible, modern database can transform your Laravel applications.

MongoDB

The latest

View all →
PHPStan 2.0 is Here image

PHPStan 2.0 is Here

Read article
Run multiple CLI commands locally at once with Solo for Laravel image

Run multiple CLI commands locally at once with Solo for Laravel

Read article
Securing Laravel Applications with Stephen Rees-Carter image

Securing Laravel Applications with Stephen Rees-Carter

Read article
Laravel on any Developer Machine with Gitpod image

Laravel on any Developer Machine with Gitpod

Read article
Build your slide deck in Laravel with Simple Slides image

Build your slide deck in Laravel with Simple Slides

Read article
Lazy JSON Pages is a JSON API Scraper for PHP image

Lazy JSON Pages is a JSON API Scraper for PHP

Read article