Using SQS FIFO Queues

Published on by

Using SQS FIFO Queues image

AWS's Simple Queue Service (SQS) is a great, high-scale, and extremely inexpensive way to get reliable queues.

One minor quibble with SQS is that you can't gurantee what order jobs will be processed in.

If there are 10 jobs in a queue, you don't know which job a queue worker will be given.

For many of us, that's just fine. If you need some jobs to be done in order, you can use Laravels Job Chaining.

However, that doesn't work in all cases.

For example, in Chipper CI, we need to process builds in the order we receive them. We have to wait on incoming webhooks that can come at any time, and so we can't simply add to a Job Chain. Instead rely on SQS FIFO queues.

FIFO Queues

FIFO stands for First In, First Out. That means that the oldest job will always be the first to be given to a queue worker.

The job "behind" the oldest job will not be given to a queue worker until the job before it is complete.

This is great when you need jobs to be processed in order, but this limits scaling queue workers.

If this was all there was to it, you'd never be able to use more than one queue worker nor process more than one job at a time (without creating and somehow managing multiple FIFO queues).

Message Group IDs

Luckily FIFO queues account for this. It turns out that messages order is only guaranteed within a Group ID.

Each job (message) in FIFO queues must have a Group ID. Any jobs that have the same Group ID will be processed in order they are received.

If you assigned a random Group ID to each job, then no order would be maintained as there would be no shared Group ID.

In Chipper CI, each Team can have multiple Projects. Each Project is a git repository that gets built when someone pushes an update to their git repository.

To keep builds in order, we use FIFO queues, creating a Group ID using the Team ID and Project ID. It looks like this:

$build = App\Build::createFromWebhook($payload);
 
$groupId = sprintf('build-%s-%s', $build->project->team_id, $build->project_id);
 
dispatch(
(new ProcessBuild($build))
->onMessageGroup($groupId)
);

SQS FIFO Laravel Package

To help with this, we can use the shiftonelabs/laravel-sqs-fifo-queue Composer package.

This exends the SQS queue message type and adds some functionality.

Most importantly, it gives us the onMessageGroup() method, which allows us to add the message group ID when adding a job (message) to the SQS FIFO queue.

Chris Fidao photo

Teaching coding and servers at CloudCasts and Servers for Hackers. Co-founder of Chipper CI.

Cube

Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

image
Tinkerwell

Version 4 of Tinkerwell is available now. Get the most popular PHP scratchpad with all its new features and simplify your development workflow today.

Visit Tinkerwell
Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Tinkerwell logo

Tinkerwell

The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.

Tinkerwell
No Compromises logo

No Compromises

Joel and Aaron, the two seasoned devs from the No Compromises podcast, are now available to hire for your Laravel project. ⬧ Flat rate of $7500/mo. ⬧ No lengthy sales process. ⬧ No contracts. ⬧ 100% money back guarantee.

No Compromises
Kirschbaum logo

Kirschbaum

Providing innovation and stability to ensure your web application succeeds.

Kirschbaum
Shift logo

Shift

Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Shift
Bacancy logo

Bacancy

Supercharge your project with a seasoned Laravel developer with 4-6 years of experience for just $2500/month. Get 160 hours of dedicated expertise & a risk-free 15-day trial. Schedule a call now!

Bacancy
Lucky Media logo

Lucky Media

Bespoke software solutions built for your business. We ♥ Laravel

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
LaraJobs logo

LaraJobs

The official Laravel job board

LaraJobs
Larafast: Laravel SaaS Starter Kit logo

Larafast: Laravel SaaS Starter Kit

Larafast is a Laravel SaaS Starter Kit with ready-to-go features for Payments, Auth, Admin, Blog, SEO, and beautiful themes. Available with Vue and Livewire stacks.

Larafast: Laravel SaaS Starter Kit
SaaSykit: Laravel SaaS Starter Kit logo

SaaSykit: Laravel SaaS Starter Kit

SaaSykit is a Laravel SaaS Starter Kit that comes with all features required to run a modern SaaS. Payments, Beautiful Checkout, Admin Panel, User dashboard, Auth, Ready Components, Stats, Blog, Docs and more.

SaaSykit: Laravel SaaS Starter Kit
Rector logo

Rector

Your partner for seamless Laravel upgrades, cutting costs, and accelerating innovation for successful companies

Rector

The latest

View all →
Configuring Laravel With Additional Environment Files image

Configuring Laravel With Additional Environment Files

Read article
Real Time Command Line Monitoring With Laravel Top image

Real Time Command Line Monitoring With Laravel Top

Read article
A New Validation Rule and the Ability to Manually Fail a Command in Laravel 11.8 image

A New Validation Rule and the Ability to Manually Fail a Command in Laravel 11.8

Read article
New Proposed Array Find Functions in PHP 8.4 image

New Proposed Array Find Functions in PHP 8.4

Read article
Laracon AU 2024 tickets are now on sale image

Laracon AU 2024 tickets are now on sale

Read article
Dash UI is a Laravel Blade Component Library Inspired by Shopify Polaris image

Dash UI is a Laravel Blade Component Library Inspired by Shopify Polaris

Read article