Laravel Cloud is here! Zero-config managed infrastructure for Laravel apps. Deploy now.

Using SQS FIFO Queues

Published on by

Using SQS FIFO Queues image

AWS's Simple Queue Service (SQS) is a great, high-scale, and extremely inexpensive way to get reliable queues.

One minor quibble with SQS is that you can't gurantee what order jobs will be processed in.

If there are 10 jobs in a queue, you don't know which job a queue worker will be given.

For many of us, that's just fine. If you need some jobs to be done in order, you can use Laravels Job Chaining.

However, that doesn't work in all cases.

For example, in Chipper CI, we need to process builds in the order we receive them. We have to wait on incoming webhooks that can come at any time, and so we can't simply add to a Job Chain. Instead rely on SQS FIFO queues.

FIFO Queues

FIFO stands for First In, First Out. That means that the oldest job will always be the first to be given to a queue worker.

The job "behind" the oldest job will not be given to a queue worker until the job before it is complete.

This is great when you need jobs to be processed in order, but this limits scaling queue workers.

If this was all there was to it, you'd never be able to use more than one queue worker nor process more than one job at a time (without creating and somehow managing multiple FIFO queues).

Message Group IDs

Luckily FIFO queues account for this. It turns out that messages order is only guaranteed within a Group ID.

Each job (message) in FIFO queues must have a Group ID. Any jobs that have the same Group ID will be processed in order they are received.

If you assigned a random Group ID to each job, then no order would be maintained as there would be no shared Group ID.

In Chipper CI, each Team can have multiple Projects. Each Project is a git repository that gets built when someone pushes an update to their git repository.

To keep builds in order, we use FIFO queues, creating a Group ID using the Team ID and Project ID. It looks like this:

$build = App\Build::createFromWebhook($payload);
 
$groupId = sprintf('build-%s-%s', $build->project->team_id, $build->project_id);
 
dispatch(
(new ProcessBuild($build))
->onMessageGroup($groupId)
);

SQS FIFO Laravel Package

To help with this, we can use the shiftonelabs/laravel-sqs-fifo-queue Composer package.

This exends the SQS queue message type and adds some functionality.

Most importantly, it gives us the onMessageGroup() method, which allows us to add the message group ID when adding a job (message) to the SQS FIFO queue.

Chris Fidao photo

Teaching coding and servers at CloudCasts and Servers for Hackers. Co-founder of Chipper CI.

Cube

Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

image
Mastering Laravel

Join the Mastering Laravel community to level up your skills and get trusted advice.

Visit Mastering Laravel
Curotec logo

Curotec

World class Laravel experts with GenAI dev skills. LATAM-based, embedded engineers that ship fast, communicate clearly, and elevate your product. No bloat, no BS.

Curotec
Bacancy logo

Bacancy

Supercharge your project with a seasoned Laravel developer with 4-6 years of experience for just $2500/month. Get 160 hours of dedicated expertise & a risk-free 15-day trial. Schedule a call now!

Bacancy
Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Tinkerwell logo

Tinkerwell

The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.

Tinkerwell
Cut PHP Code Review Time & Bugs into Half with CodeRabbit logo

Cut PHP Code Review Time & Bugs into Half with CodeRabbit

CodeRabbit is an AI-powered code review tool that specializes in PHP and Laravel, running PHPStan and offering automated PR analysis, security checks, and custom review features while remaining free for open-source projects.

Cut PHP Code Review Time & Bugs into Half with CodeRabbit
Join the Mastering Laravel community logo

Join the Mastering Laravel community

Connect with experienced developers in a friendly, noise-free environment. Get insights, share ideas, and find support for your coding challenges. Join us today and elevate your Laravel skills!

Join the Mastering Laravel community
Kirschbaum logo

Kirschbaum

Providing innovation and stability to ensure your web application succeeds.

Kirschbaum
Shift logo

Shift

Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Shift
Lucky Media logo

Lucky Media

Get Lucky Now - the ideal choice for Laravel Development, with over a decade of experience!

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
LaraJobs logo

LaraJobs

The official Laravel job board

LaraJobs
SaaSykit: Laravel SaaS Starter Kit logo

SaaSykit: Laravel SaaS Starter Kit

SaaSykit is a Multi-tenant Laravel SaaS Starter Kit that comes with all features required to run a modern SaaS. Payments, Beautiful Checkout, Admin Panel, User dashboard, Auth, Ready Components, Stats, Blog, Docs and more.

SaaSykit: Laravel SaaS Starter Kit

The latest

View all →
Chargebee Starter Kit for Billing in Laravel image

Chargebee Starter Kit for Billing in Laravel

Read article
Streamline Pipeline Cleanup with Laravel's finally Method image

Streamline Pipeline Cleanup with Laravel's finally Method

Read article
Validate Controller Requests with the Laravel Data Package image

Validate Controller Requests with the Laravel Data Package

Read article
Transform JSON into Typed Collections with Laravel's AsCollection::of() image

Transform JSON into Typed Collections with Laravel's AsCollection::of()

Read article
Auto-translate Application Strings with Laratext image

Auto-translate Application Strings with Laratext

Read article