Amazon releases a summary of the recent S3 outage

Published on by

Amazon releases a summary of the recent S3 outage image

Amazon released a summary of the S3 Service Disruption on US-EAST-1 yesterday. The outage affected many sites, apps, and utilities that rely on the service.

An S3 team member running an established playbook to remove a small number of servers accidently entered a command wrong and a larger set of servers was removed.

The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart. While these were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable.

They go on to say that because of the massive growth over the last several years the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected.

Finally, because the status board relied on S3 they were unable to update the status icons and had to rely on Twitter and the note at the top.

Eric L. Barnes photo

Eric is the creator of Laravel News and has been covering Laravel since 2012.

Filed in:
Cube

Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

image
No Compromises

Join the Mastering Laravel community to level up your skills and get trusted advice

Visit No Compromises
Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Tinkerwell logo

Tinkerwell

The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.

Tinkerwell
No Compromises logo

No Compromises

Joel and Aaron, the two seasoned devs from the No Compromises podcast, are now available to hire for your Laravel project. ⬧ Flat rate of $7500/mo. ⬧ No lengthy sales process. ⬧ No contracts. ⬧ 100% money back guarantee.

No Compromises
Kirschbaum logo

Kirschbaum

Providing innovation and stability to ensure your web application succeeds.

Kirschbaum
Shift logo

Shift

Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Shift
Bacancy logo

Bacancy

Supercharge your project with a seasoned Laravel developer with 4-6 years of experience for just $2500/month. Get 160 hours of dedicated expertise & a risk-free 15-day trial. Schedule a call now!

Bacancy
Lucky Media logo

Lucky Media

Bespoke software solutions built for your business. We ♥ Laravel

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
LaraJobs logo

LaraJobs

The official Laravel job board

LaraJobs
Larafast: Laravel SaaS Starter Kit logo

Larafast: Laravel SaaS Starter Kit

Larafast is a Laravel SaaS Starter Kit with ready-to-go features for Payments, Auth, Admin, Blog, SEO, and beautiful themes. Available with Vue and Livewire stacks.

Larafast: Laravel SaaS Starter Kit
SaaSykit: Laravel SaaS Starter Kit logo

SaaSykit: Laravel SaaS Starter Kit

SaaSykit is a Laravel SaaS Starter Kit that comes with all features required to run a modern SaaS. Payments, Beautiful Checkout, Admin Panel, User dashboard, Auth, Ready Components, Stats, Blog, Docs and more.

SaaSykit: Laravel SaaS Starter Kit
Rector logo

Rector

Your partner for seamless Laravel upgrades, cutting costs, and accelerating innovation for successful companies

Rector

The latest

View all →
Audit MySQL Databases in Laravel With the DB Auditor Package image

Audit MySQL Databases in Laravel With the DB Auditor Package

Read article
The Ability to Prevent Destructive Commands From Running is Coming to Laravel 11 image

The Ability to Prevent Destructive Commands From Running is Coming to Laravel 11

Read article
Configuring Laravel With Additional Environment Files image

Configuring Laravel With Additional Environment Files

Read article
Real Time Command Line Monitoring With Laravel Top image

Real Time Command Line Monitoring With Laravel Top

Read article
A New Validation Rule and the Ability to Manually Fail a Command in Laravel 11.8 image

A New Validation Rule and the Ability to Manually Fail a Command in Laravel 11.8

Read article
New Array Find Functions in PHP 8.4 image

New Array Find Functions in PHP 8.4

Read article