How to process large CSV files with Laravel

Last updated on by

How to process large CSV files with Laravel image

Dealing with hefty CSV files is pretty standard in the business world, especially when you've got loads of data to analyze, report on, or move around. If you're using Laravel and need to process large CSV files, you've come to the right place. We will guide you through the smoothest way to handle this task without causing a traffic jam in your application's performance.

Memory and Performance

First off, let's talk about the elephant in the room: memory and performance. Chugging through a massive CSV can be a memory hog and could slow down your app. Sure, you might think about just cranking up the memory limit or extending the timeout period. But let's be honest, that's like putting a band-aid on a leaky pipe – not the best solution.

Enter Simple Excel by Spatie

Instead of the band-aid approach, we're going to use a nifty package called Simple Excel by Spatie. If you're nodding because you expected Spatie to have a solution, you're not alone.

composer require spatie/simple-excel

Assuming you've got your CSV file ready to go, we'll use SimpleExcelReader to load it up. The cool thing is, by default, it returns you a LazyCollection – think of it as a more considerate way to handle your data without exhausting your server's memory. This means you can process the file bit by bit, keeping your app light on its feet.

$rows is an instance of Illuminate\Support\LazyCollection

Laravel Jobs to the Rescue

Now, before we dive into code, let's set up a Laravel Job to manage our CSV processing.

php artisan make:job ImportCsv

Now here is what our ImportCsv job looks like:

<?php
 
namespace App\Jobs;
 
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Spatie\SimpleExcel\SimpleExcelReader;
 
class ImportCsv implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
/**
* Create a new job instance.
*/
public function __construct()
{
//
}
 
/**
* Execute the job.
*/
public function handle(): void
{
SimpleExcelReader::create(storage_path('app/public/products.csv'))
->useDelimiter(',')
->useHeaders(['ID', 'title', 'description'])
->getRows()
->chunk(5000)
->each(
// Here we have a chunk of 5000 products
);
}
}

Here's the game plan:

  1. Chunking the CSV: We're going to break that file into manageable pieces, giving us a LazyCollection to play with.
  2. Job Dispatching: For each chunk, we'll send out a job. This way, we're processing in batches, which is way easier on your server.
  3. Database Insertion: Each chunk will then be inserted into the database, nice and easy.

Chunking the CSV

With our LazyCollection ready, we'll slice the CSV into chunks. Think of it like turning a gigantic sandwich into bite-sized pieces – much easier to handle.

php artisan make:job ImportProductChunk

For every piece of the CSV, we'll create and fire off a job. These jobs are like diligent workers, each taking a chunk and carefully inserting the data into your database.

<?php
 
namespace App\Jobs;
 
use App\Models\Product;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Database\Eloquent\Model;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Str;
 
class ImportProductChunk implements ShouldBeUnique, ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
public $uniqueFor = 3600;
 
/**
* Create a new job instance.
*/
public function __construct(
public $chunk
) {
//
}
 
/**
* Execute the job.
*/
public function handle(): void
{
$this->chunk->each(function (array $row) {
Model::withoutTimestamps(fn () => Product::updateOrCreate([
'product_id' => $row['ID'],
'title' => $row['title'],
'description' => $row['description'],
]));
});
}
 
public function uniqueId(): string
{
return Str::uuid()->toString();
}
}

Ensuring Uniqueness

One crucial thing to remember is to use $uniqueFor and uniqueId in your jobs. It's like giving each worker a unique ID badge, so you don't accidentally have two people doing the same job – a big no-no for efficiency.

Dispatching Jobs

Back in our ImportCsv job, we'll dispatch a job for each chunk within the each method. It's like saying, "You get a chunk, and you get a chunk – everybody gets a chunk!"

<?php
 
namespace App\Jobs;
 
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Spatie\SimpleExcel\SimpleExcelReader;
 
class ImportCsv implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
 
/**
* Create a new job instance.
*/
public function __construct()
{
//
}
 
/**
* Execute the job.
*/
public function handle(): void
{
SimpleExcelReader::create(storage_path('app/public/products.csv'))
->useDelimiter(',')
->useHeaders(['ID', 'title', 'description'])
->getRows()
->chunk(5000)
->each(
fn ($chunk) => ImportProductChunk::dispatch($chunk)
);
}
}

And there you have it! Your chunks are off to be processed independently, without any memory drama. If you're in a rush, just add more workers, and like a well-oiled machine, your data will be processed even quicker.

Processing large CSV files in Laravel doesn't have to be a headache. With the right tools and approach, you can keep your application running smoothly while dealing with all that data.

Happy coding!

Lokman Musliu photo

Founder and CEO of Lucky Media. Software developer with expertise in Laravel, Statamic, Next.js, React, and Inertia.

Cube

Laravel Newsletter

Join 40k+ other developers and never miss out on new tips, tutorials, and more.

image
No Compromises

Joel and Aaron, the two seasoned devs from the No Compromises podcast, are now available to hire for your Laravel project.

Visit No Compromises
Laravel Forge logo

Laravel Forge

Easily create and manage your servers and deploy your Laravel applications in seconds.

Laravel Forge
Tinkerwell logo

Tinkerwell

The must-have code runner for Laravel developers. Tinker with AI, autocompletion and instant feedback on local and production environments.

Tinkerwell
No Compromises logo

No Compromises

Joel and Aaron, the two seasoned devs from the No Compromises podcast, are now available to hire for your Laravel project. ⬧ Flat rate of $7500/mo. ⬧ No lengthy sales process. ⬧ No contracts. ⬧ 100% money back guarantee.

No Compromises
Kirschbaum logo

Kirschbaum

Providing innovation and stability to ensure your web application succeeds.

Kirschbaum
Shift logo

Shift

Running an old Laravel version? Instant, automated Laravel upgrades and code modernization to keep your applications fresh.

Shift
LoadForge logo

LoadForge

Easy, affordable load testing and stress tests for websites, APIs and databases.

LoadForge
Paragraph logo

Paragraph

Manage your Laravel app as if it was a CMS – edit any text on any page or in any email without touching Blade or language files.

Paragraph
Lucky Media logo

Lucky Media

Bespoke software solutions built for your business. We ♥ Laravel

Lucky Media
Lunar: Laravel E-Commerce logo

Lunar: Laravel E-Commerce

E-Commerce for Laravel. An open-source package that brings the power of modern headless e-commerce functionality to Laravel.

Lunar: Laravel E-Commerce
Bacancy - Staff Augmentation logo

Bacancy - Staff Augmentation

Leave your web app development hustles to the leading IT Staff Augmentation Service Providers. Choose from an extensive pool of 1050+ developers and give yourself the sigh of success you deserve with Bacancy. Get In Touch Today!

Bacancy - Staff Augmentation
DocuWriter.ai logo

DocuWriter.ai

Save hours of manually writing Code Documentation, Comments & DocBlocks, Test suites and Refactoring.

DocuWriter.ai
Rector logo

Rector

Your partner for seamless Laravel upgrades, cutting costs, and accelerating innovation for successful companies

Rector

The latest

View all →
Use Google's Gemini AI in Laravel image

Use Google's Gemini AI in Laravel

Read article
PhpStorm is getting a brand new terminal image

PhpStorm is getting a brand new terminal

Read article
Six Essential Plugins for Visual Studio Code image

Six Essential Plugins for Visual Studio Code

Read article
Modularize Your Laravel Application With the Modular Package image

Modularize Your Laravel Application With the Modular Package

Read article
Handle Money Transactions in Eloquent with Laravel Wallet image

Handle Money Transactions in Eloquent with Laravel Wallet

Read article
Tempo: The Easiest Way to Work With Dates in JavaScript image

Tempo: The Easiest Way to Work With Dates in JavaScript

Read article