Take the Pain Out of Data Imports with Laravel Ingest

Laravel Ingest by Robin Kopp is a configuration-driven ETL (Extract, Transform, Load) package that replaces one-off import scripts with declarative importer classes. It handles files from a few hundred to tens of millions of rows by processing them through PHP Generators and Laravel Queues, keeping memory usage consistent regardless of file size.

#Main Features

• Declarative importer classes using a fluent IngestConfig builder
• Automatic resolution of BelongsTo and BelongsToMany relationships
• Duplicate handling strategies: SKIP, CREATE, UPDATE, and UPDATE_IF_NEWER
• Dry-run mode to validate imports before writing to the database
• Failed row tracking with downloadable CSV exports
• Column aliasing to map varying header names to a single field
• Dynamic model resolution based on row data
• Import sources: file upload, filesystem disks (including S3), URL, FTP, and SFTP
• Auto-generated Artisan commands and REST API endpoints per importer

#Defining an Importer

After installing the package and running migrations, you create an importer class that implements IngestDefinition and returns an IngestConfig. By convention, these live in the App\Ingest namespace:

namespace App\Ingest;
 
use App\Models\Product;
use LaravelIngest\Contracts\IngestDefinition;
use LaravelIngest\DTOs\IngestConfig;
use LaravelIngest\Enums\DuplicateStrategy;
use LaravelIngest\Enums\SourceType;
 
class ProductImporter implements IngestDefinition
{
    public function getConfig(): IngestConfig
    {
        return IngestConfig::for(Product::class)
            ->fromSource(SourceType::UPLOAD)
            ->keyedBy('sku')
            ->onDuplicate(DuplicateStrategy::UPDATE)
            ->map('Product Name', 'name')
            ->relate('Category', 'category', Category::class, 'slug')
            ->validate([
                'sku' => 'required|string',
                'Product Name' => 'required|string|min:3',
            ]);
    }
}

Register the importer in your AppServiceProvider using the package's tag:

use LaravelIngest\IngestServiceProvider;
 
$this->app->tag([ProductImporter::class], IngestServiceProvider::INGEST_DEFINITION_TAG);

#Running Imports

Once registered, the package exposes both an Artisan command and an HTTP endpoint for each importer.

Via CLI:

php artisan ingest:run product-importer --file=products.csv

Via API (multipart form upload):

POST /api/v1/ingest/upload/product-importer

For dry runs, append the --dry-run flag to the Artisan command to validate the file and surface any errors without touching the database.

#Monitoring

The package includes several Artisan commands for checking on running or completed imports:

php artisan ingest:list              # List registered importers
php artisan ingest:status {id}       # Show progress and row statistics
php artisan ingest:cancel {id}       # Stop an in-progress import
php artisan ingest:retry {id}        # Reprocess only the failed rows

Equivalent REST endpoints are also available:

• GET /api/v1/ingest — recent runs
• GET /api/v1/ingest/{id} — status and statistics
• GET /api/v1/ingest/{id}/errors/summary — aggregated error breakdown
• GET /api/v1/ingest/{id}/failed-rows/download — CSV of rows that failed

#Events

The package dispatches events throughout the import lifecycle — IngestRunStarted, ChunkProcessed, RowProcessed, IngestRunCompleted, and IngestRunFailed — which you can listen to for notifications or custom side effects.

You can find Laravel Ingest on GitHub and read the full documentation at the Laravel Ingest docs.