Laravel URL AI Transformer: Extract Structured Data From URLs

Spatie's Laravel URL AI Transformer fetches a list of URLs, sends each page's content to an AI model, and stores the response in your database. You might use it to generate JSON-LD for your own pages, or to pull product details out of a page into fields your app can query. Version 2 of this package builds every transformer as a Laravel AI agent.

The package also includes:

A built-in LdJsonTransformer: it asks the model for schema.org JSON-LD and validates the JSON before it is stored
Command filters: transform-urls --url="*/blog/*" and --transformer="ldJson" accept wildcards, so you can rerun part of a site
Conditional runs: a transformer can define shouldRun() to skip URLs it processed recently, and --force ignores it
Failure tracking: the exception message and trace are stored on the result row, and a TransformerFailed event fires
Regeneration: regenerate() queues a single result again, and regenerateNow() runs it synchronously
Crawling: pair it with spatie/crawler to transform every internal URL on a site
Transformers without AI: override transform() to compute a result yourself, such as a word count

Registering URLs

You register URLs and the transformers to run on them with the Transform class, typically in a service provider:

use Spatie\LaravelUrlAiTransformer\Support\Transform;
use Spatie\LaravelUrlAiTransformer\Transformers\LdJsonTransformer;

Transform::urls(
    'https://example.com/about',
    'https://example.com/products',
)->usingTransformers(new LdJsonTransformer, new SummaryTransformer);

The urls() method also accepts a closure. It only runs when the transformation does, so loading URLs from the database doesn't add a query to every request:

Transform::urls(
    fn () => Article::published()->pluck('url')->toArray()
)->usingTransformers(new LdJsonTransformer);

The transform-urls Artisan command fetches each URL once and dispatches a queued job for every transformer registered on it. You'll need a queue worker running, or you can pass --now to run everything synchronously. Results are stored in the transformation_results table, and you read them back by URL and transformer type:

use Spatie\LaravelUrlAiTransformer\Models\TransformationResult;

$ldJson = TransformationResult::forUrl('https://example.com/about', 'ldJson');

The type defaults to the camel-cased class name without the Transformer suffix, and you can pass the class name instead. Schedule the command in routes/console.php to keep results current.

Writing a Transformer

Run php artisan make:transformer SummaryTransformer to generate a class in app/Transformers. A transformer only needs an instructions() method, which becomes the agent's system prompt. The package sends the page content along as the user prompt:

namespace App\Transformers;

use Spatie\LaravelUrlAiTransformer\Transformers\Transformer;
use Stringable;

class SummaryTransformer extends Transformer
{
    public function instructions(): Stringable|string
    {
        return 'Summarize this webpage content in 3 concise bullet points.';
    }
}

Before the content is sent, the package strips scripts, styles, and HTML tags, collapses whitespace, and cuts the text at 6,000 characters. You can override content() on a single transformer to send something different, or swap the PrepareUrlContentAction in the config to change it for every transformer.

Structured Output

A transformer that implements Laravel AI's HasStructuredOutput contract returns data validated against a schema, which the package stores as JSON:

use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\HasStructuredOutput;
use Spatie\LaravelUrlAiTransformer\Transformers\Transformer;
use Stringable;

class ProductTransformer extends Transformer implements HasStructuredOutput
{
    public function instructions(): Stringable|string
    {
        return 'Extract the product details from this webpage.';
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'name' => $schema->string()->required(),
            'price' => $schema->number()->required(),
            'in_stock' => $schema->boolean(),
        ];
    }
}

For data with an open shape, such as JSON-LD, the docs describe a schema with a single required json string property. You then decode that string in resultFrom() with JSON_THROW_ON_ERROR. Invalid JSON throws, marks the transformation as failed, and lets the queued job retry. The built-in LdJsonTransformer works this way. Laravel News has a longer walkthrough of schemas in Laravel AI, using ticket triage with structured output.

Choosing a Provider and Model

The config file sets a default provider from Laravel AI's Lab enum and a default model. The model can be a model name, Model::Cheapest, or Model::Smartest, and it defaults to the cheapest model on OpenAI. Individual transformers override this with Laravel AI attributes:

use Laravel\Ai\Attributes\MaxTokens;
use Laravel\Ai\Attributes\Model;
use Laravel\Ai\Attributes\Provider;
use Laravel\Ai\Attributes\Temperature;
use Laravel\Ai\Enums\Lab;
use Spatie\LaravelUrlAiTransformer\Transformers\Transformer;
use Stringable;

#[Provider(Lab::Anthropic)]
#[Model('claude-haiku-4-5-20251001')]
#[MaxTokens(1000)]
#[Temperature(0.1)]
class FactsTransformer extends Transformer
{
    public function instructions(): Stringable|string
    {
        return 'Extract the key facts as accurately as possible.';
    }
}

You can also use #[UseSmartestModel] to keep the configured provider and change only the model. When the choice depends on runtime data, define provider() and model() methods instead.

Testing Transformers

To test a transformer, fake the page with Laravel's Http facade and the AI response with fake() on the transformer class:

it('generates summaries for articles', function () {
    Http::fake([
        'https://example.com/article' => Http::response('<html><body>Test article content</body></html>'),
    ]);

    SummaryTransformer::fake([
        "• First point\n• Second point\n• Third point",
    ]);

    Transform::urls('https://example.com/article')
        ->usingTransformers(new SummaryTransformer);

    Artisan::call('transform-urls', ['--now' => true]);

    expect(TransformationResult::forUrl('https://example.com/article', 'summary'))
        ->toContain('•');
});

Structured transformers take an array that matches their schema, such as ProductTransformer::fake([['name' => 'Wireless keyboard', 'price' => 49.99]]).

Installation

The package requires PHP 8.4 and Laravel 12 or 13. Install it, then publish and run the migration:

composer require spatie/laravel-url-ai-transformer

php artisan vendor:publish --tag="url-ai-transformer-migrations"
php artisan migrate

Add your provider's API key to .env, such as OPENAI_API_KEY for the default OpenAI setup. Version 1 used Prism for AI calls, so if you're upgrading, the upgrade guide covers the new config format, transformer methods, and a unique index migration.

You can find the source code in the laravel-url-ai-transformer repository on GitHub, and the full documentation on Spatie's documentation site.