Serverless Search using Laravel and the Typesense Cloud

Modern web applications have become more complex over the years as user requirements continue to push for more performance, less latency, and more features. As a developer, choosing Laravel is easy. But when you get beyond the application platform, you have to be thinking about data.

Does the database fit my needs? Will it scale? Can I report from it? And most systems require a search feature. What if the database you chose was AWS DynamoDB? The Laravel ecosystem has libraries for using Eloquent to store data in DynamoDB. But since it's a simple key/value datastore, searching is limited to indexes and expensive filters.

This is where a purpose-built search database like Typesense really shines. It handles those tough searches, ranks, and scores the results, and abstracts that complexity into an API for a developer to use. But how would you populate this secondary store? Traditional approaches would say to use Scout. But in this article, I want to demonstrate a more AWS native way by using DynamoDB's Stream capabilities and AWS Lambda to keep our primary DynamoDB table in sync with a Typesense collection.

#Disclosure

But before we begin and for disclosure, Typesense sponsored me to experiment with their product and report my findings. They have rented my attention, but not my opinion. Here is my unbiased view of my experience as a developer when integrating search into a Laravel Application that includes AWS DynamoDB and Typesense.

#Architecture

Before diving into the code, let's have a look at a macro view of our solution. Think of a very traditional Laravel application which has some Blade views, Controllers, and a simple Todo model. The user can list the Todos, create new ones, and then search on the name and description. For storage, we are going to use DynamoDB and Typesense's Cloud option.

Let's dive in!

#The Build

#Todo Model

Let's jump in with our base level Todo model. I find that working from the bottom up helps stitch things together until I've got the full picture like in the architecture diagram above.

<?php
 
 
namespace App\Models;
 
 
use Illuminate\Database\Eloquent\Factories\HasFactory;
use BaoPham\DynamoDb\DynamoDbModel;
 
 
class Todo extends DynamoDbModel
{
    use HasFactory;
 
 
    protected $primaryKey = 'id';   // required
}

Notice that I'm extending my model from DynamoDbModel. This comes from the widely popular and well-maintained library in this GitHub repository..

What's nice about this library is that I can use the same Laravel Eloquent methods that I'm used to. In the TodoController, the save looks like it normally would even though I'm using DynamoDB.

public function store(request $request): redirectresponse
{
    $todo = new todo;
    $todo->id = Uuid::uuid4()->toString();
    $todo->name = $request->name;
    $todo->description = $request->description;
    $todo->save();
    return redirect('/todos')->with('status', 'todo data has been inserted');
}

#Typesense Cloud and Collection

Before looking at connecting DynamoDB to Typesense, I want to show you how easy and powerful using the Typesense Cloud offering is. You could 100% run Typesense yourself and if you are self-hosting or VM-hosting your Laravel app, that might be the approach you want to take. But if you are deploying on AWS and using DynamoDB, a Cloud-hosted Typesense makes a ton of sense.

What I really enjoy about the cloud offering is that I can manage the cluster at an abstracted level. I set things like memory, nodes, whether I want high-availability, and high disk performance and Typesense takes care of the rest. All that I need to do from there is generate some API keys, attach permissions, and my clients are free to perform the operations needed.

Cluster configuration ends up looking like the below image.

With the cluster in place, other Typesense operations are opened up. I must define a Collection, with a document schema, and then I can begin inserting documents. For instance, here's how searching our Todos Collection shows up.

#Connecting DynamoDB to Typesense

With our Typesense Cloud Cluster established, let's talk about syncing with DynamoDB. When building systems with Cloud native technologies like DynamoDB, sometimes it makes more sense to stay in that ecosystem to do additional processing like syncing data with DynamoDB and Typesense.

If you aren't familiar with AWS, DynamoDB offers a set of capabilities to respond to changes in real time via Streams. A DynamoDB stream can be read from AWS Kinesis or a Lambda function. For this example, a Lambda function is the route I'm going to take. It's simple, isolated, and I don't need the extensibility that Kinesis provides.

DynamoDB offers the option to read changes in a few varieties. I can read just Key changes, New Images, or New and Old Images. I've chosen to just read New Images so that I get the full DynamoDB item vs having to go back and do a double read to hydrate the item later. For this implementation, I'm using Golang to read from the stream and populate Typesense. I find Golang to be a solid choice when doing event-driven work in AWS Lambda.

Without diving into how it all works (there's a repository link at the end), here are the Typesense specific bits. Another solid plus for Typesense is that there are plenty of options when working with their API from an SDK standpoint.

// New Client
client = typesense.NewClient(
    typesense.WithServer(url),
    typesense.WithAPIKey(apiKey))
// Write a Document (Upsert)
result, err := client.Collection("todos").Documents().Upsert(ctx, typesenseTodo)

All the above has to be triggered by a user action. When the user creates a new Todo like in the UI image below, the Controller will create a new model and save it to the database.

Again, note that I'm using Eloquent to perform the save.

public function store(request $request): redirectresponse
{
    $todo = new todo;
    $todo->id = Uuid::uuid4()->toString();
    $todo->name = $request->name;
    $todo->description = $request->description;
    $todo->save();
    return redirect('/todos')->with('status', 'todo data has been inserted');
}

#Searching the Collection

There are some great resources for syncing Typesense when using Scout. However, what I love about building things is that there are usually multiple ways of solving a problem. And what I really enjoy with the DynamoDB pattern is that everything happens in its own isolated space. The write operation to DynamoDB happens, and the user is returned control from the API call. The synchronization process happens asynchronously from the user's action. By having the Stream trigger the Lambda function, that workload is also isolated.

With that said, using the Scout integration, my Todo model would expose search functions. That isn't what I'm going to show below. I need to use the same Typesense API that I used in the Lambda function, but instead of Golang, I'm going to use good ole PHP.

#Laravel Provider

To use the client in a consistent and reliable fashion, let's create a Provider so that I can inject it into my TodoController.

php artisan make:provider SearchProvider

<?php
 
 
namespace App\Providers;
 
 
use Illuminate\Support\ServiceProvider;
use Symfony\Component\HttpClient\HttplugClient;
use Typesense\Client;
 
 
class SearchProvider extends ServiceProvider
{
    public function register(): void
    {
        $this->app->singleton(Client::class, function ($app) {
            $client = new Client(
                [
                    'api_key' => '<API Key>',
                    'nodes' => [
                        [
                            'host' => '<Typesense Host>',
                            'port' => '443',
                            'protocol' => 'https',
                        ],
                    ],
                    'client' => new HttplugClient(),
                ]
            );
            return $client;
        });
    }
}

#Injecting and Searching

With a SearchProvider established that exposes the Typesense Client, I can now begin to handle search operations from my Web App. I start by injecting the Client into my controller.

private Client $client;
 
 
public function __construct(Client $client)
{
    $this->client = $client;
}

Once configured, I can then handle search operations in a controller method.

public function search(request $request): view
{
    $search = '';
    if ($request->search) {
        $search = $request->search;
    }
 
 
    $values = $this->client->collections["todos"]->documents->search(
        [
            'q' => $search,
            'query_by' => 'name,description',
            'sort_by' => 'created_at:desc',
        ]
    );
 
 
    $searched = [];
 
 
    foreach ($values['hits'] as $todo) {
        $item = $todo['document'];
        $t = new Todo;
        $t->id = $item['id'];
        $t->name = $item['name'];
        $t->description = $item['description'];
        $t->created_at = $item['created_at'];
        $t->updated_at = $item['updated_at'];
        array_push($searched, $t);
    }
 
 
    log::debug($searched);
    return view('todo')->with(['todos' => $searched]);
}

Notice in the query_by property of the search criteria that I'm going across multiple fields. Doing something like this in DynamoDB would be tough to accomplish but with Typesense, it's extremely simple. The search parameters allow for even further customization such as supplying the search criteria and the sort order. What I like about using Typesense is that I'm not having to prepend or append % to make some kind of wildcard syntax. Typesense handles that for me. It also handles things like typographical errors which I'll show you below.

In addition, you get what hit, the score, and some other useful items in the result. That extra data gives you the ability to produce richer UIs and user experiences.

When searching for the word Demonstration I correctly see the one record that I created a few paragraphs ago. That all makes sense. I could also have used Demo or stration or any other contains type clause.

However, what about when a user makes a mistake. They meant to search for Demo, but they typed Deom. Wouldn't it be nice if your search provider could make that correction? Well Typesense does!

These types of features are the reason that you choose a purpose-built provider when storing your data.

#Solution Recap

And that wraps up the code and the user experience. To quickly recap what we've built:

• A PHP Laravel Application
• Create Todos
  • Store in DynamoDB
  • Trigger synchronization with Typesense via AWS Lambda
• Search Todos
  • Use the PHP Typesense Client as a Service Provider
  • Inject into the Controller for shared reuse

#Thoughts and Impressions

First off, I started building for the web in the late 90s when PHP was competing with Perl's CGI. I'm a huge fan of the community and ecosystem, and it continues to get better and better each year. Laravel adds something magical to the full application experience. I love that I can mix and match components and attach just the things I want based upon my user requirements.

Which is precisely why including Typesense into a project is both so simple and so powerful. Laravel gives me the power through Eloquent to save my data where I want, and then I can make decisions about what to do from there.

My thoughts on using Laravel with DynamoDB and Typesense can be summarized like this:

1. Saving models with Eloquent is seamless
2. Using AWS DynamoDB Streams fits into my desire to build Cloud-Native solutions that scale and meet the highest level of service demands
3. Using the PHP Typesense client as a Provider was a breeze and injecting it via Dependency Injection into the Controller is exactly how I'd expect it to work.
4. The Typesense API is clean, easy to use, and powerful. The fact that I get multiple field searching, hits, ranks, and sort is just amazing. And add in that I can run the server as a managed component in their cloud is awesome.

The one thing that I wished the Typesense cloud offered was a purely serverless and consumption-based model. With DynamoDB and Lambda, I only pay for the resources I consume and that's either per data written/read or time spent computing. As opposed to the Typesense cloud where I pay for capacity upfront, which might leave unused capacity that I'm wasting during my billing period. It's on me as the builder to manage that and effectively using the resources correctly.

With all that said, that isn't a reason I wouldn't use the Typesense cloud. It works amazingly well and much easier than running the service myself.

#Wrapping Up

Thanks so much for reading about an alternative approach to saving and synchronizing a Laravel-based web application with a Typesense search datastore. I've run through a good bit of code and configuration of which I can't possibly demonstrate in an article.

So here are the repositories that I worked from. Feel free to clone and run for yourself. If you've got any feedback, please leave them in the repositories.

• PHP Laravel Code
• DynamoDB/Lambda/Typesense Code

Building software for customers is my life's work. I enjoy seeing delighted users that don't have to worry or think about the complexity that is lurking just under the surface. Choosing Laravel, AWS, and Typesense are great foundational decisions from which to start a build and accomplish those abstractions.

They come with rock solid platforms, libraries, and outstanding communities. It's never been a better time to be a part of the PHP and Laravel community!

Thanks again for reading and happy building!