Running the Laravel Scheduler and Queue with Docker

In Laravel, one of the tricky changes when switching from a virtual server to Docker is figuring out how to run a scheduler and a queue worker. I see this question come up quite a bit when PHP developers are trying to figure out how to use Laravel with Docker.

Should you run them on the host server? Should you run via cron in a Docker container?

There are a couple of ways I recommend running the scheduler command and Laravel queues in Docker, and we’re going to cover the basics of running both with a complete (albeit simple) Docker setup you can use to experiment.

#Multi-Purpose Docker Image

In the context of Docker, you can split your workloads into separate containers and scale them independently. You could have multiple containers running queue workers, one running a scheduler and a handful of containers running your web application. Laravel’s design is monolithic, meaning that your queue jobs, scheduled commands, and HTTP endpoints share a single codebase.

When you start splitting up HTTP traffic, queues, and scheduled commands with Docker, you have to make some decisions about how to build an image for each purpose.

For example, do you define a separate Dockerfile for each context you run your Laravel code? One for your web application, one for your queue, and one for your scheduler?

I would propose that with a bit of clever scripting, we can build a single flexible Docker image that can support all three roles. That means one image to build, that can run as a web server, a schedule runner, or a queue worker.

With Docker you can split up your workloads in new and exciting ways that aren’t possible on a traditional server. While on an Ubuntu server, you’d probably run your web server, queues, and the scheduler command on the same machine(s). With Docker however, it doesn’t make sense to run all of these processes in the same container.

Let’s look at how we can accomplish this with a bash script to run our Docker CMD.

#Project Setup

Before we get into how we can run our application in different roles within Docker, let’s set up a simple Laravel project with Docker using an Apache web server.

We’ll use the official php Docker image as our base image, and Docker Compose to run MySQL and Redis. First, let’s set up the files we need to set up a Docker environment:

laravel new docker-laravel
cd ./docker-laravel
mkdir docker/
touch docker-compose.yml
touch docker/Dockerfile
touch docker/start.sh
touch docker/vhost.conf

#Docker Compose

This post isn’t about using Docker Compose, so if you’re not familiar with that I’d recommend going through the Docker Compose documentation, and my Docker PHP book (there’s also a book-only version) goes through plenty of Docker Compose examples throughout the text.

Here’s the Docker Compose file with the services we’ll need to run our application services:

version: "3"
services:
  app:
    image: laravel-www
    container_name: laravel-www
    build:
      context: .
      dockerfile: docker/Dockerfile
    depends_on:
      - redis
      - mysql
    ports:
      - 8080:80
    volumes:
      - .:/var/www/html
    environment:
      APP_ENV: local
      CONTAINER_ROLE: app
      CACHE_DRIVER: redis
      SESSION_DRIVER: redis
      QUEUE_DRIVER: redis
      REDIS_HOST: redis
 
  redis:
    container_name: laravel-redis
    image: redis:4-alpine
    ports:
      - 16379:6379
    volumes:
      - redis:/data
 
  mysql:
    container_name: laravel-mysql
    image: mysql:5.7
    ports:
      - 13306:3306
    volumes:
      - mysql:/var/lib/mysql
    environment:
      MYSQL_DATABASE: homestead
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: homestead
      MYSQL_PASSWORD: secret
 
volumes:
  redis:
    driver: "local"
  mysql:
    driver: "local"

Take note of the CONTAINER_ROLE environment variable, which will come into play when we start fleshing out the custom Docker script. We also define some development database settings for MySQL and set up volumes to persist our MySQL and Redis data.

In the app service, we mount a volume so that our PHP and frontend code changes reflect immediately during development.

#The Dockerfile

We set up a MySQL, Redis, and Laravel application service which will be built from the docker/Dockerfile we need to update:

FROM php:7.2-apache-stretch
 
COPY . /var/www/html
COPY docker/vhost.conf /etc/apache2/sites-available/000-default.conf
 
RUN chown -R www-data:www-data /var/www/html \
    && a2enmod rewrite

The Dockerfile gets most of its functionality from the official PHP Apache image. We are copying the Laravel code and an Apache Vhost file. The Vhost file override is needed so that we can point the DocumentRoot to /var/www/html/public, which is what Laravel expects.

Then we’re changing the ownership of files to the www-data user for correct permissions in production. Last, we enable mod_rewrite, so that URL rewrites will work.

#The Apache Vhost

Next, we need to fill in the docker/vhost.conf file to point to our public folder. Add the following to docker/vhost.conf, which is copied into the image, overriding the 000-default.conf vhost file:

<VirtualHost *:80>
    DocumentRoot /var/www/html/public
 
    <Directory "/var/www/html/public">
        AllowOverride all
        Require all granted
    </Directory>
 
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

At this point you can build the image and you should be able to get Laravel’s welcome page on localhost:8080:

docker-compose up --build

#Custom Docker CMD Instruction

Our docker/Dockerfile is extending the official PHP Apache image using Debian Stretch. If you look closely at the base Apache image we are extending, you will see a CMD instruction:

CMD ["apache2-foreground"]

We are going to override the CMD instruction in our own Dockerfile, so let’s start out by building a structure for our bash script and using apache2-foreground in the script. We will then extend the script to include roles for a scheduler and a queue.

Update the docker/Dockerfile to look like the following:

FROM php:7.2-apache-stretch
 
COPY . /var/www/html
COPY docker/vhost.conf /etc/apache2/sites-available/000-default.conf
COPY docker/start.sh /usr/local/bin/start
 
RUN chown -R www-data:www-data /var/www/html \
    && chmod u+x /usr/local/bin/start \
    && a2enmod rewrite
 
CMD ["/usr/local/bin/start"]

We’re now copying in a bash script from docker/start.sh into /usr/local/bin/start and making it executable. Last, we override the CMD instruction to run our bash script.

Before we fill in a queue worker, let’s get the apache server working again. Add the following to docker/start.sh:

#!/usr/bin/env bash
 
set -e
 
role=${CONTAINER_ROLE:-app}
env=${APP_ENV:-production}
 
if [ "$env" != "local" ]; then
    echo "Caching configuration..."
    (cd /var/www/html && php artisan config:cache && php artisan route:cache && php artisan view:cache)
fi
 
if [ "$role" = "app" ]; then
 
    exec apache2-foreground
 
elif [ "$role" = "queue" ]; then
 
    echo "Queue role"
    exit 1
 
elif [ "$role" = "scheduler" ]; then
 
    echo "Scheduler role"
    exit 1
 
else
    echo "Could not match the container role \"$role\""
    exit 1
fi

There’s quite a bit in this script. First, we are checking for environments other than local, and running production-like caching. The fact that we can cache config and routes is worth using a custom bash script, even if you don’t plan on running a queue or a scheduler. Be aware that if you accidentally run your container without the APP_ENV=local environment that you’ll need to clear your config cache php artisan config:clear to avoid cached values during development.

In bash, you can set default values for variables, so we make the app role the default if the CONTAINER_ROLE env variable is not set. Note, that we are setting this in the docker-compose.yml file.

Last, the heart of our script, is the logic based on the container role. Right now we just echo the role type for the scheduler and the queue, and we run exec apache2-foreground to start Apache in the app role.

If you rebuild the Docker image and run Docker compose again, the start script should run Apache and serve the Laravel application:

docker-compose down
docker-compose build
docker-compose up

#Running a Scheduler

We can modify our docker/start.sh script to run a scheduler. The neat thing about this is that we can accomplish this with a single Docker image and modify how it works at runtime based on the role.

If you were setting up the Scheduler to run on a traditional server, you’d set up a Cron entry for it like so:

* * * * * php /path-to-your-project/artisan schedule:run >> /dev/null 2>&1

The Cron entry is defined so that the script will run every 60 seconds. We can emulate this behavior without installing Cron in our Docker image, using bash. The tricky part is that our container must keep a process running in the foreground, otherwise the container will exit.

We can accomplish the same design as the cron using an infinite bash loop:

if [ "$role" = "app" ]; then
 
    exec apache2-foreground
 
elif [ "$role" = "queue" ]; then
 
    echo "Queue role"
    exit 1
 
elif [ "$role" = "scheduler" ]; then
 
    while [ true ]
    do
      php /var/www/html/artisan schedule:run --verbose --no-interaction &
      sleep 60
    done
 
else
    echo "Could not match the container role \"$role\""
    exit 1
fi

If we run a container from our image with CONTAINER_ROLE=scheduler an infinite bash while loop will run in the foreground, and every 60 seconds a new schedule:run command will run in the background (using & at the end).

Running this command in the background is vital for this to work like a Cron. If we run the schedule:run command in the foreground, the script will stop execution and the sleep 60 will not fire until the schedule:run command finishes.

In order to try this out, uncomment the inspire command (or add your own) in the app/Console/Kernel.php file:

/**
 * Define the application's command schedule.
 *
 * @param  \Illuminate\Console\Scheduling\Schedule  $schedule
 * @return void
 */
protected function schedule(Schedule $schedule)
{
    $schedule->command('inspire')
             ->everyMinute();
}

Next, we need to define a scheduler service in the docker-compose.yml file that depends on our www-laravel image:

scheduler:
  image: laravel-www
  container_name: laravel-scheduler
  depends_on:
    - app
  volumes:
    - .:/var/www/html
  environment:
    APP_ENV: local
    CONTAINER_ROLE: scheduler
    CACHE_DRIVER: redis
    SESSION_DRIVER: redis
    QUEUE_DRIVER: redis
    REDIS_HOST: redis

Take note that we are duplicating some environment variables values between the app and scheduler services. You could optimize this with a dedicated Docker compose env_file property, or using Laravel’s .env file. I prefer to use an external Docker environment file for development.

Here’s the entire docker-compose.yml file for reference. We will also add the queue service now, so you don’t have to do that later on:

version: "3"
services:
  app:
    image: laravel-www
    container_name: laravel-www
    build:
      context: .
      dockerfile: docker/Dockerfile
    depends_on:
      - redis
      - mysql
    ports:
      - 8080:80
    volumes:
      - .:/var/www/html
    environment:
      APP_ENV: local
      CONTAINER_ROLE: app
      CACHE_DRIVER: redis
      SESSION_DRIVER: redis
      QUEUE_DRIVER: redis
      REDIS_HOST: redis
 
  scheduler:
    image: laravel-www
    container_name: laravel-scheduler
    depends_on:
      - app
    volumes:
      - .:/var/www/html
    environment:
      APP_ENV: local
      CONTAINER_ROLE: scheduler
      CACHE_DRIVER: redis
      SESSION_DRIVER: redis
      QUEUE_DRIVER: redis
      REDIS_HOST: redis
 
  queue:
    image: laravel-www
    container_name: laravel-queue
    depends_on:
      - app
    volumes:
      - .:/var/www/html
    environment:
      APP_ENV: local
      CONTAINER_ROLE: queue
      CACHE_DRIVER: redis
      SESSION_DRIVER: redis
      QUEUE_DRIVER: redis
      REDIS_HOST: redis
 
  redis:
    container_name: laravel-redis
    image: redis:4-alpine
    ports:
      - 16379:6379
    volumes:
      - redis:/data
 
  mysql:
    container_name: laravel-mysql
    image: mysql:5.7
    ports:
      - 13306:3306
    volumes:
      - mysql:/var/lib/mysql
    environment:
      MYSQL_DATABASE: homestead
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: homestead
      MYSQL_PASSWORD: secret
 
volumes:
  redis:
    driver: "local"
  mysql:
    driver: "local"

We also need to build our image so that the updates to the docker/start.sh file are part of the image build:

docker-compose down
docker-compose build
docker-compose up

If everything worked, you should see the following output for the scheduler container:

laravel-scheduler | No scheduled commands are ready to run.
laravel-scheduler | Running scheduled command: '/usr/local/bin/php' 'artisan' inspire > '/dev/null' 2>&1
laravel-scheduler | Running scheduled command: '/usr/local/bin/php' 'artisan' inspire > '/dev/null' 2>&1
laravel-scheduler | Running scheduled command: '/usr/local/bin/php' 'artisan' inspire > '/dev/null' 2>&1

Another thing to note about running a scheduler in Docker: as of Laravel 5.6 you can run the onOneServer() command which indicates that the command only runs on one server. You need to use the Memcached or Redis cache driver to support this feature.

#Queue Worker

The last thing we’ll update our Image to support is running a queue worker. We’re going to use Redis, so the predis/predis composer dependency is needed. Run this command locally to install predis:

composer require predis/predis

Next, update the docker/start.sh script to run a queue worker if the $role=queue:

if [ "$role" = "app" ]; then
 
    exec apache2-foreground
 
elif [ "$role" = "queue" ]; then
 
    echo "Running the queue..."
    php /var/www/html/artisan queue:work --verbose --tries=3 --timeout=90
 
elif [ "$role" = "scheduler" ]; then
# ...

Here’s the final start script file in full:

#!/usr/bin/env bash
 
set -e
 
role=${CONTAINER_ROLE:-app}
env=${APP_ENV:-production}
 
if [ "$env" != "local" ]; then
    echo "Caching configuration..."
    (cd /var/www/html && php artisan config:cache && php artisan route:cache && php artisan view:cache)
fi
 
if [ "$role" = "app" ]; then
 
    exec apache2-foreground
 
elif [ "$role" = "queue" ]; then
 
    echo "Running the queue..."
    php /var/www/html/artisan queue:work --verbose --tries=3 --timeout=90
 
elif [ "$role" = "scheduler" ]; then
 
    while [ true ]
    do
      php /var/www/html/artisan schedule:run --verbose --no-interaction &
      sleep 60
    done
 
else
    echo "Could not match the container role \"$role\""
    exit 1
fi

If you create a test queue job and dispatch it (you could dispatch it from within the welcome route for example), you will see something like this output in your docker-compose logs during queue processing:

laravel-queue | [2018-04-25 06:45:59][qv5tfCgvUsKxVTTvq0SsF6gx9VLwnd6H] Processing: App\Jobs\ExampleJob
laravel-queue | [2018-04-25 06:45:59][qv5tfCgvUsKxVTTvq0SsF6gx9VLwnd6H] Processed:  App\Jobs\ExampleJob

#Learn More

If you want to learn more about developing with Docker and PHP, including Laravel, check out my book Docker for PHP Developers. If you don’t want the starter Laravel code, you can also get the book only version. Laravel News readers get a discount on both versions of the book! You can learn more about the book here.

To learn more about running custom start scripts, I would suggest that you read more about the CMD Dockerfile instruction from the official Docker reference on how it works.

The offical Docker PHP image has some excellent documentation, and I suggest you read through the documentation to learn how quickly you can use the PHP image with your Docker projects. We used Apache, but you can use PHP-FPM in a variety of different image types (i.e., Alpine, Debian Stretch, Debian Jesse).

The links included are affiliate links which means if you decide to buy Laravel News gets a little kickback to help run this site.