Communicate with third-party applications using queues in your Laravel web application

We've communicated from our web applications to other systems over the years using various means but we've pretty much settled on queues. Our queue of choice is Amazon SQS - Laravel has built in support for SQS and being a managed service, not having to worry about the infrastructure supporting our queues helps me sleep a little better at night.

Background

I frequently partner with a local company to develop industrial solutions. We've done some really neat projects that range from simple data acquisition to vision inspection.

The software written to interact with their hardware also has a wide variety of means to communicate with other systems.

What's so great about queues?

Queues provide a way to decouple communication between systems. Sure, there are other means and some of them work quite well. But with disparate-system communication, knowing that, regardless of the state of the application you are trying to communicate with (it could have lost power, too busy to accept messages, or just some other reason it can't communicate synchronously), your message will wait in the queue until the system can process it. You don't have to worry as you would with synchronous communication.

Another benefit is, if the hardware is running behind an organization's firewall, it can still communicate with other systems regardless of where they are running provided the network supports stateful communication. This is true because, even with a queue on the public internet, the request to check for new messages that is sent to the queue is initiated from within the network and responses to that request can make their way back to the application. Using other means of communication that require messages be "pushed" to the application running on the network opens up all kinds of security risks that most organizations don't want to consider.

Using queues in Laravel

Before venturing into using queues in Laravel, I highly recommend reading the official Laravel documentation on queues. As a practice, I read through all of the Laravel documentation periodically and it's the first place I go to find answers. If you don't refer to the documentation on a regular basis, you should start.

Sending messages to a queue

Because Laravel supports queue processing out of the box, sending a message to the queue from Laravel is simple. However, because the web application is queuing a job that will be consumed by another application rather than the web app, instead of using the push method on the Queue facade, we use the Queue::pushRaw method. The pushRaw method expects a string as the first parameter - we always send a payload with this signature:

$payload = json_encode([
  'job' => 'SetTeststandMode',
    'data' => [
      'mode' => 'start'
    ]
]);

Queue::pushRaw($payload, $teststand->queue_url);

When the hardware processes the message, the job value is used to delegate to the proper routine and the data value is decoded and used as parameters during processing. We routinely have multiple test stands so our Laravel Stand model has a queue_url field where we store the SQS queue URL.

Receiving messages on a queue

Receiving a message on a queue from a third-party application is a little different. I saw a few things on Twitter from Taylor Otwell that suggested it could be done. Of course, you have the queue listener running for the queue you are receiving messages on. I use Laravel Forge for managing my servers so setting up a queue worker that is monitored by Supervisor is a cinch.

If the hardware needs to get a message to the web app, the hardware places a json-encoded message on the queue of which the body follows the same format as the messages sent to the hardware using the Queue::pushRaw method. In this case, we use the artisan command make:job and I simply change the default handle method to a fire method. If the message has the correct job (including the namespace) and a data attribute, Laravel is happy to pick up the message and process it. An example of a message from the hardware to the web app is:

"{"job":"App\\Jobs\\CloseLot","data":{"lot_number":"A12345-678"}}"

In the fire method, access the payload by using $job->payload().

Things to watch out for

• You will want to use Amazon SQS FIFO (first-in-first-out) queues if sequence is important or if you can't handle duplicate messages in your system.

Hope you find this helpful if you ever need to use queues for communication between Laravel and your other systems.

@ballenkidd