PHP

IronMQ and Laravel: Delays and Retries

Rajiv Seelam
Share

Previously, we saw how to use Iron push queues with Laravel. All we needed to do was set up an Iron account, add a subscriber URL, push a message to queue, and receive the message.

The way Laravel supports Iron push queues out-of-the-box is amazing, but there are always limitations. In this article we focus on those limitations and learn to tackle them.

Scenarios

There are three possible scenarios:

  1. You received the message and you finished the job successfully.

    • You need provision to tell Iron that the job was completed successfully and it needs to delete it from the queue.

  2. You received the message and something went wrong. Basically, your job was not successful.

    • In this case we should tell Iron that we failed and it should retry or if you couldn’t tell Iron anything, it should be clever enough to retry the job i.e., push the same message again to your subscriber after some time.

  3. You received the message, but your job is a long running process (something which takes more than a minute)

    • Iron should wait longer before it resends the message.

These are all practical scenarios and you will face them on a daily basis. And yes, you need to cover these cases in your applications.

It’s possible to use Iron to its full power if we use the Iron MQ PHP Library. During setup we included "iron-io/iron_mq": "1.4.6" in our composer file. The code for the same is hosted at https://github.com/iron-io/iron_mq_php. Let’s take a peek into the documentation and try to understand how to use this library.

We initiate a queue as follows:

$ironmq = new IronMQ(array(
        "token" => 'XXXXXXXXX',
        "project_id" => 'XXXXXXXXX'
    ));

And this is how we post to a queue:

$ironmq->postMessage($queue_name, "Test Message", array(
        'timeout' => 120,
        'delay' => 2,
        'expires_in' => 2*24*3600 # 2 days
    ));

To turn a queue into a push queue (or create one) all we have to do is post to the queue with options like a list of subscribers, push_type etc. Using the library you can do the following:

Note: If a queue doesn’t have any subscribers it is a pull queue.

$params = array(
        "push_type" => "multicast",
        "retries" => 5,
        "subscribers" => array(
            array("url" => "http://your.first.cool.endpoint.com/push"),
            array("url" => "http://your.second.cool.endpoint.com/push")
        )
    );
    
    $ironmq->updateQueue($queue_name, $params);

That’s it!

That’s exactly what Laravel does when you run the artisan queue:subscribe command. To recap, after you do the necessary setup in the queue.php file (filling it with project_id, token and queue_name), you run a command to register your subscribers:

php artisan queue:subscribe queue_name http://foo.com/queue/receive

This command runs the following code:

/vendor/laravel/framework/src/Illuminate/Queue/Console/SubscribeCommand.php

public function fire()
    {
		$iron->getIron()->updateQueue($this->argument('queue'), $this->getQueueOptions());
	}

Some content is excluded from the file for brevity.

This is how Laravel converts our queue into a push queue. This is the same way it’s done when using the Iron MQ library.

Using the Iron MQ PHP Library

  1. Create a new controller IronController in app/controllers

    <?php
        
        class IronController extends Controller {
        
            protected $ironmq;

        	public function __construct()
        	{
        		$this->ironmq = new IronMQ(array(
        					    "token" => 'XXXXXXXXXXXXXXXXXXX', //Your token from dashboard
        					    "project_id" => 'XXXXXXXXXXXXXXX' //Your project ID from dashboard
        					));
        	}
        
        }

    app/controller/IronController.php

  2. Before we create our methods, let’s create routes to understand what our controller will do:

    Route::get('iron/create','IronController@create'); //Create a Queue and add subscribers
        Route::get('iron/push','IronController@push'); //Push a message to Queue
        Route::get('iron/status/{id}','IronController@status'); //Know the status of the message we pushed
        Route::post('iron/receive','IronController@receive'); //Receive the message and process it

    app/routes.php

  3. Now, let’s write a method to create a queue with a subscriber

    public function create()
    	{
    		$params = array(
    		    "subscribers" => array(
    		        array("url" => url('iron/receive'))
    		    )
    		);
    
    		$this->ironmq->updateQueue('testing', $params);
    
    		return 'Push Queue Created';
    	}

    app/controller/IronController.php

  4. You need to visit http://953ffbb.ngrok.com/iron/create i.e., use your ngrok url appended by /iron/create. Note: Your ngrok url will be different.

    At this point you can go to Iron DashBoard -> MQ -> Queues -> Find a queue with name ‘testing’ -> Push Queues and make sure your subscriber is listed there similar to http://953ffbb.ngrok.com/iron/receive.

  5. Let’s add methods to push a message onto a queue and check the status.

    Method to push a message:

    public function push()
        {
        	//Just some data you want to pass
        	$data = array(
        		'first_name' => 'Rajiv', 
        		'last_name'  => 'Seelam'
        	);
        
        	//Convert data into a string so that we can pass it
        	$data = serialize($data);
        
        	//Post the message
        	$job = $this->ironmq->postMessage('testing', $data);
        
        	Log::info(__METHOD__.' - Message pushed with job_id : '.$job->id);
        
            return Redirect::to('iron/status/'.$job->id);
        }

    Method to check the status of a message:

    public function status($id)
	{
	$status = $this->ironmq->getMessagePushStatuses('testing',$id);
	dd($status);
	}

    app/controller/IronController.php

  6. We need a method to receive the message when we push.

    public function receive()
        {
        	$req = Request::instance();
    
    		$jobId = $req->header('iron-message-id'); //Get job id 
    
    		$data = unserialize($req->getContent()); //Get content and unserialize it
    
    		Log::info(__METHOD__.' - Job Received From Iron with Job ID : '.$jobId);
    
    		return Response::json(array(),200);
    	}

    app/controller/IronController.php

At this point you should understand that this is the method which will receive the message because in the create method we listed url('iron/receive') as the subscriber (check the routes file to confirm this url routes to this method).

Keep your app/storage/logs/laravel.log open to see what is happening.

Now, if you visit http://953ffbb.ngrok.com/iron/push, you should see a message similar to the following in your browser:

array (size=1)
          0 => 
            object(stdClass)[184]
              public 'retries_delay' => int 60
              public 'retries_remaining' => int 3
              public 'retries_total' => int 3
              public 'status_code' => int 200
              public 'status' => string 'deleted' (length=7)
              public 'url' => string 'http://953ffbb.ngrok.com/iron/receive' (length=38)
              public 'id' => string '6019569765635787179' (length=19)

And your logs should be similar to :

[2014-05-31 12:45:04] production.INFO: IronController::push - Message pushed with job_id : 6019569765635787179 [] []
[2014-05-31 12:45:05] production.INFO: IronController::receive - Job Received From Iron with Job ID : 6019569765635787179 [] []

app/storage/logs/laravel.log

These logs confirm our activity. Let’s summarize what we did here :

  • We pushed a message to a queue called ‘testing’
  • Then we checked the status of the message on the queue
  • We received the message and logged something.

Delays and Retries

Now, let’s go a bit deeper. Open your Iron dashboard and go to the queues page (Click on MQ) -> Open testing queue and check ‘Push Queues’, you will see the list of subscribers and under ‘Push Information’ you will see ‘Retries’ and ‘Retries Delay’.

What do these mean?

Retries: The default value is 3, which means Iron will retry the message 3 times.

Retries Delay: The default value is 60, which means Iron will by default push the message again to subscribers after 60 seconds if it thinks the message was not processed successfully.

When does Iron retry a message?:

  1. When it doesn’t get a response from your application:

    • Your application failed to give a response because of some error.
    • Your application is still processing the job, a long running process.

  2. When it gets an error in response:

    • Your app responded with 4xx or 5xx error.

  3. When you send a 202 response:

    • You are asking Iron to resend the message after a delay.

If you check the output of status (see above) you will notice the following line:

public 'status' => string 'deleted'

This is how we will get to know the status of the message we pushed. The possible values for status (currently) are:

  1. retrying – The job will be sent to endpoint i.e., subscriber (as many times as ‘Retries’ is set).
  2. deleted – The job is deleted.
  3. reserved – The job will be retried after timeout.
  4. error – There was an error and nothing more will be done.

You have to send a 200 response to delete a message.

One has to remember that Iron will retry a message after timeout if it doesn’t get a response (it will retry after timeout as many times as you specify in ‘retries_total’).

Can we change these parameters? Of course!

While posting a message you can mention timeout (long running jobs may need more than a minute)

$this->ironmq->postMessages('testing', "data", array(
        "timeout" => 300 // Wait for 300 seconds = 5 minutes
    ));

If you want to decrease or increase the number of retries, you have to call the updateQueue method.

$params = array(
	    "retries" => 5
	);

	$this->ironmq->updateQueue('testing', $params);

Can we change the status of a message which is in the queue? Of course! That is how you are supposed to use queues and in fact you are already doing that.

Let’s see how we can alter the status of a message:

  1. retrying – This status is set when you push a message.
  2. deleted – This status is set when the subscriber responds with a 200.
  3. reserved – This status is set when the subscriber responds with a 202.
  4. error – If Iron exhausted number of retries and if it still doesn’t get a 200 response, the status is set to error.

Note: When your application responds with 4xx or 5xx errors (which usually means something went wrong with the server) Iron waits longer than the mentioned timeout.

Conclusion

Push queues aren’t rocket science when you look at them step by step. Try it yourself and tell us what you think in the comments below! I sincerely hope this helped you understand this often intimidating topic. I highly recommend you to read: http://dev.iron.io/mq/reference/push_queues/. Note that you can find the the source code for this article at: https://github.com/rajivseelam/laravel-queues.

Thanks for reading!