Long running PHP process waiting on shell exec

PHP
1

I have a PHP script that loops through a folder of images, for each image it adds some info to the database and resizes the image by making a shell call to ImageMagick.

The problem is that this script takes a long time to execute since it waits for ImageMagick to resize each image before going onto the next image. While the script is running it seems my website becomes inaccessible, and if I try and access another page I get an Error 504 timeout. So it’s not a good situation if no-one can browse any of my websites while I’m running the script.

I’m running PHP as a fast-cgi process with 2 children, so I’m not sure why any requests sent while the script is running aren’t just sent to the php child process that isn’t handling the script?

Also, I could do with some help in re-writing the script so that it doesn’t wait for the shell commands to ImageMagick to complete. I need the commands to queue up rather than be executed in parallel as otherwise I’ll go over my memory limit. I also need to create a log of any output from the shell commands. If possible, it would also be great if I could get some sort of progress indication.

At the moment I am stuck even just getting the shell command output logged. This code

$cmd = 'echo g &>> ./p.txt';
exec($cmd);

just creates an empty file called p.txt, I’m not sure why?

2

You can limit Imagemagick memory usage but I have never done it. Try searching and posting here: Imagemagick forum
While working on some code before I was modifying hundreds of images with Imagemagick and it did not seem to cause a problem. On thing I found was that I was looping through the images and the code did not wait until one image was completed before it started on the next. The server load only reached about 20% from memory.

3

I’m already limiting memory usage of ImageMagick, but I still can’t allow any ImageMagick processes to run in parallel as my memory limit is very tight already just from running PHP and MySQL.

4

When I did the 3,000 image modifications I did them in batches e.g. All images starting with a-d then e-h etc.


foreach ( glob( $dir."[hHiI]{*.jpg,*.JPG}", GLOB_BRACE) as $filename ) {

Can you do all the resizing first then go back and do the mysql part second.

5

The way my script works currently, the resized images are saved using the database row number as part of the filename.

My idea was that I would change my script so that instead of executing the shell commands (and updating the database), I would end up with them all concatenated in one long string. Then PHP can make a shell call to execute that string and exit, e.g. the string might look like

/path/to/php/php script-that-updates-the-database.php --img=/path/to/image-1.jpg; \\
/path/to/ImageMagick/bin/convert \\
/path/to/moved-image/$IMAGE_NAME.jpg \\
/path/to/resized-image/$IMAGE_NAME.jpg; \\
/path/to/php/php script-that-updates-the-database.php --img=/path/to/image-2.jpg; \\
/path/to/ImageMagick/bin/convert \\
/path/to/moved-image/$IMAGE_NAME.jpg \\
/path/to/resized-image/$IMAGE_NAME.jpg; \\

Hopefully you get the idea. $IMAGE_NAME would a shell variable containing the file name, set by script-that-updates-the-database.php.

But so far I can’t even get shell output logged when a command is executed by PHP, let alone get it to execute multiple commands in serial and in the background.

6

After spending some time trying to write a shell script, I thought that actually the easiest way for me to do it is to use my existing PHP script, but just call it via the PHP CLI instead of the webserver.

That way the webserver PHP process(es) are still free to deal with web traffic, while the PHP CLI process gets on with the image processing script.

Regarding redirecting output, I still don’t know why the output isn’t redirected using &> when the command is executed using PHP. I found that using 2>&1 > instead of &>, the output would be redirected correctly when executed from PHP.

Anyway, I found a very helpful blog post that explains how to spawn the php cli process without it being a child of the php process you spawn it from: PHP and long running processes

There seems to be a problem with the php CLI not reading the .user.ini file though. So I had to create a separate php.ini file just for this script, combining my php.ini and .user.ini, and then specifying this file as the php.ini file when calling the php cli.

So in the end my webpage looked like this:

<?php
exec('echo \\'/path/to/php/bin/php -c /path/to/combined-php.ini batch-process.php 2>&1 > log.txt\\' | at now');
echo 'Now processing';

Hope that helps anyone else with this problem.