Maximium execution time

I have programmed my self into a deep deep hole and need a bit of advice on how to get out :frowning:

I have some php code that uses glob to read all the image files in a folder, it then runs the files through a shell script, then some extra Image magick code and finaly saves them into 5 different versions in 5 different folders.

So far so good and it works great with the longest exicution time of 1.6 seconds for one image in and 5 modified images saved.

I have tried it on a folder containing 6 images and was set to run it tonight and when I woke up this morning I realised I have a problem:

7380 images * 1.6 seconds = 3.28hours

Basicly I think running this from php is definatly out ?
The shell script is not mine and I no nothing about shell scripts but I assume it could be modified to do all the work and then run through the Putty interface ?

What is the goal of the script? Do you need the images as soon as possible to display on a website? If that’s the case I think you’re better of using the lazy approach, i.e. only then process an image when it’s really needed.

If they have some other purpose and you don’t need them right away I don’t see a problem.

My second idea was as you said ScallioXTX

only then process an image when it’s really needed.
I could check to see if the file exists when its called and if not create the image but there is a reason I can not do this easily.
Also if google etc. visited the site would it then start the process off generating each image as it was visiting the pages ?

Could you explain why you cannot do this easily? I have extended the framework I use (Yii) to do exactly that and it works like a charm. Basically in the code I call a function to generate the image, say generateImage() that generates the image if it does not exist and returns the URL where the generated image can be found. Of course if the image is already created the URL is returned imediately.
Output this URL to the HTML and your good to go.
A side advantage of processing images this way is that you’re not stuck with default resize settings. For example if your website has images 400x400 but you later decide you want to be 500x500 doing it your way you have process all images again. Doing it my way (dynamically generating) you can change the generateImage() function to resize to different dimensions and the function will then resize every image when needed.

Yes, your website would then have to generate all the images.

The problem I have is the server displaying the images does not have imagemagick installed. I have to create the images on another server and link to them. Or create the images on the second server and copy them over to the first server - a long story and this setup has to stay as is.
I have to use imagemagick as GD can not do what I need to do; I installed Imagemagick on the server but it would not work and I do not want to go there again.

Would your method still work if the image was requested from another server ? I suppose I could create a php file on the second server to link to instead of the image and that would check if the image exists and create if not?

Thinking about Google visits etc. When google looks at a page it is not going to ask for all 7380 images in one go but just the 20 or so images on that actual page.

Having a php script run for hours isn’t itself a big deal. But, you don’t want that php script running inside of one of the webserver processes.

I had assumed that it would fail on the maximium exicution time - can you give me some more information or an idea what to look for crmalibu? Would running it from a cron job or via the command line work ?

Yes, a cron job would work.

You want to execute the script using the command line version of php.
http://php.net/manual/en/features.commandline.php
By default, it has no max execution time. Although, you could use the cgi version of php too if you wanted, and just set the time limit to zero(unlimited). Personally I’d still set a time limit, but a big one.

Major things to watch out for:
could possibly be a different version of php, or have different extensions available.
could use a different php.ini file
relative file paths work differently.
certain variables don’t exist(primarily webserver related ones).

Test/develop the script via a shell so you can see some output, especially errors.

Not the most flexible way, but to help you get started, I recommend you use absolute file paths for everything unless you have a good understanding of how relative file paths will work in this environment.

Couple quick examples for you to run via a shell

see some phpinfo


/usr/bin/php -i

or


c:/full/path/to/your/php.exe -i

execute a script


/usr/bin/php /full/path/to/hello_world.php

execute a script, using a specific php.ini


/usr/bin/php -c /full/path/to/php.ini /full/path/to/hello_world.php

Do standard things like crank up error_reporting, and maybe set up some logging to help you along.

I have now run the code from a webpage and apart from problems like # white space and foreign character in the file name it works.

I used this to limit the amount of files processed:

foreach ( glob( $dir."[B]*.jpg") as $filename ) { 

When I have found a way around the filename problems above I will have a go at running the same script through the php command line. If thats OK I will go for the full 7380 files.

Thanks for the tips crmalibu.

I changed all the paths to absolute paths within the php code.

I then tried the code and had a few problems:

1/ /usr/bin/php -i only needs to be php -i ( am I using a different shell ? )
so all I need to run the script is
php /full/path/to/hello_world.php

2/ Somewhere along the line I had an error about unbalenced parenthases and I removed the \ I had on them and that error went.
That was in front of the lines like these but the forum removes them.


( mpr:image -thumbnail x1000 -write $newname_1000 )

3/ Had an error about missing -r and so I added that to the script call.

php -r /home/full_path_to/test.php

4/ I am now stuck with: syntax error near unexpected token `(’

At some point I tried changing all the " to ’ and added #!/usr/bin/php in front of the program but I started losing track of what I had tried and not tried at some point :confused:

The current state of my code:


#!/usr/bin/php
<?php 
	 
// Folder to select images from
$dir = '/home/full_path_to/images/';
 
// Find all jpg images in the pics folder and modify - one letter at a time
foreach ( glob( $dir."[aA]{*.jpg,*.JPG}", GLOB_BRACE) as $filename ) {	 

// Find and set the path for the shell script	 
$path = '/home/full_path_to/cylinderize';

// Default values for shell script
$radius = '140';
$pitch = '10';
$background = 'none';
$vertual_pixel = 'transparent';

// Calculate the length ( height ) of the new image
$size = getimagesize( $filename );
$length = (( $size[1]/$size[0] ) * ( $radius *2 ))*1.2;
if ( $length > '380' ){ $length = '380'; } 

// Create a random name for the tempory file to prevent overwriting
$temp = '/home/full_path_to/temp/'.rand().'.miff';

// Run the shell script to create the image - a tempory image created but we will delete it later
exec(' $path -r $radius -l $length -p $pitch -b $background -v $vertual_pixel $filename $temp'); 

// New image names for the different sizes as we are resizing them all in this one script
$newname_1000 = str_replace('/home/full_path_to/images/', '/home/full_path_to/test_1000/', $filename);
$newname_750 = str_replace('/home/full_path_to/images/', '/home/full_path_to/test_750/', $filename);
$newname_500 = str_replace('/home/full_path_to/images/', '/home/full_path_to/test_500/', $filename);
$newname_250 = str_replace('/home/full_path_to/images/', '/home/full_path_to/test_250/', $filename);
$newname_100 = str_replace('/home/full_path_to/images/', '/home/full_path_to/test_100/', $filename);

// Composite the photo onto the background and resize
$cmd = '/home/full_path_to/background.jpg $temp -gravity center -geometry +0+125 -composite -unsharp 1.5x1+0.7+0.02 +repage -write mpr:image +delete '.
' ( mpr:image -thumbnail x1000 -write $newname_1000 ) '.
' ( mpr:image -thumbnail x750 -write $newname_750 ) '.
' ( mpr:image -thumbnail x500 -write $newname_500 ) '.
' ( mpr:image -thumbnail x250 -write $newname_250 ) '.
' ( mpr:image -thumbnail x100 -write $newname_100 )';

exec('convert $cmd'); 

// Delete the tempory image	
unlink('$temp');
 }
 ?> 

It’s common for your shell to be configured in such a way that you only need to use the “php” command. But, since this depends on how your system is set up, I told you to use an absolute path to the php executable, because it’s more likely to just work. But, it needs to be the correct absolute path. Yours might not be located there. You can continue using just php, but make sure this is resolving to the cli, not the cgi version of php. Verify this by doing php -v and you should probably see cli or cgi in the output somewhere.

Don’t use -r flag. It will be hell trying to get the quotes and escaped strings correct. Just name the file that holds the php code, like I showed. php will read the file contents, and execute it. Optionally, you can be explicit about the file to be executed by using the -f flag.

php -f /full/path/to/hello_world.php

You should not need to put #!/usr/bin/php at the top of the php script if using the cli php executable.

I forgot that I had CHMOD the script to 777 as well.

Anyway I have tried with something simple and it worked OK:


<?php
$cmd = "-size 300x130 -background NavajoWhite ".
"-fill white -gravity center -font Luxi-Serif-Regular caption:\\"WORKING\\" ";
exec("/usr/local/bin/convert $cmd /home/full_path_to/putty.jpg");
?>

I have looked at the code for the program in the previous post again and the -r must have just been a warning and I should have read all the errors :rolleyes:

I have taken out the -r and the #!/usr/bin/php and run the program again; it looks like the shell script is not running ( I am not sure how to display the errors at the moment ).


test_generate.php
sh: -r: command not found
PHP Warning:  unlink($temp): No such file or directory in /home/full_path_to/test_generate.php on line 50

There are 4 files with the name starting with Q and I get 4 lots of errors so that the glob part must be working.

I suppose I should do everything with the shell script or php ?
The problem is I know nothing about shell scripts and the shell script code is to complicated for me to convert into php.

Since you’re individual processes are in chunks taking seconds you can use a Javascript loopback approach. I’ve had to do this multiple times for import scripts - this is how it works in abstract.

Your page will take a $_GET argument that tells it what index in the images to start with. So if it takes 1.6 seconds to do 1 image then 10 images should be a good batch size at 16 seconds / batch. If the $_GET var indicating the stage isn’t set you take a count of all images possible, do the first 10 images, then send a page back to the browser declaring that block done with a small block of Javascript at the bottom to load the next step

<script type="text/javascript">
setTimeout('window.location = "myscript.php?step=<?php echo $nextStep ?>"');
</script>

You can also you a meta header redirect but I’d advise against this for the following reason - you can put the script at the end of the page so that if something hangs before PHP evals the page the process won’t jump ahead until you can address whatever hung up.

You do have to leave the calling computer on for the entire time the script runs for this to work.

Another advantage of this approach is if it does lock up at some point you’ll know approximately where and won’t need to start from scratch.


exec('convert $cmd'); 

// Delete the tempory image     
unlink('$temp');

You used single quotes, so php didn’t expand the variables.

So the shell got the litteral string convert $cmd. But, in a shell(for example, bash shell is very common), it supports variables too :slight_smile: And the syntax to read the value of a variable happens to be just like php, dollar sign and all :slight_smile: There’s probably some predefined variable that exists in the shell named cmd, and the shell expands the value. I bet that’s where you weird error comes from.

I still have 2 errors and I think the first one is in the shell code and the second one is in my code:

sh: -c: line 0: syntax error near unexpected token (' sh: -c: line 0: convert

I think we are getting close now and I will have to contact the guy who wrote the shell script but will not hear from him until tonight.
His default variables are written like :


# set default values
mode="vertical"		# vertical or horizontal
wrap=50			# percentage of cylinder to wrap with 

As I mentioned in an earlier post I changed the ’ to " when I had problems with the code. At the time I thought it may cause problems but did it anyway :eek:

This is all interesting and its a shame I can not run it on my servers but they are all shared and I do not have access. I can not run it on my XAMPP localhost as well and will look into that.

I had not thought of using JavaScript Michael and will give your code a try - probably not today as its a busy day. I pressume I could run the javascript from two different pages at the same time which would speed up the process ?

I have looked at the shell code and it has lines like:

sed >&2 -n '/^######/q;  /^#/!q;  s/^#*//;  s/^ //;  4,$p' "$PROGDIR/$PROGNAME"

As you can see some parts use ’ and others use "