Scandir() and Array limits

I’ve just finished writing a PHP page that creates a time lapse video file from a collection of JPEGs stored in a folder. Everything is working beautifully right now with a few images, but I’m worried about what might happen down the road when I begin to get too many images.

I have a video camera set up to take a snapshot every 10 minutes, so I’m getting about 144 images per day. This time-lapse project could potentially last 5 years, or 262,800 images. After doing a little research is seems that the # of files in the folder, won’t be a problem and neither will the storage space on the drive. However I am wondering if I am going to crash my server scanning this many files and run out of RAM.

Here are the steps I’m taking to create the video.

#1 - Use scandir() to load every file in the folder into an array
#2 - Loop through the array and add only JPG files into a second array
#3 - Calculate the # of images (frames) I need to create a video x seconds long
#4 - Loop through the second array selecting spaced out frames to create a text file with the final candidates
#5 - Encode the video into an AVI file using the text file as the file list.

What I’m worried about is whether scandir() can handle 250,000+ files and if creating two arrays with 250,000+ items will be a problem. The filenames are normal length : axis233_10_10_11_15_24_25.jpg etc. My server is only handling this task and nothing else, and only one user at a time would be using this web app.

Is there a way I can calculate this out and see if it will cause a problem? I’d rather find out now and work around it than 2 years into the project.

Thanks!!

Handling 262,800 files within a single directory might get you a headache.
Accessing a directory with that many files in it will become very slow.
I’d suggest grouping those files in multiple directories, e.g. /Y/m/d/Hi.jpg so you’ll turn up with images with a path like /2011/07/02/1210.jpg
That means you’ll have a maximum of 144 images per directory.

Processing those arrays in combination with the rest of your script may become a problem memory wise, but that’s easily solved by setting PHP’s memory_limit to e.g. 128M instead of 32M since each array of 262,800 images will use ~7Mb of memory.

Also, instead of using scandir(), you could take a look at the RecursiveDirectoryIterator.