Does anyone know why Apache would store large file uploads (> 100 MB) entirely in memory instead of writing them to disk?
I’m under the impression that Apache should be able to upload files larger than the memory available to the server.
Even stranger, Apache seems to use more than twice the file size in memory, on my server. Here’s the output from “top” immediately before PID 18367 was killed (I was trying to upload a 130 MB file — it got to ~80% before the server ran out of memory):
[Tue Feb 01 09:43:07 2011] [error] (12)Cannot allocate memory: fork: Unable to fork new process
[Tue Feb 01 09:43:17 2011] [notice] child pid 18367 exit signal Segmentation fault (11)
To clarify: it’s an HTTP upload, submitted as “multipart/form-data”.
The problem is that the upload never reaches PHP — the Apache process dies before it can pass the file on. The only limit that’s hit is the actual memory available to the server.
What you’re seeing the 1st code block from my 1st post is that the httpd process (Apache) has used 393 MB of physical RAM, completely exhausting my VPS, and (seconds later), resulting in the process being killed.
This one exchange I found on the PHP mailing list directly addresses the situation:
>> I have never seen a case where Apache has stored the file on the file
>> system. In my past experience it has always held the file in memory,
>> therefor limited to the max physical memory that was install, minus a
>> little for other things.
>
> Well, I can easily show you such a case - I can upload a 1Gb file
> without apache memory usage changing one bit. Even if Apache did
> upload into memory, why would that make the file limited to the max
> amount of physical memory??
>
> How about if I upload a 1Gb file to a webserver on a machine that only
> has 256Mb memory - would you say that’s impossible?
Just did that (1Gb file uploaded to machine with 256M RAM) - Apache
definitely does not store the uploaded file in memory, I also straced
the upload, and it’s 1) read from socket followed by 2) write to disk.
I’m afraid I’m not much help because I can’t see it as an Apache problem.
From my experience with upload problems, there are several problem areas (none of which seem to apply):
PHP max upload (default to 8Mb)
; Maximum allowed size for uploaded files.
upload_max_filesize = 8M
PHP timeout (default to 30 sec)
; Default timeout for socket based streams (seconds)
default_socket_timeout = 60
upload_tmp_dir="C:\\Windows\\Temp"
Establish and set permissions (chmod 777) on TEMPORARY upload directory
upload_tmp_dir="C:\\Windows\\Temp"
Of course, after that, you need to check the file type, that it was UPLOADED, then move to the directory in your website where you need it saved - but you didn’t get that far.
All those settings (code blocks) are within php.ini on my WinDoze test server.
The is another possible explanation, though, and you’ll need to contact your host to confirm (or deny) it:
Your host’s limits on your VPS are precluding the upload script from using as much of the SHARED server resources as it needs to deal with the large upload. Yes, this could be it as the memory involved can be substantial IF there is not a temp directory involved and the file is being uploaded to memory (although I can’t understand why any host would not have a temp upload directory specified).
Remember, please, before your curse Apache, it’s a FILE server, not an FTP server. The file upload you’re performing is a function of the PHP file upload function which uses Apache to interface with you.
Remember, please, before your curse Apache, it’s a FILE server, not an FTP server. The file upload you’re performing is a function of the PHP file upload function which uses Apache to interface with you.
Hey, no cursing here — I love Apache :).
All the same, I’ve very confident that it is an Apache issue. I think it works something like this:
What Apache starts, it reads the PHP memory limits in to the running Apache process. When you try and upload a file, it is a straight HTTP upload. PHP plays no part in the actual upload, except for the upload limits set in place by the php settings found in the php.ini or other Apache config files. Now, Apache actually handles the Upload. Once the file has been received completely, Apache then passes the file and process running to PHP. http://www.mail-archive.com/php-general@lists.php.net/msg231760.html
To clarify, I can upload files up to about ~100 MB, and my PHP script does its job just swimmingly. And I’m sure that if it were to receive the hand-off from Apache, that it’d be able to handle files up to 500 MB, which is where I have my PHP INI limits set.
I had the same idea about the /tmp directory, and I tried changing permissions, ownership, etc. — in addition to modifying “upload_tmp_dir” (which I don’t think is relevant) — all to no avail.
Anyway, I think I’ve exhausted my good-will quota from you, and I’m going to go troll some linux forums, I think!
thanks again.
Also, I stand corrected. I use PHP scripting so that was the area I was familiar with and went to that naturally. HTML uploads are performed by Apache without PHP intervention (until verifying the file it wants to move was, indeed, uploaded).