SitePoint Sponsor

User Tag List

Results 1 to 12 of 12
  1. #1
    SitePoint Enthusiast
    Join Date
    Sep 2012
    Location
    Northern New Brunswick, Atlantic Canada
    Posts
    44
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    How big can a data file get?

    I have a PHP program that opens a text data file, reads it, adds to it, and saves it again.
    The question is: how big can that file get, or how many lines can it contain, before PHP runs out of space or memory?

    Before I start getting error codes I'd like to have some idea of what the limits are, or how to find out - and how to change them.

    Thank you;
    Ian

  2. #2
    Hosting Team Leader silver trophybronze trophy
    cpradio's Avatar
    Join Date
    Jun 2002
    Location
    Ohio
    Posts
    5,118
    Mentioned
    152 Post(s)
    Tagged
    0 Thread(s)
    First are you reading the entire file or just the last line?

    If you are only reading the last line in the file, then PHP memory limit shouldn't be much of a problem, however, I would strongly recommend doing a prepend (using the 'x' instead of 'w') to the file so you consistently read the first line instead of the last, and then you insert a new first line when writing (makes coding easier and less work).

    Your limits will be the hard drive storage space and the php max memory limit (if you read the entire file).

  3. #3
    SitePoint Enthusiast
    Join Date
    Sep 2012
    Location
    Northern New Brunswick, Atlantic Canada
    Posts
    44
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Unfortunately, the nature of the file means I have to read the entire file, change the first line, add additional lines to the end, then save it.
    This should be all server-side, I think.

    Will the capacity of the variable that the file is 'read into' be a problem? I'm still used to VB where the variable size had to be defined - but that does date back to a time when a big computer had 4mb of RAM!

  4. #4
    Hosting Team Leader silver trophybronze trophy
    cpradio's Avatar
    Join Date
    Jun 2002
    Location
    Ohio
    Posts
    5,118
    Mentioned
    152 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by imillard View Post
    Unfortunately, the nature of the file means I have to read the entire file, change the first line, add additional lines to the end, then save it.
    This should be all server-side, I think.

    Will the capacity of the variable that the file is 'read into' be a problem? I'm still used to VB where the variable size had to be defined - but that does date back to a time when a big computer had 4mb of RAM!
    Since you will be reading the file into a variable, your only limit will be the max memory limit in your php.ini file.

  5. #5
    SitePoint Enthusiast
    Join Date
    Sep 2012
    Location
    Northern New Brunswick, Atlantic Canada
    Posts
    44
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Thank you for that info. I should have some breathing room. Now I have to find the php.ini file.

    This isn't running on "my" server. My pages are hosted on a very large server system, somewhere in Arizona.
    I have looked through my options, and there appears to be no way I can change the individual settings for the PHP extensions - which makes sense because it's probably a shared server.

  6. #6
    Hosting Team Leader silver trophybronze trophy
    cpradio's Avatar
    Join Date
    Jun 2002
    Location
    Ohio
    Posts
    5,118
    Mentioned
    152 Post(s)
    Tagged
    0 Thread(s)
    If you run
    PHP Code:
    <?php echo phpinfo(); ?>
    You can look through the output and find the max memory limit. Remove that line/file immediately after finding what you need though, as it gives out a lot of information you won't want to make public.

  7. #7
    SitePoint Enthusiast
    Join Date
    Sep 2012
    Location
    Northern New Brunswick, Atlantic Canada
    Posts
    44
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Yes, I was just looking through that readout, and apparently I have 64m to work with - which should be sufficient.
    Thank you for your help.

  8. #8
    SitePoint Wizard silver trophybronze trophy Cups's Avatar
    Join Date
    Oct 2006
    Location
    France, deep rural.
    Posts
    6,869
    Mentioned
    17 Post(s)
    Tagged
    1 Thread(s)
    Maybe this idea is redundant now you don't think memory will be an issue.

    This advice about using sed might have been useful IF on a *nix machine AND exec() had not been disabled (check phpinfo() again). The search on using php exec and sed seems to suggest that could be a bit more efficient.

    I have to say I have not tested this, merely sharing an idea.

  9. #9
    SitePoint Guru bronze trophy
    Join Date
    Dec 2003
    Location
    Poland
    Posts
    930
    Mentioned
    7 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by imillard View Post
    Unfortunately, the nature of the file means I have to read the entire file, change the first line, add additional lines to the end, then save it.
    If your file consists of separate lines then think whether you will be able to get away with reading each line separately and dealing only with one line at a time (in a loop). This way you will not have to load the whole file into a variable, you will only need to use fgets() to read a single line so then you need enough memory to hold just one line. Of course, this will mean not adding the result to any string or array - just loading each new line to the same variable.

    Adding additional lines is not a problem at all if you open the file in a or a+ mode.

  10. #10
    SitePoint Enthusiast
    Join Date
    Sep 2012
    Location
    Northern New Brunswick, Atlantic Canada
    Posts
    44
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Well, like I said, I have to read - and then change the first line, then add one or more lines to the end of the file. If it was only a matter of simple adding to the file I imagine there must be some sort of Append function.
    If the file becomes larger than 64M I may have to rethink the entire process. Ugh!

  11. #11
    SitePoint Guru bronze trophy
    Join Date
    Dec 2003
    Location
    Poland
    Posts
    930
    Mentioned
    7 Post(s)
    Tagged
    0 Thread(s)
    Okay, if you have to change the first line and the line may have to be of different length then yes, simply appending will not do. But do you really have to read the whole file into a variable instead of reading it line by line and performing all your required logic in this way? If you can read it line by line then there is a different solution that should work: read the original file line by line and instead of creating a whole new output in a variable open another (temporary) file for writing and write the changed data to the new file line by line as you are reading from the original. If linear reading is not enough for you then you can use fseek() to start reading from any point in the file. As you are done then simply rename the temporary file to the original which will replace the original file with the new file. This will also require very little memory provided the lines are not very long.

  12. #12
    SitePoint Enthusiast
    Join Date
    Sep 2012
    Location
    Northern New Brunswick, Atlantic Canada
    Posts
    44
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Yes, that looks like the best solution. Use a simple loop to read each line, modify as necessary, and save to a temp file, then rename back.
    Thank you for the idea.
    Sometimes it's easy to get fixated on a single method of achieving the desired results. A second perspective is always helpful.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •