SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    SitePoint Guru brent5392's Avatar
    Join Date
    Dec 2005
    Location
    Australia
    Posts
    636
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Folder Structure

    Just a quick question.

    Would 3 folders, with 100 folders in each be quicker than 1 folder with 300 in it? This is assuming the direct path is known. e.g. "folder3/folder49/" or "folder1/folder239/".

    Secondly

    Would 3 folders, with 10 subfolders, each with 10 subfolders be quicker than 3 folders with 100 folders? Yet again, assuming the direct path is known.

    These are all situations where PHP is grabbing a file to use.

    Thanks,
    Brent.
    PHP | MySQL | (X)HTML | CSS

  2. #2
    SitePoint Zealot Dachande663's Avatar
    Join Date
    Feb 2005
    Location
    Birmingham, UK
    Posts
    151
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I don't think it makes an overly important bottleneck compared to code execution time etc, but as Flickr found out you may soon hit the limit for the number of directories you can have in one directory (I know this affects ext3, not sure about other FS).
    Web Developer & Geek: hybridlogic.co.uk ~ lukelanchester.com

  3. #3
    SitePoint Guru brent5392's Avatar
    Join Date
    Dec 2005
    Location
    Australia
    Posts
    636
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Well the total of 300 folders is just a quick number i threw in. Im talking about large scale. Tens of thousands, hundreds of thousands, even millions.
    PHP | MySQL | (X)HTML | CSS

  4. #4
    SitePoint Wizard TheRedDevil's Avatar
    Join Date
    Sep 2004
    Location
    Norway
    Posts
    1,198
    Mentioned
    4 Post(s)
    Tagged
    1 Thread(s)
    Have you ever tried to have a lot of files or folders inside one directory and then try to load the directory? It is not really a problem as long as you include each file directly, but the second you try to view the directory you will need to spend quite some time waiting.

    You should structure your system in such a way that you can keep a "lower" number of directories/files in view. Just create a algorithm which will decide the folder placements for the directory/files.

  5. #5
    SitePoint Guru brent5392's Avatar
    Join Date
    Dec 2005
    Location
    Australia
    Posts
    636
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Im storing user files, and as the userbase grows this may become a problem. Im definitely going to need to divide it up into many subfolders. I don't plan on waiting for ages for the list of folders when i need to get files via ftp.

    Maybe I could sort them by their id in the database? Zerofill their id, and make each invidual number a folder. So user "0019456" would have their files at "0/0/1/9/4/5/6/".

    Anyway, thanks for the help. I think ive got it now ^_^
    PHP | MySQL | (X)HTML | CSS

  6. #6
    SitePoint Wizard wheeler's Avatar
    Join Date
    Mar 2006
    Location
    Gold Coast, Australia
    Posts
    1,369
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I've discovered the wrong way that at around 40-50,000 files and folders in one directory and you run into all sorts of problems. For one, there is an operating system limit somewhere around here (in fact I think its more like 32,000 so I really pushed it ). Secondly, trying to manage the files is basically impossible if you even want to browse the files.

    My solution is to have folders 1-4999, 5000-9999 etc so 1 folder contains up to 5,000 sub folders, each sub folder corresponding to an ID.

    Maths is not my strong suite, but I figure than means I can have about 250,000,000 folders before I stress out the file system and you'd think that the number of files and folders within will be the least of your problems when you have this much activity!
    Studiotime - Time Management for Web Developers
    to-do's, messages, invoicing, reporting - 30 day free trial!
    Thomas Multimedia Web Development

  7. #7
    SitePoint Wizard TheRedDevil's Avatar
    Join Date
    Sep 2004
    Location
    Norway
    Posts
    1,198
    Mentioned
    4 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by brent5392 View Post
    Maybe I could sort them by their id in the database? Zerofill their id, and make each invidual number a folder. So user "0019456" would have their files at "0/0/1/9/4/5/6/".
    Yes, it is simplest to sort it by the id. Though instead of dividing it up like that you can also do like this:

    PHP Code:
    $id 10003;

    $child_dir floor(($id 100));

    $master_dir floor(($child_dir 100));

    $folder $master_dir.'/'.$child_dir.'/'.$id
    Which would create the directory path: 1/100/10003

    The first directory would contain 100 directories, and each inside it would contain 100 as well, totaling 10000 directories.
    Of course you would modify this system accordingly to how many directories you expect to be created.

  8. #8
    SitePoint Guru brent5392's Avatar
    Join Date
    Dec 2005
    Location
    Australia
    Posts
    636
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I think my way might be a little better for my case. With only 10 directories in each folder, it makes it much faster when i need to find a file via ftp. Also, my way will make it easier to expand. Should I reach user 9999999 I can always just create a new folder called 0 and move all the others into there, and then with a few updates, everything will be going perfectly again without any major change.
    PHP | MySQL | (X)HTML | CSS


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •