SitePoint Sponsor

User Tag List

Page 1 of 2 12 LastLast
Results 1 to 25 of 34

Hybrid View

  1. #1
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)

    Should all Images be One Type?

    I am allowing Users to upload their Photos.

    GIF, JPEG, and PNG are valid formats, however - for security reasons - I am running everything through PHP's GD.

    Would it make sense to convert everything to ONE FORMAT?

    Is there any Security Benefit in doing so?

    Is there any Web Design benefit in doing so?

    Is there any File Management benefit in doing so?

    Thanks,


    Debbie

  2. #2
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Converting from one format to another will not add anything to your security. It will, however degrade the quality of images in some cases.
    What you may want to do is strip all exif data from jpeg and png images. It will make the size of image smaller.

  3. #3
    . shoooo... silver trophy logic_earth's Avatar
    Join Date
    Oct 2005
    Location
    CA
    Posts
    9,013
    Mentioned
    8 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by DoubleDee View Post
    Is there any Security Benefit in doing so?
    Nope.
    Is there any Web Design benefit in doing so?
    Nope.
    Is there any File Management benefit in doing so?
    Nope.
    Logic without the fatal effects.
    All code snippets are licensed under WTFPL.


  4. #4
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by logic_earth View Post
    Nope.

    Nope.

    Nope.
    Anything else you would like to add?


    Debbie

  5. #5
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    How do you do file management for your uploaded images?
    There are different ways to do this (some are wrong way, some are right way)

  6. #6
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ultra1 View Post
    How do you do file management for your uploaded images?
    There are different ways to do this (some are wrong way, some are right way)
    I don't understand the question.

    When Users upload Photos, I will be storing them in a directory on my Webserver and the corresponding Filename will be stored in each User's record in the database.


    Debbie

  7. #7
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    So you will store all of them in the same directory? What if there are 20,000 images? Will they all be stored in the same directory?

  8. #8
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ultra1 View Post
    So you will store all of them in the same directory? What if there are 20,000 images? Will they all be stored in the same directory?
    At this point, yes.

    When I have 20,000 Users on my website, I can pay you to figure out a better way of storing things!

    Why, would would you do?


    Debbie

  9. #9
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You got a deal! What I would do is definitely create sub-directories. You can create subdirectory based on current day, current month, whatever you want.

  10. #10
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ultra1 View Post
    You got a deal! What I would do is definitely create sub-directories. You can create subdirectory based on current day, current month, whatever you want.
    Does anyone else agree with what ultra1 is saying?

    And at what point would this be an issue anyway?

    What would happen if I did have 2,000 images or 20,000 images or 200,000 images in one directory on my webserver?


    Debbie

  11. #11
    . shoooo... silver trophy logic_earth's Avatar
    Join Date
    Oct 2005
    Location
    CA
    Posts
    9,013
    Mentioned
    8 Post(s)
    Tagged
    0 Thread(s)
    Is their a limit to the number of files a directory can hold? Depends on the file system. For example on Windows with NTFS...Maximum number of files in a single folder: 4,294,967,295. So does it matter? Probably not, you'll need several thousands of users.
    Logic without the fatal effects.
    All code snippets are licensed under WTFPL.


  12. #12
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by logic_earth View Post
    Is their a limit to the number of files a directory can hold? Depends on the file system. For example on Windows with NTFS...Maximum number of files in a single folder: 4,294,967,295. So does it matter? Probably not, you'll need several thousands of users.
    I think it is good to "design for the future" but I agree with Logic_Earth that this probably won't be an issue anytime soon.

    First off, I am on Linux, so that is pretty hardy and scalable.

    If you have an idea that you truly feel is better and that I can EASILY implement - since I have already spent 2+ weeks on my File Upload module - then I'll listen...

    Thanks,


    Debbie

  13. #13
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    The limit of number of files is quite large but it does not mean you should push that limit.
    Think about performance. What if you have to delete one image from a directory? If there are 100,000 images in a directory then deleting will be slow.
    also what if you need to find a file in a directory by the file name? What if you want to peek inside you directory from a command line using ls command?
    Have you ever tried using ls command on a directory with 100,000 images while connected remotely to your server?
    Have you tried to delete just a few files from directory that has 100,000 files?
    It's often fails completely but first you wait for a couple of minutes in front of monitor.

  14. #14
    . shoooo... silver trophy logic_earth's Avatar
    Join Date
    Oct 2005
    Location
    CA
    Posts
    9,013
    Mentioned
    8 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ultra1 View Post
    Think about performance. What if you have to delete one image from a directory? If there are 100,000 images in a directory then deleting will be slow.
    No it won't.
    also what if you need to find a file in a directory by the file name?
    That is fast to do.
    What if you want to peek inside you directory from a command line using ls command?
    Have you ever tried using ls command on a directory with 100,000 images while connected remotely to your server?
    Why would one need to? It all in the database.
    Have you tried to delete just a few files from directory that has 100,000 files?
    Quite fast as well. If it is slow for you, you must be using a crummy file system.

    A properly designed file system will have no performance issues no matter how many files are in a directory. In reality a directory is a logical abstract. Directories are for people, computers don't care this is why in NTFS the Maximum number files that can be contained in a directory is also the maximum number of files that can be on the drive. It doesn't care about directories.

    All in all, it doesn't actually matter. You first need several thousand of users each uploading something. Thus going over board designing something that is not even an issue is a waste of time.

    But let me share something, too many directories can have a far more negative impact then files.
    Logic without the fatal effects.
    All code snippets are licensed under WTFPL.


  15. #15
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You can read this article, it talks about just issue and possible solutions. http://michaelandrews.typepad.com/th...structure.html

    also a discussion here http://serverfault.com/questions/129...acceptable-per

  16. #16
    SitePoint Enthusiast
    Join Date
    Dec 2011
    Posts
    44
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    crummy file system? There are not too many choices. If you are on Linux you almost always using ext3. It's not crummy at all but it does have limitations especially the performance slow-down when there are too many files in a single directory. I've played around with reiserfs in the past but it's not really maintained anymore.

  17. #17
    . shoooo... silver trophy logic_earth's Avatar
    Join Date
    Oct 2005
    Location
    CA
    Posts
    9,013
    Mentioned
    8 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ultra1 View Post
    If you are on Linux you almost always using ext3. It's not crummy...
    It is a crummpy file system. For perfomances applications ext3 is a bad choice.
    There are dozens upon dozens of choices on linux.
    Logic without the fatal effects.
    All code snippets are licensed under WTFPL.


  18. #18
    Foozle Reducer ServerStorm's Avatar
    Join Date
    Feb 2005
    Location
    Burlington, Canada
    Posts
    2,699
    Mentioned
    89 Post(s)
    Tagged
    6 Thread(s)
    Hi,

    Not to morph the thread too far, but in Linux we have used XFS and Btrfs with great results. If I was worried about performance and storage capacity on Linux, I would consider XFS as it has a mature set of tools, has been integrated and kept up to date in the latest kernel and has few disadvantages that you would look at planning for a growing number of files. Btrfs has great promise but Oracle buying Sun has somewhat clouded the view of where this is going to go, but Oracle says they plan to keep going with it, so it is something to watch for. Our tests of it had amazing results, but it was difficult to configure.

    For your reference:
    https://en.wikipedia.org/wiki/Comparison_of_file_systems
    https://en.wikipedia.org/wiki/Btrfs
    https://en.wikipedia.org/wiki/XFS

    ext4 is fairly common in linux but can be really difficult if some of your data corrupts, and in general if you are able to choose then go with something like XFS.

    I am sure that people will have many views on this, because many of the 'journalling type of File Systems in Linux can be tunned up and made-workable. I don't intend to start a flame war about what File Systems would best help Debbie plan for the future, I am simply relaying that our experience with these two files systems has been very good with our databases, and file storage, retrieval, writes, and large scale deletes. As well low data corruption.

    Regards,
    Steve
    ictus==""

  19. #19
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    ServerStorm,

    So you are saying that putting all of my Users' Thumbnails in one directory is fine for now?

    Like I said earlier, I *obviously* plan for the future, but even if I didn't, when I get to 20,000+ Users then I'll be in a much better place. (Just don't hold your breath that it happens!)


    Debbie

  20. #20
    Foozle Reducer ServerStorm's Avatar
    Join Date
    Feb 2005
    Location
    Burlington, Canada
    Posts
    2,699
    Mentioned
    89 Post(s)
    Tagged
    6 Thread(s)
    Hi Debbie,

    Yes putting all your User's thumbnails in one directory should be fine. You are most likely using ext4 while not being the best for large or high performance will not impede your abilities.

    Let's face it, you start off with a few hundred; maybe in a year a few thousand. Your current Linux Files System will be able to handle this without a problem. If you have to scale because it becomes wildly popular then you will not be on a basic VFS; you will most likely be on a Cloud or have your own server(s) configured to handle the growth. The easiest part will be migrating those Thumbnails from you existing setup to whatever will be your next stage.

    When it comes to I.T. infrastructure, long term planning is not typically as much about picking the ultimate setup for now that will grow in the future. It is more about how you can grow with your needs. You start off with a more basic setup that still does the job, in time you might need to move to a better File System on a new more robust VSM, the more popular your site becomes you may further move into clustering distributed database handling and file caching servers. To plan for that now does not seem to me to be the best use of time, I might investigate what my second stage might be. What costs (based on current market values that may change wildly in the future) and complications might I be facing and proactive know when I am going to 'pull the trigger' to move to the next stage... once there I would begin planning the next stage...

    Hope this helps and also hope that your application is a 'Wild Success'!

    Regards,
    Steve
    ictus==""

  21. #21
    SitePoint Wizard DoubleDee's Avatar
    Join Date
    Aug 2010
    Location
    Arizona
    Posts
    3,761
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ServerStorm View Post
    Hi Debbie,

    Hope this helps and also hope that your application is a 'Wild Success'!

    Regards,
    Steve
    Yes, that helps and seems in line with my own thinking.

    I appreciate the other competing views, but I have a long ways to go before the # of Files in a Directory or anything else becomes an issue?!

    Thanks for the nice words and well-wishes. I'll need them!!


    Debbie

    P.S. I am reviewing my upload.php script now and hope to post in online later for a "code review". If you wanna really help me out, please keep an eye out for it and let me know if I internalized everyone's suggestions.

  22. #22
    Foozle Reducer ServerStorm's Avatar
    Join Date
    Feb 2005
    Location
    Burlington, Canada
    Posts
    2,699
    Mentioned
    89 Post(s)
    Tagged
    6 Thread(s)
    Quote Originally Posted by DoubleDee View Post
    Yes, that helps and seems in line with my own thinking.

    I appreciate the other competing views, but I have a long ways to go before the # of Files in a Directory or anything else becomes an issue?!

    Thanks for the nice words and well-wishes. I'll need them!!


    Debbie

    P.S. I am reviewing my upload.php script now and hope to post in online later for a "code review". If you wanna really help me out, please keep an eye out for it and let me know if I internalized everyone's suggestions.
    Hi Debbie,

    I will look out for you upload script and if I have anything valuable to add I will.

    ext4 has support for volumes with sizes up to 1 exabyte ( 1 exabyte = 1,048,576 terabytes ) and files with sizes up to 16 terabytes. It supports up to 64, 000 directories. Again lot's of room to grow into and you can either store your thumbnails in one directory or divide them into some logical directory structure; say by month. With the ext4 limits it gives you lots of room before you will have to move to something else.

    Running RAID 10 on ext4 and the kernel patches to move large scale deletions/updates to memory have all but wiped out the problems ext3 suffered from. The inadequacies of ext3 is why ext4 was embarked upon.

    Regards,
    Steve
    ictus==""

  23. #23
    SitePoint Wizard
    Join Date
    Dec 2003
    Location
    USA
    Posts
    2,582
    Mentioned
    29 Post(s)
    Tagged
    0 Thread(s)
    I support ultra's idea, not necessarily for performance, but to help avoid naming collisions. It's a lot easier to avoid collisions with 10 files then it is 100,000 files.

    Also for the original topic: no, there is no benefit. In fact, there are several reasons to use multiple types (GIF, JPG, and PNG all have different storage methods, and those methods make them better suited for certain types of images).

  24. #24
    . shoooo... silver trophy logic_earth's Avatar
    Join Date
    Oct 2005
    Location
    CA
    Posts
    9,013
    Mentioned
    8 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by samanime View Post
    I support ultra's idea, not necessarily for performance, but to help avoid naming collisions. It's a lot easier to avoid collisions with 10 files then it is 100,000 files.
    Easier way to do that then with directories. sha1_file for example. Name all files using the result from that will make sure A) you have no name conflicts B) no duplicate files. One would spend more time making sub directory after sub durectory when there are much easier and faster solutions.
    Logic without the fatal effects.
    All code snippets are licensed under WTFPL.


  25. #25
    SitePoint Wizard
    Join Date
    Dec 2003
    Location
    USA
    Posts
    2,582
    Mentioned
    29 Post(s)
    Tagged
    0 Thread(s)
    sha1_file just does a sha1 hash. Those can have naming conflicts (or there are only 40^16 things that can be hashed in the world =p)

    Making the subfolders is pretty easy and automated:
    Code:
    $folder_name = $uploads . date('Y/m/d');
    mkdir($folder_name, 0755, true);
    
    // copy file


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •