Should all Images be One Type?

[DoubleDee]

I am allowing Users to upload their Photos.

GIF, JPEG, and PNG are valid formats, however - for security reasons - I am running everything through PHP's GD.

Would it make sense to convert everything to ONE FORMAT?

Is there any Security Benefit in doing so?

Is there any Web Design benefit in doing so?

Is there any File Management benefit in doing so?

Thanks,


Debbie

[ultra1]

Converting from one format to another will not add anything to your security. It will, however degrade the quality of images in some cases.
What you may want to do is strip all exif data from jpeg and png images. It will make the size of image smaller.

[logic_earth]

Is there any Security Benefit in doing so?

Nope.

Is there any Web Design benefit in doing so?

Nope.

Is there any File Management benefit in doing so?

Nope.

[DoubleDee]

Nope.

Nope.

Nope.

Anything else you would like to add?


Debbie

[ultra1]

How do you do file management for your uploaded images?
There are different ways to do this (some are wrong way, some are right way)

[DoubleDee]

How do you do file management for your uploaded images?
There are different ways to do this (some are wrong way, some are right way)

I don't understand the question.

When Users upload Photos, I will be storing them in a directory on my Webserver and the corresponding Filename will be stored in each User's record in the database.


Debbie

[ultra1]

So you will store all of them in the same directory? What if there are 20,000 images? Will they all be stored in the same directory?

[DoubleDee]

So you will store all of them in the same directory? What if there are 20,000 images? Will they all be stored in the same directory?

At this point, yes.

When I have 20,000 Users on my website, I can pay you to figure out a better way of storing things!

Why, would would you do?


Debbie

[ultra1]

You got a deal! What I would do is definitely create sub-directories. You can create subdirectory based on current day, current month, whatever you want.

[DoubleDee]

You got a deal! What I would do is definitely create sub-directories. You can create subdirectory based on current day, current month, whatever you want.

Does anyone else agree with what ultra1 is saying?

And at what point would this be an issue anyway?

What would happen if I did have 2,000 images or 20,000 images or 200,000 images in one directory on my webserver?


Debbie

[logic_earth]

Is their a limit to the number of files a directory can hold? Depends on the file system. For example on Windows with NTFS...Maximum number of files in a single folder: 4,294,967,295. So does it matter? Probably not, you'll need several thousands of users.

[DoubleDee]

Is their a limit to the number of files a directory can hold? Depends on the file system. For example on Windows with NTFS...Maximum number of files in a single folder: 4,294,967,295. So does it matter? Probably not, you'll need several thousands of users.

I think it is good to "design for the future" but I agree with Logic_Earth that this probably won't be an issue anytime soon.

First off, I am on Linux, so that is pretty hardy and scalable.

If you have an idea that you truly feel is better and that I can EASILY implement - since I have already spent 2+ weeks on my File Upload module - then I'll listen...

Thanks,


Debbie

[ultra1]

The limit of number of files is quite large but it does not mean you should push that limit.
Think about performance. What if you have to delete one image from a directory? If there are 100,000 images in a directory then deleting will be slow.
also what if you need to find a file in a directory by the file name? What if you want to peek inside you directory from a command line using ls command?
Have you ever tried using ls command on a directory with 100,000 images while connected remotely to your server?
Have you tried to delete just a few files from directory that has 100,000 files?
It's often fails completely but first you wait for a couple of minutes in front of monitor.

[ultra1]

You can read this article, it talks about just issue and possible solutions. http://michaelandrews.typepad.com/th...structure.html

also a discussion here http://serverfault.com/questions/129...acceptable-per

[logic_earth]

Think about performance. What if you have to delete one image from a directory? If there are 100,000 images in a directory then deleting will be slow.

No it won't.

also what if you need to find a file in a directory by the file name?

That is fast to do.

What if you want to peek inside you directory from a command line using ls command?
Have you ever tried using ls command on a directory with 100,000 images while connected remotely to your server?

Why would one need to? It all in the database.

Have you tried to delete just a few files from directory that has 100,000 files?

Quite fast as well. If it is slow for you, you must be using a crummy file system.

A properly designed file system will have no performance issues no matter how many files are in a directory. In reality a directory is a logical abstract. Directories are for people, computers don't care this is why in NTFS the Maximum number files that can be contained in a directory is also the maximum number of files that can be on the drive. It doesn't care about directories.

All in all, it doesn't actually matter. You first need several thousand of users each uploading something. Thus going over board designing something that is not even an issue is a waste of time.

But let me share something, too many directories can have a far more negative impact then files.

[ultra1]

crummy file system? There are not too many choices. If you are on Linux you almost always using ext3. It's not crummy at all but it does have limitations especially the performance slow-down when there are too many files in a single directory. I've played around with reiserfs in the past but it's not really maintained anymore.

[logic_earth]

If you are on Linux you almost always using ext3. It's not crummy...

It is a crummpy file system. For perfomances applications ext3 is a bad choice.
There are dozens upon dozens of choices on linux.

[ServerStorm]

Hi,

Not to morph the thread too far, but in Linux we have used XFS and Btrfs with great results. If I was worried about performance and storage capacity on Linux, I would consider XFS as it has a mature set of tools, has been integrated and kept up to date in the latest kernel and has few disadvantages that you would look at planning for a growing number of files. Btrfs has great promise but Oracle buying Sun has somewhat clouded the view of where this is going to go, but Oracle says they plan to keep going with it, so it is something to watch for. Our tests of it had amazing results, but it was difficult to configure.

For your reference:
https://en.wikipedia.org/wiki/Comparison_of_file_systems
https://en.wikipedia.org/wiki/Btrfs
https://en.wikipedia.org/wiki/XFS

ext4 is fairly common in linux but can be really difficult if some of your data corrupts, and in general if you are able to choose then go with something like XFS.

I am sure that people will have many views on this, because many of the 'journalling type of File Systems in Linux can be tunned up and made-workable. I don't intend to start a flame war about what File Systems would best help Debbie plan for the future, I am simply relaying that our experience with these two files systems has been very good with our databases, and file storage, retrieval, writes, and large scale deletes. As well low data corruption.

Regards,
Steve

[DoubleDee]

ServerStorm,

So you are saying that putting all of my Users' Thumbnails in one directory is fine for now?

Like I said earlier, I *obviously* plan for the future, but even if I didn't, when I get to 20,000+ Users then I'll be in a much better place. (Just don't hold your breath that it happens!)


Debbie

[ServerStorm]

Hi Debbie,

Yes putting all your User's thumbnails in one directory should be fine. You are most likely using ext4 while not being the best for large or high performance will not impede your abilities.

Let's face it, you start off with a few hundred; maybe in a year a few thousand. Your current Linux Files System will be able to handle this without a problem. If you have to scale because it becomes wildly popular then you will not be on a basic VFS; you will most likely be on a Cloud or have your own server(s) configured to handle the growth. The easiest part will be migrating those Thumbnails from you existing setup to whatever will be your next stage.

When it comes to I.T. infrastructure, long term planning is not typically as much about picking the ultimate setup for now that will grow in the future. It is more about how you can grow with your needs. You start off with a more basic setup that still does the job, in time you might need to move to a better File System on a new more robust VSM, the more popular your site becomes you may further move into clustering distributed database handling and file caching servers. To plan for that now does not seem to me to be the best use of time, I might investigate what my second stage might be. What costs (based on current market values that may change wildly in the future) and complications might I be facing and proactive know when I am going to 'pull the trigger' to move to the next stage... once there I would begin planning the next stage...

Hope this helps and also hope that your application is a 'Wild Success'!

Regards,
Steve

[DoubleDee]

Hi Debbie,

Hope this helps and also hope that your application is a 'Wild Success'!

Regards,
Steve

Yes, that helps and seems in line with my own thinking.

I appreciate the other competing views, but I have a long ways to go before the # of Files in a Directory or anything else becomes an issue?!

Thanks for the nice words and well-wishes. I'll need them!!


Debbie

P.S. I am reviewing my upload.php script now and hope to post in online later for a "code review". If you wanna really help me out, please keep an eye out for it and let me know if I internalized everyone's suggestions.

[samanime]

I support ultra's idea, not necessarily for performance, but to help avoid naming collisions. It's a lot easier to avoid collisions with 10 files then it is 100,000 files.

Also for the original topic: no, there is no benefit. In fact, there are several reasons to use multiple types (GIF, JPG, and PNG all have different storage methods, and those methods make them better suited for certain types of images).

[logic_earth]

I support ultra's idea, not necessarily for performance, but to help avoid naming collisions. It's a lot easier to avoid collisions with 10 files then it is 100,000 files.

Easier way to do that then with directories. sha1_file for example. Name all files using the result from that will make sure A) you have no name conflicts B) no duplicate files. One would spend more time making sub directory after sub durectory when there are much easier and faster solutions.

[samanime]

sha1_file just does a sha1 hash. Those can have naming conflicts (or there are only 40^16 things that can be hashed in the world =p)

Making the subfolders is pretty easy and automated:

$folder_name = $uploads . date('Y/m/d');
mkdir($folder_name, 0755, true);

// copy file

[logic_earth]

sha1_file just does a sha1 hash.

Of the file contents.

Those can have naming conflicts

Uh huh...if you get a conflict then it is likely the SAME FILE.