best way to uplaod a 165mb file

[wazo]

Hi guys

I need to upload a big .csv file to my server

its 165mb

whats the quickest way to upload this file?

[risoknop]

1) Archive it to *.7z.

2) Maybe split into few chunks in case you have crappy ftp and are afraid it will die in the middle of upload.

3) Unzip it on server.

[buldozerce]

FTP or wgeting from your home server to your host server.

[risoknop]

Though on linux servers it might be a problem to unzip 7z file.

[Jake Arkinstall]

If it's a CSV file which is to be uploaded into MySQL, you may be better doing it from a desktop application rather than the site itself.

For example, you could connect directly to the MySQL database from an application on the user's computer. This method would skip the need to upload at all, and would also save the server alot of work.

Compressing the file requires alot of processing on both sides, and adding weight to a live site isn't a good idea. I'm pretty sure that a direct connection and upload to a MySQL database would be as fast as just compressing the file, maybe faster - let alone upload time and uncompression followed by the queries.

It's an all-round simpler and faster solution that would save your client time and hastle.

[wazo]

do you know of any software that does this?

[Raju Gautam]

- Set your max_upload_size in php.ini higher (it’s only 2M out of the box)
- Set your script timeout to never (set_time_limit(0); in your scripts… don’t do this in your php.ini)
- memory_limit - This may also be an obvious one to some people, but for those of you who don’t know, this restricts how much memory PHP is allowed to consume while processing. When working with images and large files, this needs to be upped to accommodate these needs.
- post_max_size - This was the one killing me, and it was a major “Duh” moment as well. If you aren’t allowed to create a large post, how can you expect to upload a large file?? This should be set the same as your max_upload_size.
- max_input_time - the time that the script should spend in accepting input. This is setting defaults to 60 seconds, and you will probably need to update this as well

These are the few lines from a tutorial which might be helpful to you.

[Jake Arkinstall]

do you know of any software that does this?

Not any that I know of, but I build stuff like this all the time, in C#.Net or Java (depending on your preference).

I can cook one up for you if you're willing to pay for it (PM me with a budget if so )

[felgall]

- Set your max_upload_size in php.ini higher (it’s only 2M out of the box)
- Set your script timeout to never (set_time_limit(0); in your scripts… don’t do this in your php.ini)
- memory_limit - This may also be an obvious one to some people, but for those of you who don’t know, this restricts how much memory PHP is allowed to consume while processing. When working with images and large files, this needs to be upped to accommodate these needs.
- post_max_size - This was the one killing me, and it was a major “Duh” moment as well. If you aren’t allowed to create a large post, how can you expect to upload a large file?? This should be set the same as your max_upload_size.
- max_input_time - the time that the script should spend in accepting input. This is setting defaults to 60 seconds, and you will probably need to update this as well

These are the few lines from a tutorial which might be helpful to you.

Trying to upload a file that big using HTTP (which is optimised for downloading files) will take forever and degrade performance on the whole server while the file is uploading. You should use a protocol intended for uploading files (such as FTP) to upload big files. The default file size in PHP is set to 2Mb because by the time you get to files that big the server performance will be noticably degraded by trying to use HTTP to upload them. There is no upper size limit if you use the correct protocol.

[ruby-lang]

For example, you could connect directly to the MySQL database from an application on the user's computer. This method would skip the need to upload at all, and would also save the server alot of work.

I don't think exposing your database to the world is a wise decision and your ISP probably won't allow it anyway. A simple desktop client to gzip and ftp the file and a cron job to check if there's a new file to decompress and process on the server would probably be a more sound technical solution.

[Jake Arkinstall]

I don't think exposing your database to the world is a wise decision and your ISP probably won't allow it anyway. A simple desktop client to gzip and ftp the file and a cron job to check if there's a new file to decompress and process on the server would probably be a more sound technical solution.

On the contrary, all hosting companies I've used (and I've used quite alot of them) allow access to the database externally. Some allow filters to only allow localhost, but that is easily changed through cpanel.

Gzipping and ftp can take a long time, especially with a file so big and a connection which may not be great - not to mention the server impact. The application (which I'm building now for testing purposes) breaks down the file into chunks and creates queries, each containing a maximum of 1000 rows (which can be changed). Then it queries the database using the given credentials with those queries, meaning it's not all done in one go causing strain.

This allows the client machine to take on the processing that would otherwise have to be taken by the site itself.

[sk89q]

It depends on your situation. I would not consider uploading a uncompressed 1 GB database dump from my computer with my upstream speed of 30 kb/sec. It would take an ungodly amount of time. Rather, I would tar + bzip2 the database dump / database files and upload it then. If I'm working with a very large SQL/CSV/text file, the time and bandwidth savings are 60%-80%.

[wheeler]

165mb is not exactly a big file to process, on either client side or server side.

If you can, transfer the file from server to server - it should be pretty fast (eg. take a minute or two all things going well).

[frank1]

and one thing most of the people forgots is...
use(make) some upload progress bar...when uploading files through php other wise u dont have any idea how much file is uploaded and it can be really annoying to wait for hours....
and another bad news is ...one doesnt have any good upload progress bar with in php..either u need cgi or u need flash...
no thing much can be done

[wheeler]

one doesnt have any good upload progress bar with in php..

There is something I came across recently, supposedly an upload progress bar in php.

Has anyone implemented this?

http://au.php.net/manual/en/apc.conf...ni.apc.rfc1867

[emz]

bz2 compression makes the file the smallest (as far as I know) -- 165 megs isn't that big... I've uploaded multi-gig DB backups dozens of times

I just FTP those bad boys

[felgall]

I just FTP those bad boys

There is no problem with uploading files many times that size using FTP or any other similar protocol designed for file transfers. The protocol to avoid for files bigger than 1Mb is HTTP which starts to get really slow as upload file sizes get bigger since it is intended to be used for downloading files primarily and expects uploads to be only a few hundred characters of text at most.

[emz]

If you want a web interface, PHP facilitates FTP transactions: http://ca3.php.net/ftp

You'd need to ini_set() timeout params so your page doesn't pile drive itself after 45 seconds

[Antnee]

do you know of any software that does this?

If I need to add large amounts of data to a MySQL application I use HeidiSQL, which works a treat, but I've never tried inputting CSV data with it I must admit