SitePoint Sponsor

User Tag List

Results 1 to 19 of 19
  1. #1
    SitePoint Guru
    Join Date
    Oct 2004
    Location
    uk
    Posts
    853
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    best way to uplaod a 165mb file

    Hi guys

    I need to upload a big .csv file to my server

    its 165mb

    whats the quickest way to upload this file?

  2. #2
    SitePoint Guru risoknop's Avatar
    Join Date
    Feb 2008
    Location
    end($world)
    Posts
    834
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    1) Archive it to *.7z.

    2) Maybe split into few chunks in case you have crappy ftp and are afraid it will die in the middle of upload.

    3) Unzip it on server.

  3. #3
    SitePoint Addict
    Join Date
    Jul 2008
    Location
    sudo rm -rf /
    Posts
    386
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    FTP or wgeting from your home server to your host server.

  4. #4
    SitePoint Guru risoknop's Avatar
    Join Date
    Feb 2008
    Location
    end($world)
    Posts
    834
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Though on linux servers it might be a problem to unzip 7z file.

  5. #5
    Theoretical Physics Student bronze trophy Jake Arkinstall's Avatar
    Join Date
    May 2006
    Location
    Lancaster University, UK
    Posts
    7,062
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    If it's a CSV file which is to be uploaded into MySQL, you may be better doing it from a desktop application rather than the site itself.

    For example, you could connect directly to the MySQL database from an application on the user's computer. This method would skip the need to upload at all, and would also save the server alot of work.

    Compressing the file requires alot of processing on both sides, and adding weight to a live site isn't a good idea. I'm pretty sure that a direct connection and upload to a MySQL database would be as fast as just compressing the file, maybe faster - let alone upload time and uncompression followed by the queries.

    It's an all-round simpler and faster solution that would save your client time and hastle.
    Jake Arkinstall
    "Sometimes you don't need to reinvent the wheel;
    Sometimes its enough to make that wheel more rounded"-Molona

  6. #6
    SitePoint Guru
    Join Date
    Oct 2004
    Location
    uk
    Posts
    853
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    do you know of any software that does this?

  7. #7
    rajug.replace('Raju Gautam'); bronze trophy Raju Gautam's Avatar
    Join Date
    Oct 2006
    Location
    Kathmandu, Nepal
    Posts
    4,013
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    - Set your max_upload_size in php.ini higher (it’s only 2M out of the box)
    - Set your script timeout to never (set_time_limit(0); in your scripts… don’t do this in your php.ini)
    - memory_limit - This may also be an obvious one to some people, but for those of you who don’t know, this restricts how much memory PHP is allowed to consume while processing. When working with images and large files, this needs to be upped to accommodate these needs.
    - post_max_size - This was the one killing me, and it was a major “Duh” moment as well. If you aren’t allowed to create a large post, how can you expect to upload a large file?? This should be set the same as your max_upload_size.
    - max_input_time - the time that the script should spend in accepting input. This is setting defaults to 60 seconds, and you will probably need to update this as well

    These are the few lines from a tutorial which might be helpful to you.
    Mistakes are proof that you are trying.....
    ------------------------------------------------------------------------
    PSD to HTML - SlicingArt.com | Personal Blog | ZCE - PHP 5

  8. #8
    Theoretical Physics Student bronze trophy Jake Arkinstall's Avatar
    Join Date
    May 2006
    Location
    Lancaster University, UK
    Posts
    7,062
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    do you know of any software that does this?
    Not any that I know of, but I build stuff like this all the time, in C#.Net or Java (depending on your preference).

    I can cook one up for you if you're willing to pay for it (PM me with a budget if so )
    Jake Arkinstall
    "Sometimes you don't need to reinvent the wheel;
    Sometimes its enough to make that wheel more rounded"-Molona

  9. #9
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,875
    Mentioned
    25 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by rajug View Post
    - Set your max_upload_size in php.ini higher (itís only 2M out of the box)
    - Set your script timeout to never (set_time_limit(0); in your scriptsÖ donít do this in your php.ini)
    - memory_limit - This may also be an obvious one to some people, but for those of you who donít know, this restricts how much memory PHP is allowed to consume while processing. When working with images and large files, this needs to be upped to accommodate these needs.
    - post_max_size - This was the one killing me, and it was a major ďDuhĒ moment as well. If you arenít allowed to create a large post, how can you expect to upload a large file?? This should be set the same as your max_upload_size.
    - max_input_time - the time that the script should spend in accepting input. This is setting defaults to 60 seconds, and you will probably need to update this as well

    These are the few lines from a tutorial which might be helpful to you.
    Trying to upload a file that big using HTTP (which is optimised for downloading files) will take forever and degrade performance on the whole server while the file is uploading. You should use a protocol intended for uploading files (such as FTP) to upload big files. The default file size in PHP is set to 2Mb because by the time you get to files that big the server performance will be noticably degraded by trying to use HTTP to upload them. There is no upper size limit if you use the correct protocol.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">

  10. #10
    SitePoint Addict ruby-lang's Avatar
    Join Date
    Aug 2007
    Posts
    389
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by arkinstall View Post
    For example, you could connect directly to the MySQL database from an application on the user's computer. This method would skip the need to upload at all, and would also save the server alot of work.
    I don't think exposing your database to the world is a wise decision and your ISP probably won't allow it anyway. A simple desktop client to gzip and ftp the file and a cron job to check if there's a new file to decompress and process on the server would probably be a more sound technical solution.

  11. #11
    Theoretical Physics Student bronze trophy Jake Arkinstall's Avatar
    Join Date
    May 2006
    Location
    Lancaster University, UK
    Posts
    7,062
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    I don't think exposing your database to the world is a wise decision and your ISP probably won't allow it anyway. A simple desktop client to gzip and ftp the file and a cron job to check if there's a new file to decompress and process on the server would probably be a more sound technical solution.
    On the contrary, all hosting companies I've used (and I've used quite alot of them) allow access to the database externally. Some allow filters to only allow localhost, but that is easily changed through cpanel.

    Gzipping and ftp can take a long time, especially with a file so big and a connection which may not be great - not to mention the server impact. The application (which I'm building now for testing purposes) breaks down the file into chunks and creates queries, each containing a maximum of 1000 rows (which can be changed). Then it queries the database using the given credentials with those queries, meaning it's not all done in one go causing strain.

    This allows the client machine to take on the processing that would otherwise have to be taken by the site itself.
    Jake Arkinstall
    "Sometimes you don't need to reinvent the wheel;
    Sometimes its enough to make that wheel more rounded"-Molona

  12. #12
    SitePoint Wizard
    Join Date
    Mar 2008
    Posts
    1,149
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    It depends on your situation. I would not consider uploading a uncompressed 1 GB database dump from my computer with my upstream speed of 30 kb/sec. It would take an ungodly amount of time. Rather, I would tar + bzip2 the database dump / database files and upload it then. If I'm working with a very large SQL/CSV/text file, the time and bandwidth savings are 60&#37;-80%.

  13. #13
    SitePoint Wizard wheeler's Avatar
    Join Date
    Mar 2006
    Location
    Gold Coast, Australia
    Posts
    1,369
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    165mb is not exactly a big file to process, on either client side or server side.

    If you can, transfer the file from server to server - it should be pretty fast (eg. take a minute or two all things going well).
    Studiotime - Time Management for Web Developers
    to-do's, messages, invoicing, reporting - 30 day free trial!
    Thomas Multimedia Web Development

  14. #14
    SitePoint Wizard frank1's Avatar
    Join Date
    Oct 2005
    Posts
    1,392
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    and one thing most of the people forgots is...
    use(make) some upload progress bar...when uploading files through php other wise u dont have any idea how much file is uploaded and it can be really annoying to wait for hours....
    and another bad news is ...one doesnt have any good upload progress bar with in php..either u need cgi or u need flash...
    no thing much can be done

  15. #15
    SitePoint Wizard wheeler's Avatar
    Join Date
    Mar 2006
    Location
    Gold Coast, Australia
    Posts
    1,369
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by frank1 View Post
    one doesnt have any good upload progress bar with in php..
    There is something I came across recently, supposedly an upload progress bar in php.

    Has anyone implemented this?

    http://au.php.net/manual/en/apc.conf...ni.apc.rfc1867
    Studiotime - Time Management for Web Developers
    to-do's, messages, invoicing, reporting - 30 day free trial!
    Thomas Multimedia Web Development

  16. #16
    SitePoint Member
    Join Date
    Jul 2007
    Location
    Toronto
    Posts
    17
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    bz2 compression makes the file the smallest (as far as I know) -- 165 megs isn't that big... I've uploaded multi-gig DB backups dozens of times

    I just FTP those bad boys

  17. #17
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,875
    Mentioned
    25 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by emz View Post
    I just FTP those bad boys
    There is no problem with uploading files many times that size using FTP or any other similar protocol designed for file transfers. The protocol to avoid for files bigger than 1Mb is HTTP which starts to get really slow as upload file sizes get bigger since it is intended to be used for downloading files primarily and expects uploads to be only a few hundred characters of text at most.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">

  18. #18
    SitePoint Member
    Join Date
    Jul 2007
    Location
    Toronto
    Posts
    17
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    If you want a web interface, PHP facilitates FTP transactions: http://ca3.php.net/ftp

    You'd need to ini_set() timeout params so your page doesn't pile drive itself after 45 seconds

  19. #19
    SitePoint Guru
    Join Date
    Nov 2004
    Location
    England
    Posts
    702
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by wazo View Post
    do you know of any software that does this?
    If I need to add large amounts of data to a MySQL application I use HeidiSQL, which works a treat, but I've never tried inputting CSV data with it I must admit


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •