SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Member
    Join Date
    Apr 2010
    Posts
    21
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Large (5GB) uploads

    Hi,

    we need to be able to provide our client with the ability to upload large files through their PHP application. They actually requested 'unlimited' file sizes, but in reality we're looking at 5GB max.

    We started off using a standard form with a file field, this was fine for large files up to about 1GB although we did have to adjust the server's PHP config max_post and max_upload values to cater for this. As they have requested the ability to upload larger and larger files this method has become problematic both in terms of the memory usage and the reliability of the a HTTP connection remaining stable long enough for the uploads to complete.

    I know it must be possible to give them the ability to upload 5GB files because I see other web services offering much larger uploads (YouTube can handle over 20GB).

    We've looked at the HTML5 Files API which should allow uploads to be completed in 'chunks' which are then patched together on the server. I'm not entirely clear on how this works - for example does each 'chunk' just count as a single 200kb upload, or will we still need a large memory setting on the server? Even this is not ideal as the majority of the company are using IE8 in which the HTML5 APIs don't work.

    I just wondered which the best way to approach this is, we're open to trying pretty much anything.

    Thank you

  2. #2
    Community Advisor bronze trophy
    fretburner's Avatar
    Join Date
    Apr 2013
    Location
    Brazil
    Posts
    1,446
    Mentioned
    45 Post(s)
    Tagged
    13 Thread(s)
    Hi punkstjimmy,

    I've used Plupload to handle uploads in the past, and that has built-in support for chunking.. you should be able to handle files of any size that way, and as far as I know it shouldn't require large amounts of memory, because each chunk is sent in a separate request. Also, the range of browsers supported is broad, so you don't have to confine yourself to HTML5 only.

  3. #3
    SitePoint Wizard siteguru's Avatar
    Join Date
    Oct 2002
    Location
    Scotland
    Posts
    3,631
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    I've used chunking before too. (I created my own code to do it - it was a long time ago and there were no readily-available code snippets at the time). As long as it is handled right it can work really well for large files.
    Ian Anderson
    www.siteguru.co.uk

  4. #4
    SitePoint Member
    Join Date
    Jul 2013
    Posts
    1
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Maybe the best option is 'chunking' very large files but I don't think PHP is going to make uploading large files easy if consistently possible at all. Especially anyone utilizing a shared hosting environment. In those cases you need a script that is really handling FTP services like Java. http://www.simple2ftp.com has a php script that handles the application around the Java FTP code and makes it pretty easy for users and web owners to setup and use. Depending on what your needs are and what resources you have access to you might want to try this script.

  5. #5
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,869
    Mentioned
    25 Post(s)
    Tagged
    1 Thread(s)
    Sounds like the client is trying to do the equivalent of flying around the world on a bicycle.

    The form upload option uses HTTP which is fast in the download direction but very slow in the upload direction because it is intended for displaying of web pages. A far better solution when dealing with large files would be to use a service that is actually intended for handling files such as FTP, SFTP, XCom etc. where 50Tb files are no bigger problem than 5k files are.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">


Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •