SitePoint Sponsor

User Tag List

Results 1 to 13 of 13
  1. #1
    SitePoint Wizard silver trophybronze trophy Stormrider's Avatar
    Join Date
    Sep 2006
    Location
    Nottingham, UK
    Posts
    3,133
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    File Uploads & Version Control

    I just wanted to see what people's techniques were for dealing with file uploads on a site in version control.

    I'm sure many of you, like me, have your websites under some kind of version control - for me it's Subversion but there are many different ones out there.

    Websites quite often allow file uploads for the users, or even generate files itself which it puts in a specified folder, but these files aren't under version control like the codebase itself, or the other files you might have in the project. When it comes to checking the code out elsewhere, or updating the site, these files get 'left out' and aren't put in the new version - I export the site from SVN when I want to update rather than serve from a working copy, which can be a bad idea if your server isn't set up correctly.

    I have thought about using a symbolic link to deal with it, but I wanted to see if anyone else has come across this problem, and how they solved it.

    Any suggestions?

  2. #2
    SitePoint Member
    Join Date
    Jan 2010
    Posts
    12
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    It is good practice to use test data, rather than live data, on your development server, so SVN should never see the user files.

    Just set svn-ignore to exclude the folders where your application stores these files, and you should then be able to rsync your exported application code to your web server. Just don't set the --delete flag!

    Personally, I don't always do it that way. If I know I've only changed a few files, I'll check the svn history and manually scp them.

  3. #3
    SitePoint Wizard silver trophy kyberfabrikken's Avatar
    Join Date
    Jun 2004
    Location
    Copenhagen, Denmark
    Posts
    6,157
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Stormrider View Post
    I have thought about using a symbolic link to deal with it, but I wanted to see if anyone else has come across this problem, and how they solved it.

    Any suggestions?
    I generally have a deploy script that exports the repository and then symlinks certain folders into a shared place. So yes, I use symlinks for that. I think that's a pretty common solution.

  4. #4
    SitePoint Wizard silver trophybronze trophy Stormrider's Avatar
    Join Date
    Sep 2006
    Location
    Nottingham, UK
    Posts
    3,133
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Yeh, I have a rollout script at the moment that does an export to a folder like exports/r56/ or whatever, then points a symlink called 'site' to the new export - apache serves from this symlink.

    The only trouble with using symlinks for this is that I do my development under Windows.. which has no such thing. But I'm sure I can figure out something.

    I already have an 'environment' config that sets up different paths etc depending on whether it is the dev, test or live environments, so I should be able to use that to change the uploads path as well.

  5. #5
    Floridiot joebert's Avatar
    Join Date
    Mar 2004
    Location
    Kenneth City, FL
    Posts
    823
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I keep three things going.

    1) Traditional version control with SVN for the programming logic, templates, etc
    2) An uncompressed backup of the site maintained via rsync that includes generated content/etc
    3) A compressed version of the rsync backup with the last known good configuration

  6. #6
    SitePoint Evangelist AlienDev's Avatar
    Join Date
    Feb 2007
    Location
    UK
    Posts
    591
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Stormrider View Post
    The only trouble with using symlinks for this is that I do my development under Windows.. which has no such thing.
    Vista has symlinks

    I'm using them right now.
    Me on StackOverflow | Blog & personal website.

    I mostly use: PHP, Java, JavaScript, Android.

  7. #7
    <?php while(!sleep()){code();} G.Schuster's Avatar
    Join Date
    Mar 2007
    Location
    Germany
    Posts
    428
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Stormrider View Post
    The only trouble with using symlinks for this is that I do my development under Windows.. which has no such thing. But I'm sure I can figure out something.
    They're called "JunctionPoints".
    To create them, use a software like "JunctionLink Magic": http://www.rekenwonder.com/linkmagic.htm

  8. #8
    SitePoint Wizard silver trophy kyberfabrikken's Avatar
    Join Date
    Jun 2004
    Location
    Copenhagen, Denmark
    Posts
    6,157
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by G.Schuster View Post
    They're called "JunctionPoints".
    To create them, use a software like "JunctionLink Magic": http://www.rekenwonder.com/linkmagic.htm
    I believe that Vista and Windows 7 (Or is it just Windows 7?) has real symlinks. Older versions of ntfs had junctionpoints, which are only sort-of-like symlinks.

  9. #9
    SitePoint Wizard Ren's Avatar
    Join Date
    Aug 2003
    Location
    UK
    Posts
    1,060
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I'd use content addressing, all uploaded files get stored based on the hash of their content.
    Then a database tracks application names to the hash over time.

    In essence application implemented symlinks, with the added benefit of history, so can revert/rollback to any prior version.

    Also would allow branching, so could run the same application, but with different skins/templates.

  10. #10
    SitePoint Enthusiast
    Join Date
    Oct 2008
    Posts
    42
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You can create a repo just for backups. SVN will work with binary data and even SQL files. Kind of neat how it only stores the difference in the SQL exports, so its an incremental backup solution of sorts.

    But again you may want to just have a separate repo so it keeps the logs and revision #s separate. I don't like the rsync idea because rsync would also copy over, say... a corrupted file, overwriting the backed up version.

  11. #11
    Floridiot joebert's Avatar
    Join Date
    Mar 2004
    Location
    Kenneth City, FL
    Posts
    823
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by jshpro2 View Post
    I don't like the rsync idea because rsync would also copy over, say... a corrupted file, overwriting the backed up version.
    That's what a compressed "last good known configuration" copy of the rsync directory is for.

  12. #12
    SitePoint Enthusiast
    Join Date
    Oct 2008
    Posts
    42
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Scenario.

    You overwrite your "last good" with a new backup you think is good. You realize you did not test *everything* and some files got corrupted. How do you get them back?

  13. #13
    SitePoint Wizard
    Join Date
    Mar 2008
    Posts
    1,149
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by kyberfabrikken View Post
    I believe that Vista and Windows 7 (Or is it just Windows 7?) has real symlinks. Older versions of ntfs had junctionpoints, which are only sort-of-like symlinks.
    Vista and up have real symlinks. Junction points are also handled much better than in XP (it's harder to do something stupid like delete the original files).

    Use mklink in command prompt to create either symlinks or junction points. It's pretty useful.

    And a better analogy for "content addressing" might be inodes.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •