SitePoint Sponsor

User Tag List

Results 1 to 11 of 11
  1. #1
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Cross Server File Transfer

    I need to transfer files between a network of servers all over the world. I have looked into using cURL and ftp_fput() but am having little luck. I can get a transfer going from my local to the server or visa versa but not from server to server.

    Basically im trying to replicate Akamai on a very small scale. This is the scenario. I am working from home. I search for a file on Web Server 1 in Oregon. The file doesn't exist. I search my other servers and find that it is located on Server 2 in Germany. I then need to COPY the file from Web Server 2 to Web Server 1. That's it.

    I have it all in place except getting the server-to-server transfer to work. Who can help? Thanks much.

  2. #2
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    One other thing. The connections need to be made via FTP. I have IP, USER & PASSWORDS for each account. Thanks.

  3. #3
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2008
    Posts
    5,757
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    What problem are you having with server to server? Nothing is different. Something will need to trigger the ftp script to execute. You could do it manually, or set a cron job to start some type of sync routine.

  4. #4
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks for all your input. Let me explain a little more about what i have done. First of, each of the files I'm transferring are over 1Gb so file_get_contents() wont work. Files are not transferred until requested by the user. I would prefer to do it all in CURL, but as i said im having issues.

    I have looked into SST scripts but none seems to satisfy.

    My thinking is that cURL cannot directly transfer a file server to server (please correct if wrong). I thought that the way to make it work would be to call a file on Web Server 2 using cURL and add some parameters (file name to transfer) to the call so that when ran it would using cURL transfer the file to Web Server 1. Sound good? Problem, Web Server 2 doesn't have a URL that will resolve to a specific file on the server. Therefor again only cURL via FTP login can be used (as i see it).

    So do further define my question, with cURL, is there a way to transfer files between servers directly using a curl statement like
    Code:
    $ch = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v");
    but one that works more like below with a location to get and set from/to. I know the following code is incorrect, i just would like to know how to get it right.
    Code:
    $c = curl_init("ftp://theuser:thepass@72.47.251.180/testfile.m4v","ftp://theuser:thepass@42.76.151.236/testfile.m4v");
    I know i have left out parts of the cURL process. curl_exec() and curl_close() are pretty standard. Thanks.

  5. #5
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2008
    Posts
    5,757
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You need to host a copy of a script on each server.

    The script can do one of two things:
    1) send one of its own local files to another server via ftp
    2) grab a remote file from another server via ftp and save locally


    You could allow the script to be commanded remotely. Eg
    http://server2/sync.php?filename=foo...rverid=server6

    Now server2 will try to get the file from server6(or send to it, w/e you want)

    Obviously, this needs to be secured. Each script should have access to a list of allowed servers and the appropriate login info. Grats, you've built a network.

    Transfering a gb will take a while, and theres a decent chance you fail before finished.
    You could resume the transfer where you left off. php's native ftp functions give you the tools needed to resume a transfer. curl probably does too, I'm just not familiar with it.


    But....have you seen rsync and similar tools?

  6. #6
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    But....have you seen rsync
    This is my next step.

  7. #7
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    SOLVED -

    Thanks for everyones help. I am posting my full function for those who may find it useful. To recap, this function "try's" to simulate Akamai's service.

    1. GeoLoc user.
    2. GeoLoc all servers
    3. Calculate distance between user and each server.
    4. Sort by distance.
    5. Check to see if file is at closest server. If so download.
    6. If file not at closest server, copy file from mother server to server missing file. Then notify user via email with download link to closest server.

    Code:
    function sst() {
    
    	global $_GET, $_SERVER;
    
    	/*   start geo loc   */
    	// http://www.imaginerc.com/software/GeoCalc/
    	require_once('standards/classes/geoplugin.class.php'); 
    	$geoplugin = new geoPlugin();
    	// http://www.phpclasses.org/browse/file/25516.html
    	include_once("standards/classes/GeoCalc.class.php"); 
    	$oGC = new GeoCalc();
    
    	/*   set server ip's and FTP logins		*/
    	$servers = array($_SERVER['REMOTE_ADDR'],'xxx.xx.xx.xxx','xxx.xx.xx.xxx');
    	$user = array('','usr1','usr2');
    	$pass = array('','pass1','pass2',);
    
    	/*   get user location   */
    	$geo[0]['ip'] = $servers[0];
    	$geoplugin->locate($geo[0]['ip']);
    	$geo[0]['lat'] = $geoplugin->latitude;
    	$geo[0]['lon'] = $geoplugin->longitude;
    	$geo[0]['city'] = $geoplugin->city;
    	$geo[0]['countryName'] = $geoplugin->countryName;
    
    	/*   get  servers data  */
    	for ($i = 1; $i < count($servers); $i++) {
    
    		$geo[$i]['ip'] = $servers[$i];
    		$geo[$i]['user'] = $user[$i];
    		$geo[$i]['pass'] = $pass[$i];
    		$geoplugin->locate($geo[$i]['ip']);
    		$geo[$i]['lat'] = $geoplugin->latitude;
    		$geo[$i]['lon'] = $geoplugin->longitude;
    		$geo[$i]['city'] = $geoplugin->city;
    		$geo[$i]['countryName'] = $geoplugin->countryName;
    		$geo[$i]['dist'] = $oGC->GCDistance($geo[0]['lat'],$geo[0]['lon'],$geo[$i]['lat'],$geo[$i]['lon']);
    
    	}
    
    	/*   order by distance.   */
    	$geo = orderBy($geo, 'dist');
    
    	/*   connect to closeest server   */
    	$conn_id = ftp_connect($geo[1]['ip']) or die("Couldn't connect to $ftp_server");
    	ftp_login($conn_id, $geo[1]['user'], $geo[1]['pass']);
    	$contents = ftp_nlist($conn_id, "tts/incoming");
    	ftp_close($conn_id);
    
    	$localfile = '/user/dac15/tts/incoming/';
    	$remotefile = '/user/'.$geo[1]['user'].'/tts/incoming/';
    
    	/*  if file is on the closest server.    */
    	if (find($_GET['name'],$contents)) {
    
    		// http://www.awesomephp.com/?Tutorials*16/Download-file-with-resume,-stream-and-speed-options.html
    		downloadFile($remotefile,$_GET['name'],900,false); 
    
    	}else{ 
    		
    		/*   push file to the closest server requesting server.*/
    		$ch = curl_init();
    		$fp = fopen($localfile.$_GET['name'], 'r');
    		curl_setopt($ch, CURLOPT_URL, 'ftp://'.$geo[1]['user'].':'.$geo[1]['pass'].'@'.$geo[1]['ip'].'/'.$remotefile.$_GET['name']);
    		curl_setopt($ch, CURLOPT_UPLOAD, 1);
    		curl_setopt($ch, CURLOPT_INFILE, $fp);
    		curl_setopt($ch, CURLOPT_INFILESIZE, filesize($localfile.$_GET['name']));
    		curl_exec($ch);
    		$error_no = curl_errno($ch);
    		curl_close($ch);
    		if ($error_no == 0) {
    			$error = 'File uploaded succesfully.';
    
    			$mailto[0][0] = $_SERVER['HTTP_EMAIL'];
    			$mailto[0][1] = $_SERVER['HTTP_FNAME'];
    			$mailto[0][2] = $_SERVER['HTTP_LNAME'];
    
    			$content="
    			Dear ".$mailto[0][1]."
    			<br>
    			Your file is ready for download. Go to this link in your preferred web browser.
    			";
    			
    			// http://www.xpertmailer.com/
    			if (smtp_mail($mailto,"DoD File Download",$content)) { Return 1; }
    
    		}else{
    			$error = 'File upload error.';
    			echo $error;
    		}
    
    	}
    
    	die;
    
    }
    
    function orderBy($data, $field, $reverse_sort='') {
    	$code = "return strnatcmp(\$a['$field'], \$b['$field']);";
    	usort($data, create_function('$a,$b', $code));
    	if ($reverse_sort==1) { krsort($data); }
    	return $data;
    }

  8. #8
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I have another question to go along with this post. Once the cURL transfer is started on my 1GB file the screen just sits white while doing its thing (page load bar crawls). Is there a way to finish loading the page while the transfer is in process. This way the user doesn't have to wait for the transfer to finish, they are just there to make it happen. (they are not downloading at the time so it would be okay for them to continue). Thanks.

  9. #9
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2008
    Posts
    5,757
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I'd recomend you just insert a record into a "queue" table.
    A seperate php script can run via a cron job which pulls jobs out of the queue table and completes them. Since it may take a while to do certain uploads, make sure either only a limited number of instances of the cron job script can run at a time, and make sure they don't work on the same upload.

  10. #10
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    @crmalibu - thanks. your the second to suggest this.

  11. #11
    SitePoint Addict
    Join Date
    Jun 2005
    Posts
    196
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Code:
    /**
     * Takes a needle and haystack (just like in_array()) and does a wildcard search on it's values.
     *
     * @param    string        $string        Needle to find
     * @param    array        $array        Haystack to look through
     * @result    array                    Returns the elements that the $string was found in
     */
    function find ($string, $array = array ())
    {
    	foreach ($array as $key => $value) {
    		unset ($array[$key]);
    		if (strpos($value, $string) !== false) {
    			$array[$key] = $value;
    		}
    	}
    	return $array;
    }


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •