grabing a part of page

[dacool]

almost forgot..

**** Just noticed: if sitename or site_desc has a comma or apstrophe, the code is broken and cannot update the database, is there a way of stripping these as to avoid the errors?

add mysql_escape_string() function if the value is string.. change code like this:

        if ($fields[$k] != $sql_unique_field) 
        { 
          $sql .= ($fields[$k] . ' = ' . ((is_string($val)) ? ("'".mysql_escape_string($val)."'") : $val) . ','); 
        } 
        else 
        { 
          $sql_where = (" WHERE $sql_unique_field = " . ((is_string($val)) ? ("'".mysql_escape_string($val)."'") : $val)); 
        } 

this should do the trick..

[zelda]

that makes sense, although changed code now produces an error

Warning: Invalid argument supplied for foreach() in

which refers to the line

foreach ($arr_values as $k => $val) 

adding mysql_escape_string code now, thanks for that

[dacool]

my mistake.. try this..

foreach($values as $arr_values) 
{ 
    $sql = "UPDATE $dest_table SET "; 
    $arr_values = explode(',', $arr_values); // <<<<<<
    foreach ($arr_values as $k => $val) 
    { 

[zelda]

error is now gone but still only updating the one site's data, loop does not seem to be working

[dacool]

print the values array to check how entries are in it..

$values = get("http://www.yourdomain.com/_stats.php", "<div>", "</div>");
print_r($values); 

also can check what SQL queries is the script doing:

echo "<br>SQL: $sql<br>";
mysql_query($sql) or die('Could not update the database!'); 

[zelda]

If I add a print under each url, its printing out the results for both sites stats, which is correct, is reading them as it should

Array ( [0] => 
yourdomain,yourdesc,yourdomain.com,1022524652,710,8280,566
) Array ( [0] => 
yourdomain1,yourdesc1,yourdomain1.com,1078947446,51,100,26
)

Echoing the $sql only returns one result

SQL: UPDATE site_toplist SET sitename = '
yourdomain1',site_desc = 'yourdesc1',board_startdate = '1078947446',users = '51',posts = '100',unique_hits = '26
' WHERE server_name = 'yourdomain1.com'

which is the second site listed, would I need to assign the second site to $values1 or something? Is it due to the two sites being set as the $values ?

[dacool]

If I add a print under each url, its printing out the results for both sites stats

what do you mean? aren't all urls in one array? does the get() function return all urls in one array or you have to call it for each url? from the code of anurag i think that this function will return all urls in one array and when you call print_r($values) it will return list of all urls? so call the function ONLY ONCE and check if it returns you all the urls....

$values = get("http://www.yourdomain.com/_stats.php", "<div>", "</div>"); 
print_r($values); 

this must return you array with all urls.. check it and post a reply is this right or not..

[zelda]

no i have both urls on sep $value lines, as per my code posted a few posts back still. How do I include them in the one $value array? And will this allow the adding of more later the same way?

[dacool]

try this class.. i wrote it with my function and it's pretty simple.. i haven't tested it yet and i'm waiting for suggestions for optimization from all of you people

class power_fetch {

  var $url;
  var $content;
  var $offset;
  var $max_offset;
  var $src_file;

function power_fetch($url) {
  $this->offset = 1;
  $this->url = $url;
  $this->src_file = tempnam('/tmp', 'src');
  copy($this->url, $this->src_file);
  $src = fopen($this->src_file, 'r');
  if ($src) {
    $this->content = fread($src, filesize($this->src_file));
    fclose($src);
    $this->max_offset = strlen($this->content);
  } else {
    echo 'ERROR: cannot connect to site (' . $this->url . ')!';
  }
}

function destroy() {
  unlink($this->src_file);
}

function adv_fetch($start_str, $end_str) {
  $start_pos = strpos($this->content, $start_str, $this->offset);
  if (($start_pos !== false) AND ($start_pos > $this->offset)) {
    $start_pos += strlen($start_str);
    $end_pos = strpos($this->content, $end_str, $start_pos);
    if ($end_pos !== false) {
      if ($end_pos < $this->max_offset) {
        $this->offset = $end_pos + strlen($end_str);
      }
      else {
        $this->offset = $this->max_offset;
      }
      $temp = substr($this->content, $start_pos, ($end_pos-$start_pos));
      return $temp;
    }
  }
  return false;
}

} 

use it this way:

mysql_select_db('****', $link); 

// Urls to crawl for statistics files 
$pf = new power_fetch('http://www.yourdomain.com/_stats.php');
       
$dest_table = 'site_toplist'; 
$fields = array('sitename', 'site_desc', 'server_name', 'board_startdate', 'users', 'posts', 'unique_hits'); 
$sql_unique_field = 'server_name'; 

// Update sites with new statistics 
while ($values = $pf->adv_fetch('<div>', '</div>')) 
{ 
    $sql = "UPDATE $dest_table SET ";
    $values = explode(',', $values); 
    foreach ($values as $k => $val) 
    { 
        if ($fields[$k] != $sql_unique_field) 
        { 
          $sql .= ($fields[$k] . ' = ' . ((is_string($val)) ? ("'$val'") : $val) . ','); 
        } 
        else 
        { 
          $sql_where = (" WHERE $sql_unique_field = " . ((is_string($val)) ? ("'$val'") : $val)); 
        } 
    } 
    $sql = substr($sql, 0, -1); // this will remove the last ',' 
    if (!empty($sql_where)) $sql .= $sql_where; 
    mysql_query($sql) or die('Could not update the database!'); 
}
$pf->destroy(); 

[zelda]

I replaced all the coding in my file with your new function and tested with one site. The data is being updated in the database all fine with one exception, the <div> tag is being added to the sitename field.

How would I go about adding a second site?

$pf = new power_fetch('http://www.yourdomain.com/_stats.php','http://www.yourdomain1.com/_stats.php',); 

or do I duplicate the whole line?

[zelda]

I've fixed the <div> tag problem by included more of the html code to grab

while ($values = $pf->adv_fetch('<td><div>', '</div></td>')) 

Changed this line to hardcode the filename as is same on all sites

$this->offset = 1;
$this->url = $url . '/_stats.php'; 

Just can not get it to work with two sites, I've tried both urls in the same line sep with comma, on sep lines ?

[anurag]

I think you should create a new object to fetch the second site's data

Something like this

$pf_one = new power_fetch('http://www.yourdomain.com/_stats.php');

$pf_two = new power_fetch('http://www.yourdomain2.com/_stats.php'); 

[zelda]

That didnt work, at least one has to be named $pf or else just produces error. Then if I set one to $pf_one is isnt being read

[zelda]

It's finally finished!!!!!!!! with some help from another friend who read the server_names from the database and then set the $pf to scrape each entry from their is working like a dream

Completely automated via a cron job it reads the urls from the db, scrapes the sites on a regular basis and updates the database with the new stats. Members can submit sites via a form, which must be approved via an Admin interface before being displayed publically.

An Option for members to add an image to their site, from which the id number reads and displays their site details and rating!

Exactly what I wanted, huge love to aranug and dacool, without you guys i would have not even come close. I have added a credit for you on the bottom of the live script, if you would like a link to your site added, please by all means just let me know

For those interested in the finished result, see it Here.

Onwards.......

[anurag]

Sounds Great!

Congrats on getting your project working and listed on SourceForge

[dacool]

zelda, i'm very happy that i could help you and that your project is finished and working
thank you for the credit, i really appreciate it..