Spidering a website

[Latox]

I have coded a link exchange script, somebody submits their website, adds our link and then their site is shown after we have received a hit.

I want to advance it so a cron/spider automatically runs and checks X website in our database to see if our link is still on their site, and if not set their link to inactive in our database and e-mail them.

I know how to do all of this apart from coding a feature to check if our link is on their site, I have never done this before and I was just wondering how it is done and if anybody can help.

Thanks

[AnthonySterling]

It sounds like you have the logic sorted and you just need a pointer or two.

Use cURL to obtain the website homepage, parse it using DOMDocument & XPath to extract all the links, then loop through these links to see if your site is one of them.

Depending on whether or not your want to go into sub pages, cycle through the links to find all which contain the correct base address, then start the process again for this page.

This could quickly become a quite intensive and lengthy process, as such, you should probably implement some kind of queuing system for a script (launched via cron) to cycle through.

[Latox]

OK Thanks a lot

Do you know of any tutorials? I get a better understanding when reading :P

[AnthonySterling]

Sorry, no. Although a quick search using the tools mentioned above as keywords should sort you out.

[Latox]

XPath = XML?

Do I need XML experience for this?

[AnthonySterling]

No, as HTML is structured, especially xHTML, so you can provide a pattern to target specific elements.

Although XPath's origins do come from XML.

[Latox]

OK Thanks a lot.

Do you think many people would pay for a link exchange system like this if I decided to sell it?

http://www.glavna.com/view/exchange (Tracks hits from actual referring domain, not a unique count URL so it's good for SEO too.)

[uniqueumang]

yep curl should do it.

[shadow_of__soul]

Hi,

i going to give you a tip to search the code of your link in other website with cURL

$ch=curl_init("http : / / remoteurl . com");
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,true);
$content=curl_exec($ch);
curl_close($ch);
if(strpos($content,"youhtmllinkcode")===false){
echo "the code it's not there :( ";
}

i hope you find this useful

Regards,
Shadow.

PD: i was need to put spaces in the url to submit it

[Latox]

Thank you

[Latox]

Done and coded.

A sample e-mail a user gets when the cron is ran:

Hello,

We are sending you this e-mail because our automatic link exchange spider has visited your website, http://www.youtubetomp3.net and cannot find our website link.

Please follow the steps on http://www.glavna.com/validate/22/exchange to re-activate your link.

Kind regards,
Glavna.com Admin