what would be the best way to code this?

[LAMP]

hello everyone,

I am trying to find a best way to achieve this:
I need to create a web crawler that constantly crawls a page, picks the links, crawls those links before further crawling more links. I was wondering if PHP would be a good choice to accomplish to this. Can a PHP script be coded to run forever (may be by setting the timeout to infinite)? is it going to be a good idea? are there better options for this kind of thing?

LAMP

[Kayzio]

cron job to execute PHP script every X amount of time, use file_get_contents to get contents of page to crawl, use link regex to get links out of contents (have a look here http://regexlib.com/Search.aspx?k=link), then with new links put them in a database. The cron job could run for each link in the database and then remove the link after the script is done, putting more links in the database. (basically a queue that continues to add more links as links are removed) would basically continue to crawl without crawling too much at one time (limit number of links to crawl at a time - all automated without crashing your server). I have never tried this but I believe this would probably be a good way to do it. Actually don't remove links from the database just have a column called date, and when you crawl the link you can put a timestamp of when it was crawled. This way you don't end up crawling links over and over again (write script to achieve this), and you could also write the script so that if it has been over X amount of time since a link has been crawled, crawl it again.

[Andrewaclt]

I personally wouldn't use php for this, I would go with perl or python. There are a lot of classes to do a lot of the hard work for you for these languages all over the internet.