SitePoint Sponsor

User Tag List

Page 2 of 2 FirstFirst 12
Results 26 to 33 of 33

Thread: Php crawler

  1. #26
    SitePoint Enthusiast
    Join Date
    Feb 2009
    Location
    Athens, Greece
    Posts
    68
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi theblackjacker

    you seem like wanting other people to write your script

    You should better use an existing script like http://php-crawler.sourceforge.net for example or search for "spider" in Hot Scripts.

  2. #27
    SitePoint Member
    Join Date
    Dec 2005
    Posts
    24
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by 01globalnet View Post
    Hi theblackjacker

    you seem like wanting other people to write your script

    You should better use an existing script like http://php-crawler.sourceforge.net for example or search for "spider" in Hot Scripts.
    Sorry if it seems that way.. I can assure you that I'm putting a lot of hours in to it. I'm running in to syntax problems all the time which I just haven't been able to figure out on my own. I have found a lot of code on google though which has helped me.

    It is probably way to hard for me to do something like this, so the progress is super slow.. but at least I'm learning something doing it. and it's a script that I will be able to use a lot for my projects in the future.

  3. #28
    SitePoint Member
    Join Date
    Apr 2005
    Posts
    8
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    We're creating crawlers and bots from years, best way to start coding them in PHP is not cURL lib, or fopen() or file_get_contents() but the PEAR's HTTP Request class. http://pear.php.net/package/HTTP_Request/

  4. #29
    SitePoint Wizard silver trophybronze trophy Cups's Avatar
    Join Date
    Oct 2006
    Location
    France, deep rural.
    Posts
    6,869
    Mentioned
    17 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by theblackjacker View Post
    It is probably way to hard for me to do something like this, so the progress is super slow.. but at least I'm learning something doing it. and it's a script that I will be able to use a lot for my projects in the future.
    Thats pretty typical when starting out. You have to try and adopt debugging techniques early on which will help you tick off what is working and help you zoom in on where the actual problem is.

    This is especially true when using one langauge (PHP) to construct another (SQL or JS) into HTML markup - each of which have their own syntaxes and pitfalls.

    Debugging PHP - you dont need to go the whole hog of xdebug etc, just the judicious use of echo and var_dump() of your variables so you can peer into what is going on.

    - Always look for evidence that a step you have taken in code actually worked.
    - Turn error reporting on when working in a development environment

  5. #30
    SitePoint Addict
    Join Date
    May 2006
    Location
    Amsterdam
    Posts
    206
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    @BlakeAnthony,

    Quote Originally Posted by BlakeAnthony View Post
    What is a PHP Crawler? Can anyone explain to me what it is?
    A PHP Crawler uses PHP to do the following:
    http://en.wikipedia.org/wiki/Web_crawler

  6. #31
    SitePoint Addict
    Join Date
    May 2006
    Location
    Amsterdam
    Posts
    206
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    @theblackjacker,

    Quote Originally Posted by theblackjacker View Post
    Sorry if it seems that way.. I can assure you that I'm putting a lot of hours in to it. I'm running in to syntax problems all the time which I just haven't been able to figure out on my own. I have found a lot of code on google though which has helped me.

    It is probably way to hard for me to do something like this, so the progress is super slow.. but at least I'm learning something doing it. and it's a script that I will be able to use a lot for my projects in the future.
    You may want to do some reading on multi-dimensional arrays and inserting values in MySQL.

  7. #32
    SitePoint Addict NetNerd85's Avatar
    Join Date
    Aug 2005
    Location
    Australia
    Posts
    298
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    a new day, a new beginning
    never follow the crowd, the crowd is poor!

  8. #33
    SitePoint Member
    Join Date
    Oct 2009
    Posts
    13
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Grate Post!
    Just my opinion, but I think it is better to make in other programming language, because all Crawlers in PHP are weary slow and they may stop in if small count of pages crawled.
    what do u see outside the Window?
    Car hire Spain / Airport Car hire


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •