SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Member
    Join Date
    Jan 2002
    Location
    Prince Frederick Maryland 20678
    Posts
    12
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    What are the names of the robots/spiders?

    I have had two bots/spiders vist my site. I was wondering if there was a list that explained where these bots/spiders
    are from.

    ia_archiver

    Scooter

    Who are these bots above?


    Does anyone have the full list of names
    and dates they spider the net?

    Thanks
    Billy

  2. #2
    Non-Member SiteTutor's Avatar
    Join Date
    May 2002
    Location
    Texas
    Posts
    97
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Billy
    Scooter is on this list.. not sure of the other one you mention. Hope this is of use!


    http://www.robotstxt.org/wc/active/html/index.html

  3. #3
    SitePoint Enthusiast hbird64's Avatar
    Join Date
    Apr 2002
    Location
    Netherlands
    Posts
    64
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Staff member of JoeAnt.com
    Also providing content for Top Moving Companies

  4. #4
    + platinum's Avatar
    Join Date
    Jun 2001
    Location
    Adelaide, Australia
    Posts
    6,441
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Well last month I got these ones:
    Scooter is at near the bottom

    SlySearch
    SlySearch/1
    Lycos_Spider_(modspider)
    Googlebot
    Googlebot/2
    Openfind data gatherer, Openbot
    Openfind data gatherer, Openbot/3
    ia_archiver
    FAST-WebCrawler
    FAST-WebCrawler/3
    (Teradex Mapper; mapper@teradex.com; http:
    (Teradex Mapper; mapper@teradex.com; http://www
    isurf (tszhu@canada.com)
    Galeon
    Galeon/1
    libwww-perl
    libwww-perl/5
    Sqworm
    Sqworm/2
    Scooter-3.2
    Scooter-ARS-1.1

  5. #5
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •