SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Member
    Join Date
    Dec 2000
    Location
    NC, USA
    Posts
    11
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Is there a definitive list of search engine bot user agents? I'm seeing a few spiders crawling my site but have no idea who they belong to. It would be nice to know as it will prevent me from re-submitting to engines unnecessarily and risk being banned.

    Here's the agents I don't recognize:
    • roach.smo.av.com-1.0
    • Slurp/si
    • lwp-trivial/1.32
    • Scooter/1.0
    • BlueStar/1.0
    • Marvin v0.3
    • libwww-perl/5.33
    • SplatSearch.com
    • Crawler V 0.2.1
    • PCBeaconBot/1.0
    • Gulliver/1.3
    • URL Indexer/1.0
    • MARS SV
    • Whiskers v.3.1

  2. #2
    AdSpeed.com Son Nguyen's Avatar
    Join Date
    Aug 2000
    Location
    Silicon Valley
    Posts
    2,241
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Nice list! Have you done some search query on these? Probably they have it!

    I know for sure Scooter/1.0 is from selfpromotion.com
    - Son Nguyen
    AdSpeed.com - Ad Serving and Ad Management Made Easy

  3. #3

    Join Date
    Oct 1999
    Location
    UK
    Posts
    120
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Try http://www.spiderhunter.com/spiderlist/ for the main ones to look out for from the search engines. You could try a NIC lookup on the IP address to identify where some of the others come from.

    There are all sorts of spiders roaming the web from e-mail harvesters to bespoke bots used by companies for their own purposes.
    Simon Conroy,
    PPC Bid Management and ROI Tracking Solutions
    www.BidBuddy.co.uk www.ROIstats.co.uk


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •