SitePoint Sponsor

User Tag List

Results 1 to 2 of 2

Thread: robots txt

Hybrid View

  1. #1
    SitePoint Enthusiast
    Join Date
    Apr 2000
    Location
    Henderson, NV USA
    Posts
    40
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I have a question regarding robots txt files. If I want a Inktomi's bot to stay out of a specific directory, do I state the bots name "slurp" (I think that is one of them?) or the search engine name "Inktomi". If I have to name specifically the bot, does this mean I would then need to list every known version of Inktomi's bots so they stay out of that directory? I only want Inktomi to spider specific directories.

    Thanks

    Laura

  2. #2

    Join Date
    Oct 1999
    Location
    UK
    Posts
    120
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Try http://info.webcrawler.com/mak/proje...exclusion.html which goes into this in some detail
    Simon Conroy,
    PPC Bid Management and ROI Tracking Solutions
    www.BidBuddy.co.uk www.ROIstats.co.uk


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •