SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Member
    Join Date
    Sep 1999
    Posts
    19
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    First of all I'm assuming it is search engine related, if not I can post it in a more appropriate forum. If it is search engine related how do you write one? Any help would be appreciated.

  2. #2

    Join Date
    Oct 1999
    Location
    UK
    Posts
    120
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    When a search engine spider comes to look at a page it first checks to see if there is a robots.txt file. This file can be configured to exclude spiders from different parts of the site.

    There is lots of information about how the robots.txt works at http://info.webcrawler.com/mak/proje...ts/robots.html or try AltaVistas FAQS at http://doc.altavista.com/adv_search/..._avoiding.html

    The default is to index everything and it doesn't matter if you don't have this file unless you want to exclude spiders.
    Simon Conroy,
    PPC Bid Management and ROI Tracking Solutions
    www.BidBuddy.co.uk www.ROIstats.co.uk

  3. #3
    SitePoint Member
    Join Date
    Sep 1999
    Posts
    19
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thank You! Those two links should answer all my questions.

  4. #4
    SitePoint Member
    Join Date
    Oct 2000
    Posts
    9
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Anyone knows a good website that can check the robots.txt file. I got one but it did not work well.
    http://www.tardis.ed.ac.uk/~sxw/robots/check/
    It showed that something was wrong but it did not.
    Knowing Thailand from Thai perspectives
    http://www.gorgai.com


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •