SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Hybrid View

  1. #1
    ********* Scotland Saltire's Avatar
    Join Date
    Jan 2000
    Location
    Scotland
    Posts
    660
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    User-agent: * # directed to all robots
    Disallow: /thisfolder/
    Disallow: /thisfolder2/
    Disallow: /thisfolder3/

    This is an example robots.txt file. Will it stop any files under these directories from being indexed?

    ------------------
    Methven Online - www.methven.org.uk

  2. #2
    Destiny Manager Plebius's Avatar
    Join Date
    Nov 1999
    Posts
    682
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    it will stop robots from indexing them as long as the robot pays attention to robots.txt files.



    ------------------
    Plebius Press - An independent media network

  3. #3
    SitePoint Wizard
    Join Date
    Jul 1999
    Location
    Chicago
    Posts
    2,629
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Every search engine that is worthwile (and which you'd want to list you) follows the robots.txt guideline. Also many large sites use them...as evidenced by: http://microsoft.com/robots.txt http://www.sun.com/robots.txt http://www.apache.org/robots.txt
    Just don't make Sun's mistake and put some files which you may not want the public to know about in your robots.txt file. I'm not sure what's in those Sun binaries in the second row if their robots.txt but it looks like some important stuff.

    For the spec of robots.txt, look here: http://info.webcrawler.com/mak/proje...ts/robots.html
    For several interesting thoughts on robots.txt, look here: http://www.eiffel.com/private/meyer/robots.html

    ------------------
    Who the heck is Gen. Failure and why does he want to read my hard drive?
    Soon to come: Aspology.com

    [This message has been edited by d3v (edited April 29, 2000).]


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •