Excluding Directories?

[Saltire]

User-agent: * # directed to all robots
Disallow: /thisfolder/
Disallow: /thisfolder2/
Disallow: /thisfolder3/

This is an example robots.txt file. Will it stop any files under these directories from being indexed?

------------------
Methven Online - www.methven.org.uk

[Plebius]

it will stop robots from indexing them as long as the robot pays attention to robots.txt files.



------------------
Plebius Press - An independent media network

[qslack]

Every search engine that is worthwile (and which you'd want to list you) follows the robots.txt guideline. Also many large sites use them...as evidenced by: http://microsoft.com/robots.txt http://www.sun.com/robots.txt http://www.apache.org/robots.txt
Just don't make Sun's mistake and put some files which you may not want the public to know about in your robots.txt file. I'm not sure what's in those Sun binaries in the second row if their robots.txt but it looks like some important stuff.

For the spec of robots.txt, look here: http://info.webcrawler.com/mak/proje...ts/robots.html
For several interesting thoughts on robots.txt, look here: http://www.eiffel.com/private/meyer/robots.html

------------------
Who the heck is Gen. Failure and why does he want to read my hard drive?
Soon to come: Aspology.com

[This message has been edited by d3v (edited April 29, 2000).]