What is the robots.txt file?

[bluemarine]

What is the robots.txt file?

[RetroNetro]

http://www.robotstxt.org/

[water2filter]

basically speaking,robot.txt is the file to tell search engine,on your site which content is allowed to index,which content to keep private.

[mcaronan]

robot.txt is a text file that instruct robots what they need or not to crawl certain pages in your website.

Why do people want spiders not to crawl certain pages?
There are lot of things why, one good example is that your back up your website files in your host. If spiders detected this files, search engines might think that these files are duplicate files.

[Legendary11]

to be really basic it tells crawlers where to go and not to go, what to index and what not to index, which crawlers are allowed and not allowed to search the site

[felgall]

Spambots will of course completely ignore the instructions in that file or use it to work out where you don't want bots to go so that they can make sure that they go there.

[lukemeister]

Spambots will of course completely ignore the instructions in that file or use it to work out where you don't want bots to go so that they can make sure that they go there.

Spambots are pretty much Internet Gremlins, evil