Web robots

How does one go about creating a robots.txt file and placing it in a web site? I hear a lot about robots.txt and how important it is for SEO?

robots.txt is just a plain text file (think Notepad), that you put in the root folder of your web server.

You only need to use it if you want to block search engines and other robots from accessing certain parts of your site. If you’re happy for them to look at all of it, you don’t need a robots.txt file at all.

If you have a sitemap.xml file that you do not call sitemap.xml (e.g. you might call it mysite_sitemap.xml) then the robots.txt allows you to point various crawlers to this location.

Otherwise everything as Stevie D says but if you don’t want to feel left out then Google Webmaster Tools will generate one for you on request!

Robots.txt is a text file which might prevent the search engine robots not to crawl the pages that you would like them not to visit. This file is placed in the root directory that provides the required information for the search engine. The main use of robots.txt is to navigate and index a site.

Robots.txt is a file to be placed in the root folder. Robots.txt will be first visited by search engines when it finds your site, in robots.txt you can determine which bots to crawl your site. You can also hide pages which need not to be crawled by search engines. For eg: print this page, internal search pages need not to be crawled.

Here is good resource for this…The Web Robots Pages

Robots.txt file is just a text file placed in the main directory of a website that tells search engine spiders and other bots which website pages are to be indexed and which should be ignored. The file is a pure advisory and not all spiders take time to pay attention nor read it. But most search engine spiders meant to index your site is going to read it and follow the guidelines written in the file. That’s why it is very important to have this file in your website if you want to have a good ranking on search engines. Otherwise, if they don’t find it then it will be seen as a 404 error

Robot.txt file is a normal text file which should be placed in the website root itself. This text files simply tells the search engine about your webpage crawling. You can specify the pages that don’t want to crawl by the Search Engines.

I think this thread has run it’s course as it’s basically saying the same thing.

If you have a specific question, please feel free to open a new thread, otherwise THREAD CLOSED