I have not written the robots.txt file for my web site, I don’t know whether this file will improve page rank status or not, is it necessary for a web site? Thanks!
wow i didn’t know this one too, i thought you just have create robots.txt if you have something to exclude on your site… now I’m planning to create one for my site, how can i test or check if it really works?
To create a robots.txt file just to get rif of all the 404 errors simply create an empty file called robots.txt and upload that to your site.
You should be able to tell if it works by examining site stats for periods before and after uploading as the number of 404 errors on the site should be dramatically reduced.
Yeah - I always add a robots.txt to my new sites for that exact reason. Just create an empty one and you’ll be fine - and if you need it one day it will already be there for you.
The robots.txt protocol is a useful tool to prevent certain bots from spidering certain pages/sections of sites while allowing others through. It’s also a lot easier to modify a robots.txt file than it is to update dozens, hundreds or even thousands of Web pages to make a single change. Also, unlike the META tag, the robots.txt file can block SPECIFIC search engine spiders from crawling and indexing your site.
And has already been mentioned, not having a robots.txt file can clutter your server logs with needless 404 error returns.
That is new info to me …Thanks for sharing it.
There is a very good reason for including an empty robots.txt file if you don’t have anything to put in it - it gets rid of the thousands of 404 errors that will get added to your log every day when the search engines can’t find the file.
Yes…normally we will use robots.txt text when ever we want to exclude some pages from search engines.
I only use robot.txt if I don’t want google bot to see a certain page on my site that’s all. There’s no porn on my site, it is just a sub-directory where I test stuff on.
I don’t think robot.txt helps your ranking but I can be wrong.
If you do not want to indexed your secret pages in search engines then you have to use robots.txt.
i dont think it is necessary. i dont have robots.txt for majority of my websites
It’s very hard to configure it wrongly, if you manage to screw up the creation of a robots.txt file and can’t get it right, you have no business building websites. 
if you don’t want to restrict robots to access/index to your site it is certainly NOT needed.
It does improve rankings for those search engines where the search engine only looks at the site at all if the robots.txt file exists. At least then your page will be in the results even if at position 1000000000 which is a huge improvement over not being listed at all.
Robots.txt file consists of information that inform the search engines which directories or pages of your website should be indexed and which should be ignored. I think it is necessary in every website.
Such as if you want search engines to index your site. There is at least one search engine that treats a missing robots.txt file as an instruction to ignore the site completely.
Of course then there’s all the gazillions of 404 errors in your log if you don’t have it - making genuine errors harder to spot.
Also how can you possibly stuff up creating a robots.txt with nothing in it (which is all you need in order to get rid of the 404 errors and tell all search engines to index your site) I don’t know. If you can’t create an empty file then how much more trouble are you going to have creating one that contains HTML?
That means either you have something filtering out those errors or you don’t have any requests being made for the file (which would indicate that there are no search engines visiting the site). Before I added the file to my sites the logs were overflowing with 404 errors for that file.
I do remember that when Microsoft introduced one of their search engines a few years ago (the one before Bing I think) - the search engine would treat the lack of a robots.txt file as a request to not include the site and that I had to add an empty robots.txt file just to get the search engine to even acknowledge that my site existed.
Adding an empty robots.txt file is certainly the easiest SEO action you can take.
Hmmm, I just checked two of my sites that don’t have a robots.txt file, and there are no errors in either error log.
robots.txt should be used only when absolutely necessary. There are chances that it can backfire if configured wrongly.