I don’t have robots.txt file for my website. Does it effect my site in terms of SEO
On December 1, 2008, John Mueller, a Google analyst, said that if the Googlebot can’t access the robots.txt file (say the server is unreachable or returns a 5xx error result code) then it won’t crawl the website at all. In other words, the robots.txt file must be there if you want the web site to be crawled and indexed by Google.
At the very least having the file there will prevent the log filling up with 404 not found errors from every time any search engine or spambot tries to reference it.
It is useful for restricting some terms to not index in search engines.
yes it is… because it is helpful to keep out unwanted search engine spiders like email retrievers, image strippers, etc. It defines which paths are off limits for spiders to visit. This is useful if you want to hide some personal information or some secret files.
No it does not. BAD bots do not care what your little text file says. Secret files are not secret when you announce them in robots.txt. Personal information shouldn’t even be accessible in the first place.
I think yes,if you are dealing with google or any search engine than you have to upload robots.txt file on your server.Its very important ,if you have good ranking on search engine many website don’t offer this file.
can you please tell me how robots.txt work for websites and what is benefit of this.
This topic has been answered many times in this forum (and reasonably so in this thread itself), so before it gets too much more fluffy, I’m going to close it. If you have any more specific questions, please search the forums first and if you don’t find your answer, please feel free to ask in a new thread.