Google Using Words From Robots.txt as Keywords

Get Started
1

Sitepoint,

I recently found in my google webmaster account that the top keywords for a small site of mine are disallow, user, and agent, which have nothing to do with my site, but they are words used many times in my long robots.txt file. Any ideas on how to stop google from using words in my robots.txt file as keywords? What would happen if I put a line in my .htaccess file preventing Google from accessing my robots.txt file?

Thanks,

Chris

2

I researched this some more and came up with

<files robots.txt>
order deny,allow
deny from googlebot
</files>

Do I have to have the order deny,allow line?

Thanks,

Chris

3

The robots.txt file is specifically intended to be read by Google and other spiders. Blocking Google from reading it would be at cross purposes.

If these keywords are outranking legitimate keywords in your pages, you need to instead investigate whether the legitimate keywords are being properly presented to search engines. If you have a meta description and keywords tag whose contents are not relevant to the actual page content, Google may assume that you’re gaming the system and give those pages lower rankings.

4

WWW,
I found this on Google’s site,
“You need a robots.txt file only if your site includes content that you don’t want search engines to index. If you want search engines to index everything in your site, you don’t need a robots.txt file (not even an empty one).”

/support/webmasters/bin/answer.py?hl=en&answer=156449

So if I don’t even need an empty robots.txt page and I don’t mind Google crawling everything, and I don’t want Google crawling my robots.txt, what reason is there to not block google from the index.php page. Google has been blocked from my robots.txt page for about 2 days and traffic to that site has increased considerably - now that google doesn’t see all the unrelated words in my long robots.txt file?

Anyway, will the code above work without, “order deny,allow” ?

Thanks,

Chris