Meta Tag Robots?

[RonSanSolo]

Is the Meta Tag "Robots" still useful?
Is it recommended?

If it is, should I place it in my Index page and have the Robots follow through the entire site? Or just index the home page and do the "nofollow"?

Also, how do you prevent folders (or files) from being indexed by the robots? I remember something about creating a text file...

Thanks,

Ron

[HarryR]

Hi,
To be honest I'm not really sure if the 'robots' meta tag is ever used (although Google should be obeying it), but as a result of things such as blogspam/wikispam (trying to get a higher PR) people have started using the 'nofollow' attribute on links.

As for disallowing robots to access certain parts of your site: the robots.txt file which goes in your web-root serves as a sort of access control list for spiders.

For a tutorial on robots.txt you can either search google for 'robots.txt' or read the following tutorial: http://www.searchengineworld.com/rob...s_tutorial.htm

But as always, there are some robots that won't obey robots.txt

[RonSanSolo]

Harry,

Thanks very much for the info. I'll check out the link on robots.

- Ron

[Nikolas]

I think is good to have them all, because there are so many bots out there, that you are not able to know which of them using them or not.