SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Zealot
    Join Date
    May 2005
    Posts
    138
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Meta Tag Robots?

    Is the Meta Tag "Robots" still useful?
    Is it recommended?

    If it is, should I place it in my Index page and have the Robots follow through the entire site? Or just index the home page and do the "nofollow"?

    Also, how do you prevent folders (or files) from being indexed by the robots? I remember something about creating a text file...

    Thanks,

    Ron

  2. #2
    SitePoint Wizard HarryR's Avatar
    Join Date
    Dec 2004
    Location
    London, UK
    Posts
    1,376
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    To be honest I'm not really sure if the 'robots' meta tag is ever used (although Google should be obeying it), but as a result of things such as blogspam/wikispam (trying to get a higher PR) people have started using the 'nofollow' attribute on links.

    As for disallowing robots to access certain parts of your site: the robots.txt file which goes in your web-root serves as a sort of access control list for spiders.

    For a tutorial on robots.txt you can either search google for 'robots.txt' or read the following tutorial: http://www.searchengineworld.com/rob...s_tutorial.htm

    But as always, there are some robots that won't obey robots.txt

  3. #3
    SitePoint Zealot
    Join Date
    May 2005
    Posts
    138
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Harry,

    Thanks very much for the info. I'll check out the link on robots.

    - Ron

  4. #4
    SitePoint Wizard Nikolas's Avatar
    Join Date
    Feb 2005
    Location
    Greece
    Posts
    1,222
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I think is good to have them all, because there are so many bots out there, that you are not able to know which of them using them or not.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •