SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    HAHA!
    Join Date
    Mar 2006
    Posts
    656
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Question Prevent indexing of feed pages on Wordpress

    It's easy enough to exclude the general feed and the comments feed via robots.txt but how do I prevent access to the post page and category page feeds?
    Cheap web hosting directory listing the cheapest web hosting

    Submit articles to an article directory

  2. #2
    SitePoint Addict
    Join Date
    Mar 2006
    Posts
    367
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Hostpitable View Post
    It's easy enough to exclude the general feed and the comments feed via robots.txt but how do I prevent access to the post page and category page feeds?
    Google, Yahoo, and I think MSN support wildcards in robots.txt now so you can do something like this

    User-Agent: *
    Disallow: */feed/

    Make sure you check it in Google Webmaster Tools to make sure you don't block something you want indexed.

  3. #3
    HAHA!
    Join Date
    Mar 2006
    Posts
    656
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks, I saw some people use wildcards in their robots.txt but other people said they are not valid. Do you have a source that confirms one can use wildcards in robots.txt?

    edit: ok I found a post by Aaron wall saying that at least Google supports it. A commenter also says that SLURP supports wildcards, too.
    Cheap web hosting directory listing the cheapest web hosting

    Submit articles to an article directory

  4. #4
    SitePoint Addict
    Join Date
    Mar 2006
    Posts
    367
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    technically it's not valid and it won't prevent other robots from accessing those pages. But G and Y will stay out which sounds like your main concern.

  5. #5
    HAHA!
    Join Date
    Mar 2006
    Posts
    656
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    yes, that'll do for starters.

    thanks.
    Cheap web hosting directory listing the cheapest web hosting

    Submit articles to an article directory

  6. #6
    SitePoint Guru
    Join Date
    Apr 2007
    Posts
    813
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    check out robot org

  7. #7
    SitePoint Addict
    Join Date
    Mar 2006
    Posts
    367
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by SimplyFu View Post
    check out robot org
    Which says he can't use wildcards but Google has said they interpret them and so has Yahoo!

  8. #8
    HAHA!
    Join Date
    Mar 2006
    Posts
    656
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    thanks again for the direct links!
    Cheap web hosting directory listing the cheapest web hosting

    Submit articles to an article directory


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •