SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Zealot
    Join Date
    Nov 2007
    Location
    Canada
    Posts
    180
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Question Robots.txt Code check

    I am using this code on my site.I have added this under root.

    PHP Code:
    User-Agent: *
    Allow: /

    User-Agentmsnbot
    Disallow
    : /ppc/
    Allow: /

    User-AgentSlurp
    Disallow
    : /ppc/
    Allow: /

    User-AgentGooglebot
    Disallow
    : /ppc/
    Allow: / 

    trying to exclude ppc folder from search engines.is this correct?

    2-also i have a subdomain in which i do not want search engines crawling.I have added this 2nd robots.txt under subdomain.maindomain.com

    PHP Code:
    User-Agent: *
    Disallow: / 
    will these 2 robots.txt work?

    thank you

  2. #2
    Life is not a malfunction gold trophysilver trophybronze trophy
    TechnoBear's Avatar
    Join Date
    Jun 2011
    Location
    Argyll, Scotland
    Posts
    6,227
    Mentioned
    265 Post(s)
    Tagged
    5 Thread(s)
    To exclude the folder ppc from all bots, all you need is
    Code:
    User-agent: *
    Disallow: /ppc/
    I read somewhere (and I can't now remember where) that Googlebot likes to be called by name, and therefore one should add
    Code:
    User-Agent: Googlebot
    Disallow: /ppc/
    I always keep to that practice and have run into no problems, but it's probably redundant.

    The robots.txt protocol does not include "Allow", only "Disallow". By default, the whole site will be crawled unless folders/files are specifically excluded.

    This page may help.

  3. #3
    SitePoint Zealot
    Join Date
    Nov 2007
    Location
    Canada
    Posts
    180
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    thank you for your help


Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •