SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Mentor silver trophybronze trophy
    Mikl's Avatar
    Join Date
    Dec 2011
    Location
    Edinburgh, Scotland
    Posts
    1,541
    Mentioned
    63 Post(s)
    Tagged
    0 Thread(s)

    Robots.txt to allow only one robot

    I need to create a robots.txt file that does the following:

    - Disallow all robots from accessing certain pages on the site.

    - But, as an exception, allow robot X to access the entire site.

    My robots.txt contains something like this:

    User-agent: robot-x
    Disallow:

    User-agent: *
    Disallow: /somepage.htm
    Disallow: /anotherpage.htm

    Can someone tell me if this will achieve my aim? I'm not sure whether the second record will completely override the first (in which case, the * would also apply to robot-x) or whether the * means, in effect, "all robots except the one mentioned above".

    Hope this makes sense.

    Mike

  2. #2
    Non-Member
    Join Date
    Jan 2012
    Posts
    5
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi in my opinion you use Disallow: /logon page and in your post somepage.htm so google not consider about this page only for relevant pages consider for robot.txt

  3. #3
    Mouse catcher silver trophy Stevie D's Avatar
    Join Date
    Mar 2006
    Location
    Yorkshire, UK
    Posts
    5,881
    Mentioned
    122 Post(s)
    Tagged
    1 Thread(s)
    According to http://www.robotstxt.org/faq/robotstxt.html, the User-agent: * line means "Any other robot not already listed", so yes, that would work fine for allowing robot-x global access and restricting it for all other (well-behaved) robots.

  4. #4
    SitePoint Mentor silver trophybronze trophy
    Mikl's Avatar
    Join Date
    Dec 2011
    Location
    Edinburgh, Scotland
    Posts
    1,541
    Mentioned
    63 Post(s)
    Tagged
    0 Thread(s)
    Thanks, Stevie. That's spot on - exactly what I needed.

    Mike


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •