SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Zealot LibrianFriend's Avatar
    Join Date
    Jul 2008
    Location
    USA
    Posts
    174
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Blogger Robot.txt Help

    I am confused that why google add following code in blogger robot.txt

    User-agent: *
    Disallow: /search
    Allow: /

    If I am not wrong, Disallow: /Search means your blog will be not visible in search results?

    Complete robot.txt is as under

    User-agent: *
    Disallow: /search
    Allow: /

    And secondly, when I add the custom robot.txt , it get saved but don't take effect practically. I check it by visiting

  2. #2
    SitePoint Mentor silver trophybronze trophy
    Mikl's Avatar
    Join Date
    Dec 2011
    Location
    Edinburgh, Scotland
    Posts
    1,606
    Mentioned
    66 Post(s)
    Tagged
    0 Thread(s)
    If I am not wrong, Disallow: /Search means your blog will be not visible in search results?
    No. That's not what it means.

    What it means is that any pages in the Search sub-folder of your blog won't be visible in search results (or, more accurately, won't be accessed by robots).

    In Blogger, the Search sub-folder is where the label versions of your posts are stored. For example, if your blog has a label named "recipes", then any posts with that label will appear in a folder named something like www.MyBloggerBlog.com/search/label/recipes. It's that folder that the robots won't access.

    But that's not a problem, because the same posts will also appear in another folder, the URL of which is based on the date of the post. So all that this particular line of the robots.txt is doing is telling the bots not to index the same post twice.

    In general, I would suggest you leave the Blogger robots.txt file alone, unless you have a good reason for editing it.

    Mike

  3. #3
    SitePoint Zealot LibrianFriend's Avatar
    Join Date
    Jul 2008
    Location
    USA
    Posts
    174
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thank you Mike for this help. Now I understand the reason behind robot.txt coding.

    BTW, many guys are talking on their blogs about how and why to change it. But your approach is more rational.


Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •