SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Enthusiast codlib's Avatar
    Join Date
    Jan 2007
    Posts
    59
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Sitemap position in robots.txt

    Hi

    Is there any importance in the position of sitemap keyword in robots.txt?

    Specifically my query is, I have some disallow URL's in my robots.txt. But the same links are appearing in sitemap.xml and sitemap.xml is placed at the end of robots.txt file. Will the links specified in disallow section omitted when crawling? Or do i need to add the disallow links after sitemap declaration?

    Also, If i add an old url to disallow section, will it remove from the google search index?

  2. #2
    It's all Geek to me silver trophybronze trophy
    ralph.m's Avatar
    Join Date
    Mar 2009
    Location
    Melbourne, AU
    Posts
    24,117
    Mentioned
    448 Post(s)
    Tagged
    8 Thread(s)
    Quote Originally Posted by codlib View Post
    Is there any importance in the position of sitemap keyword in robots.txt?
    As I understand it, it doesn't matter where you place the sitemap directive in the .htaccess file. See this page for further reference:
    http://www.sitemaps.org/protocol.php#submit_robots
    Facebook | Google+ | Twitter | Web Design Tips | Free Contact Form

    Forum Usage: Tips on posting code samples, images and more

    Forrest Gump: "IE is like a box of chocolates: you never know what you're gonna get."

  3. #3
    SitePoint Enthusiast codlib's Avatar
    Join Date
    Jan 2007
    Posts
    59
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi Ralph

    Thanks for your reply. But still I am not clear about the following.
    1.) Will the sitemap command override the disallow command in robots.txt?
    2.) If a url is present in both site map and disallow section, which will get precedence?

  4. #4
    SitePoint Member weathor's Avatar
    Join Date
    May 2010
    Posts
    16
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You could test it in google masters! It doesnt change your robots.txt, just for test!


Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •