SitePoint Sponsor

User Tag List

Results 1 to 7 of 7
  1. #1
    SitePoint Zealot
    Join Date
    Nov 2004
    Location
    Westerville
    Posts
    127
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Is robots.txt neccessary for a good ranking???

    I was curious, is it really neccessary to use a robots.txt for a good ranking? I understand that a robots.txt is to exclude directories from a crawl? Not real sure here. If so, is there a basic robots.txt that I could modify, maybe a template of some sort? Where might I find a good web page describing robots.txt. From what I see, you only use a robots.txt when you don't want something crawled. Just trying to squeeze some more seo from my sites... Thanks!

  2. #2
    SitePoint Addict SRTech's Avatar
    Join Date
    Mar 2005
    Posts
    224
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Well, I don't think it would hurt having one, even if it is an empty file. Having one will also make sure you don't get excess 404 errors in your logs.

    Here is some basic information on them:
    http://www.robotstxt.org/wc/robots.html

  3. #3
    SitePoint Addict heropage's Avatar
    Join Date
    May 2004
    Location
    canada
    Posts
    238
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    can anyone give me a example of robot.txt

  4. #4
    SitePoint Addict SRTech's Avatar
    Join Date
    Mar 2005
    Posts
    224
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by heropage
    can anyone give me a example of robot.txt
    Take a look at:
    http://www.sitepoint.com/robots.txt

  5. #5
    King of Paralysis by Analysis bronze trophy
    Join Date
    Jul 2004
    Location
    Ottawa, Canada
    Posts
    5,840
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Not having a robots.txt is not to going to impact your search engine rankings in any way.

  6. #6
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,426
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by tke71709
    Not having a robots.txt is not to going to impact your search engine rankings in any way.
    Exactly. All robots.txt does is keep search engine crawlers off of pages you don't want indexed. Although you can use this to block pages that may affect your ranking in some search engines (i.e. pages with lots of external links), the robots.txt file will not directly impact your rankings.

  7. #7
    SitePoint Enthusiast
    Join Date
    Mar 2005
    Posts
    25
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    The only thing it will really help with is not having a large amount of 404 errors in your stats logs, maybe make an empty file, thats what I often find myself doing, gives me peace of mind when I'm checking my errors + I don't see thousands of errors, just maybe the odd one or two which I can then isolate and fix.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •