SitePoint Sponsor

User Tag List

Results 1 to 13 of 13
  1. #1
    SitePoint Member
    Join Date
    Oct 2007
    Posts
    6
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Should I Modify Robots.txt? What is the risk?

    Hi Sitepoint fam!
    I've noticed in my google webmaster account that I've got a significant number of 404s from a recent spider. Having moved hosts and made a few other changes last month, I ended up moving pages to a new folder. Google is still trying to spider those old URLs with the old folder...

    I've read that I can make a change to the robots.txt file that will tell G and others to always look at the new URLs for that directory.

    Question is: I'm apprehensive to play with the robots.txt file. Is there any risk to my current and future SEO standings to modify this file?

    Thanks so much!

    TheThinkTank

  2. #2
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    80
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You are perhaps mixing up with .htaccess file.

    In robots.txt, you cannot tell the Googlebot to look at a new url.
    You can only tell it what not to look into.

    If you want to tell all visitors (including Bots) to go to a new
    URL when they access the old url, you have to use 301
    redirect using .htaccess file.

  3. #3
    SitePoint Member
    Join Date
    Oct 2007
    Posts
    6
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks Skaykay-- noted!! I suppose I explained my question poorly. Say I tell Robots.txt to NOT spider the URLs I don't want G to see. Should I expect any level of risk here?

  4. #4
    Word Painter silver trophy Shyflower's Avatar
    Join Date
    Oct 2003
    Location
    Winona, MN USA
    Posts
    10,053
    Mentioned
    142 Post(s)
    Tagged
    2 Thread(s)
    There is no risk if you use robots txt to "no follow" old files. However, you are better off to do a 301 redirect in htaccess to point Googlebot to the path of the files (if they are the same) to the new folder. This will also help your visitors to find the new location of pages that may be cached in search engines but now located in a different area of your site.
    Linda Jenkinson
    "Say what you mean. Mean what you say. But don't say it mean." ~Unknown

  5. #5
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by thethinktank View Post
    Question is: I'm apprehensive to play with the robots.txt file. Is there any risk to my current and future SEO standings to modify this file?
    As with anytime you are writing code make sure you don't make any syntax errors. In the case of robots.txt you don't want to accidentally block content you want indexed. You can test out your robots.txt file by using the robots.txt checker included in Google Webmaster Tools.

  6. #6
    Error 404: Life not found silver trophybronze trophy
    Join Date
    Dec 2007
    Location
    UK Nr Manchester
    Posts
    3,460
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by skaykay View Post
    In robots.txt, you cannot tell the Googlebot to look at a new url.

    .
    Yes you can but it would be a bit pointless since that's it's default action anyway unless instructed otherwise.

  7. #7
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    If you really want to tell them about a new URL you can always use an XML sitemap.

  8. #8
    Error 404: Life not found silver trophybronze trophy
    Join Date
    Dec 2007
    Location
    UK Nr Manchester
    Posts
    3,460
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by stymiee View Post
    If you really want to tell them about a new URL you can always use an XML sitemap.
    That too

    I use this site to make maps now - http://www.xml-sitemaps.com/ - and to think I used to do it by hand running python scripts through the unix shell interface .... makes my brain hurt just typing those words.

  9. #9
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Since my sites are DB driven I write a PHP scripts that generates my XML sitemap for me. That way I never have to update it.

  10. #10
    SitePoint Enthusiast
    Join Date
    Oct 2006
    Posts
    71
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    do people here feel that php is preferable from an seo point of view, given the ease of .htaccess file redirection when contrasted with the windows server 'equivalents'?

  11. #11
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by SeriousKevin View Post
    do people here feel that php is preferable from an seo point of view, given the ease of .htaccess file redirection when contrasted with the windows server 'equivalents'?
    That is unrelated to the programming language used and is more of a web server issue. So the real question is do people prefer Apache or IIS?

  12. #12
    SitePoint Enthusiast
    Join Date
    Oct 2006
    Posts
    71
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by stymiee View Post
    That is unrelated to the programming language used and is more of a web server issue. So the real question is do people prefer Apache or IIS?
    yeah that's what i meant, most times its a choice between asp/iis and php/apache - right?

  13. #13
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by SeriousKevin View Post
    yeah that's what i meant, most times its a choice between asp/iis and php/apache - right?
    Those are common setups but many people run php on IIS. There's also asp.net, perl, python, ruby, etc.. being used more commonly nowadays.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •