|
|||||||
New to SitePoint Forums? Register here for free!
|
![]() |
|
|
Thread Tools | Display Modes |
|
|
#1 |
|
SitePoint Member
Join Date: Oct 2007
Posts: 6
|
Should I Modify Robots.txt? What is the risk?
Hi Sitepoint fam!
I've noticed in my google webmaster account that I've got a significant number of 404s from a recent spider. Having moved hosts and made a few other changes last month, I ended up moving pages to a new folder. Google is still trying to spider those old URLs with the old folder... I've read that I can make a change to the robots.txt file that will tell G and others to always look at the new URLs for that directory. Question is: I'm apprehensive to play with the robots.txt file. Is there any risk to my current and future SEO standings to modify this file? Thanks so much! TheThinkTank |
|
|
|
|
|
#2 |
|
SitePoint Enthusiast
![]() Join Date: Aug 2007
Posts: 80
|
You are perhaps mixing up with .htaccess file.
In robots.txt, you cannot tell the Googlebot to look at a new url. You can only tell it what not to look into. If you want to tell all visitors (including Bots) to go to a new URL when they access the old url, you have to use 301 redirect using .htaccess file. |
|
|
|
|
|
#3 |
|
SitePoint Member
Join Date: Oct 2007
Posts: 6
|
Thanks Skaykay-- noted!! I suppose I explained my question poorly. Say I tell Robots.txt to NOT spider the URLs I don't want G to see. Should I expect any level of risk here?
|
|
|
|
|
|
#4 |
|
Firmly Planted in Content
![]() ![]() ![]() ![]() ![]() ![]() ![]() Join Date: Oct 2003
Location: Winona, MN USA
Posts: 7,203
|
There is no risk if you use robots txt to "no follow" old files. However, you are better off to do a 301 redirect in htaccess to point Googlebot to the path of the files (if they are the same) to the new folder. This will also help your visitors to find the new location of pages that may be cached in search engines but now located in a different area of your site.
|
|
|
|
|
|
#5 |
|
He's No Good To Me Dead
![]() ![]() ![]() ![]() ![]() ![]() ![]() Join Date: Feb 2003
Location: Slave I
Posts: 24,090
|
As with anytime you are writing code make sure you don't make any syntax errors. In the case of robots.txt you don't want to accidentally block content you want indexed. You can test out your robots.txt file by using the robots.txt checker included in Google Webmaster Tools.
|
|
|
|
|
|
#6 |
|
Site Point Gnu
![]() ![]() ![]() ![]() ![]() ![]() Join Date: Dec 2007
Location: UK Nr Manchester
Posts: 2,335
|
|
|
|
|
|
|
#7 |
|
He's No Good To Me Dead
![]() ![]() ![]() ![]() ![]() ![]() ![]() Join Date: Feb 2003
Location: Slave I
Posts: 24,090
|
If you really want to tell them about a new URL you can always use an XML sitemap.
|
|
|
|
|
|
#8 | |
|
Site Point Gnu
![]() ![]() ![]() ![]() ![]() ![]() Join Date: Dec 2007
Location: UK Nr Manchester
Posts: 2,335
|
Quote:
![]() I use this site to make maps now - http://www.xml-sitemaps.com/ - and to think I used to do it by hand running python scripts through the unix shell interface .... makes my brain hurt just typing those words. |
|
|
|
|
|
|
#9 |
|
He's No Good To Me Dead
![]() ![]() ![]() ![]() ![]() ![]() ![]() Join Date: Feb 2003
Location: Slave I
Posts: 24,090
|
Since my sites are DB driven I write a PHP scripts that generates my XML sitemap for me. That way I never have to update it.
![]() |
|
|
|
|
|
#10 |
|
SitePoint Enthusiast
![]() Join Date: Oct 2006
Posts: 72
|
do people here feel that php is preferable from an seo point of view, given the ease of .htaccess file redirection when contrasted with the windows server 'equivalents'?
|
|
|
|
|
|
#11 |
|
He's No Good To Me Dead
![]() ![]() ![]() ![]() ![]() ![]() ![]() Join Date: Feb 2003
Location: Slave I
Posts: 24,090
|
That is unrelated to the programming language used and is more of a web server issue. So the real question is do people prefer Apache or IIS?
|
|
|
|
|
|
#12 |
|
SitePoint Enthusiast
![]() Join Date: Oct 2006
Posts: 72
|
|
|
|
|
|
|
#13 |
|
He's No Good To Me Dead
![]() ![]() ![]() ![]() ![]() ![]() ![]() Join Date: Feb 2003
Location: Slave I
Posts: 24,090
|
|
|
|
|
![]() |
| Bookmarks |
«
Previous Thread
|
Next Thread
»
| Thread Tools | |
| Display Modes | |
|
|
|
All times are GMT -7. The time now is 17:39.













Linear Mode
