SitePoint.com

thethinktank · Dec 16, 2007, 22:15

Hi Sitepoint fam!
I've noticed in my google webmaster account that I've got a significant number of 404s from a recent spider. Having moved hosts and made a few other changes last month, I ended up moving pages to a new folder. Google is still trying to spider those old URLs with the old folder...

I've read that I can make a change to the robots.txt file that will tell G and others to always look at the new URLs for that directory.

Question is: I'm apprehensive to play with the robots.txt file. Is there any risk to my current and future SEO standings to modify this file?

Thanks so much!

TheThinkTank

skaykay · Dec 16, 2007, 23:18

You are perhaps mixing up with .htaccess file.

In robots.txt, you cannot tell the Googlebot to look at a new url.
You can only tell it what not to look into.

If you want to tell all visitors (including Bots) to go to a new
URL when they access the old url, you have to use 301
redirect using .htaccess file.

thethinktank · Dec 16, 2007, 23:24

Thanks Skaykay-- noted!! I suppose I explained my question poorly. Say I tell Robots.txt to NOT spider the URLs I don't want G to see. Should I expect any level of risk here?

Shyflower · Dec 17, 2007, 06:10

There is no risk if you use robots txt to "no follow" old files. However, you are better off to do a 301 redirect in htaccess to point Googlebot to the path of the files (if they are the same) to the new folder. This will also help your visitors to find the new location of pages that may be cached in search engines but now located in a different area of your site.

stymiee · Dec 17, 2007, 08:29

Quote:

Originally Posted by thethinktank

Question is: I'm apprehensive to play with the robots.txt file. Is there any risk to my current and future SEO standings to modify this file?

As with anytime you are writing code make sure you don't make any syntax errors. In the case of robots.txt you don't want to accidentally block content you want indexed. You can test out your robots.txt file by using the robots.txt checker included in Google Webmaster Tools.

JJMcClure · Dec 17, 2007, 09:15

Quote:

Originally Posted by skaykay

In robots.txt, you cannot tell the Googlebot to look at a new url.

.

Yes you can but it would be a bit pointless since that's it's default action anyway unless instructed otherwise.

stymiee · Dec 17, 2007, 11:50

If you really want to tell them about a new URL you can always use an XML sitemap.

JJMcClure · Dec 17, 2007, 16:27

Quote:

Originally Posted by stymiee

If you really want to tell them about a new URL you can always use an XML sitemap.

That too

I use this site to make maps now - http://www.xml-sitemaps.com/ - and to think I used to do it by hand running python scripts through the unix shell interface .... makes my brain hurt just typing those words.

stymiee · Dec 17, 2007, 18:09

Since my sites are DB driven I write a PHP scripts that generates my XML sitemap for me. That way I never have to update it.

SeriousKevin · Dec 19, 2007, 11:24

do people here feel that php is preferable from an seo point of view, given the ease of .htaccess file redirection when contrasted with the windows server 'equivalents'?

stymiee · Dec 19, 2007, 12:34

Quote:

Originally Posted by SeriousKevin

do people here feel that php is preferable from an seo point of view, given the ease of .htaccess file redirection when contrasted with the windows server 'equivalents'?

That is unrelated to the programming language used and is more of a web server issue. So the real question is do people prefer Apache or IIS?

SeriousKevin · Dec 20, 2007, 16:31

Quote:

Originally Posted by stymiee

That is unrelated to the programming language used and is more of a web server issue. So the real question is do people prefer Apache or IIS?

yeah that's what i meant, most times its a choice between asp/iis and php/apache - right?

stymiee · Dec 20, 2007, 19:12

Quote:

Originally Posted by SeriousKevin

yeah that's what i meant, most times its a choice between asp/iis and php/apache - right?

Those are common setups but many people run php on IIS. There's also asp.net, perl, python, ruby, etc.. being used more commonly nowadays.

Dec 16, 2007, 22:15	#1
thethinktank SitePoint Member Join Date: Oct 2007 Posts: 6	Should I Modify Robots.txt? What is the risk? Hi Sitepoint fam! I've noticed in my google webmaster account that I've got a significant number of 404s from a recent spider. Having moved hosts and made a few other changes last month, I ended up moving pages to a new folder. Google is still trying to spider those old URLs with the old folder... I've read that I can make a change to the robots.txt file that will tell G and others to always look at the new URLs for that directory. Question is: I'm apprehensive to play with the robots.txt file. Is there any risk to my current and future SEO standings to modify this file? Thanks so much! TheThinkTank

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Dec 16, 2007, 23:18	#2
skaykay SitePoint Enthusiast Join Date: Aug 2007 Posts: 80	You are perhaps mixing up with .htaccess file. In robots.txt, you cannot tell the Googlebot to look at a new url. You can only tell it what not to look into. If you want to tell all visitors (including Bots) to go to a new URL when they access the old url, you have to use 301 redirect using .htaccess file.

Dec 16, 2007, 23:24	#3
thethinktank SitePoint Member Join Date: Oct 2007 Posts: 6	Thanks Skaykay-- noted!! I suppose I explained my question poorly. Say I tell Robots.txt to NOT spider the URLs I don't want G to see. Should I expect any level of risk here?

Dec 17, 2007, 06:10	#4
Shyflower Firmly Planted in Content Join Date: Oct 2003 Location: Winona, MN USA Posts: 7,203	There is no risk if you use robots txt to "no follow" old files. However, you are better off to do a 301 redirect in htaccess to point Googlebot to the path of the files (if they are the same) to the new folder. This will also help your visitors to find the new location of pages that may be cached in search engines but now located in a different area of your site.

Dec 17, 2007, 11:50	#7
stymiee He's No Good To Me Dead Join Date: Feb 2003 Location: Slave I Posts: 24,090	If you really want to tell them about a new URL you can always use an XML sitemap.

Dec 17, 2007, 18:09	#9
stymiee He's No Good To Me Dead Join Date: Feb 2003 Location: Slave I Posts: 24,090	Since my sites are DB driven I write a PHP scripts that generates my XML sitemap for me. That way I never have to update it.

Dec 19, 2007, 11:24	#10
SeriousKevin SitePoint Enthusiast Join Date: Oct 2006 Posts: 72	do people here feel that php is preferable from an seo point of view, given the ease of .htaccess file redirection when contrasted with the windows server 'equivalents'?