Google blocked from entering my website

**cass27** · Jul 26, 2012 03:54

Hi,

I have just been into web master tools to find a message from Google which reads:

http://www.web-writer-articles.co.uk/: Googlebot can't access your siteJul 17, 2012
Over the last 24 hours, Googlebot encountered 2 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 40.0%.

It goes on to recommend: Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors.
The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website.

The history behind this is I recently had to move my website from one server to another because the hosts server was compromised by a virus. This could have been at the time of the error?

Anyway as recommended in the message I contacted my host and they said everything was fine with the server and there were no issues now. I couldn't really argue - I didnt know what to ask for.

So now I'm a bit stuck and worried Google cant access the site. how do I know when it is accessing the site? also where do i find my error logs?

Is there a problem with the robots text file? It is as follows:

User-agent: *
Disallow: /insert.php

thank you for your time with this

**TechnoBear** · Jul 26, 2012 06:10

Have there been any problems with Google accessing your site since 17th July? I can access your robots.txt file no problem, so I assume Google can, too.

If you can check your access logs for that day, you could see if any other search engine managed to access your robots file that day. If you're using cPanel, you (hopefully) have Awstats which will let you see a breakdown of your 404 "Not Found" errors. Have a look at that and see if there's a high number of errors for your robots.txt file.

If you can't find any other signs of a problem and your hosts assure you everything is fine, then it may just have been something that occurred on that one day. Just keep an eye on it, but don't panic.

**cass27** · Jul 26, 2012 06:21

Originally Posted by TechnoBear

Have there been any problems with Google accessing your site since 17th July? I can access your robots.txt file no problem, so I assume Google can, too.

If you can check your access logs for that day, you could see if any other search engine managed to access your robots file that day. If you're using cPanel, you (hopefully) have Awstats which will let you see a breakdown of your 404 "Not Found" errors. Have a look at that and see if there's a high number of errors for your robots.txt file.

If you can't find any other signs of a problem and your hosts assure you everything is fine, then it may just have been something that occurred on that one day. Just keep an eye on it, but don't panic.

I have unfortunately already paniced....still quaking.

however it does look as if Google can access it now. I did a fresh google fetch on webmaster tools for the robots.txt file today. there were errors up until yesterday as fas I can see from webmaster tools. and it still says there is a server error. I dont have cpanel I have plesk. I guess I'll just have to see what it states tomorrow now it seems to have successfully submitted robots.txt file on webmaster tools.

**TechnoBear** · Jul 26, 2012 06:45

I've never used Plesk, but presumably you have some kind of stats that would show you if you've been getting a high number of errors. Do you have any other kind of access log? It would be useful to see whether other bots have been encountering problems. Have any humans reported difficulty accessing your site?

**cass27** · Jul 27, 2012 05:12

Originally Posted by TechnoBear

I've never used Plesk, but presumably you have some kind of stats that would show you if you've been getting a high number of errors. Do you have any other kind of access log? It would be useful to see whether other bots have been encountering problems. Have any humans reported difficulty accessing your site?

Hi,

I have tracked down where the logs are on plesk. Not being a technician technician it seems a mammoth load of gobbledegook. I am not sure exactly what I am looking for. how is an error signified? I cannot see a mention of Google but perhaps it didn't crawl today?

thanks

**TechnoBear** · Jul 27, 2012 05:35

You should have entries in there that look like these:

Code:

157.55.50.12 - - [02/Mar/2012:00:04:40 +0000] "GET /robots.txt HTTP/1.1" 200 237 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
66.249.72.145 - - [03/Mar/2012:05:55:54 +0000] "GET /robots.txt HTTP/1.1" 200 117 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The 200 code means that the bot was able to fetch the file OK.

Try doing a search for "robots" within your logs, and see which bots you can find trying to access the file. If there's a 200 code, there's no problem. If there's another code, post back and let us know what it is.

**cass27** · Jul 27, 2012 07:04

Originally Posted by TechnoBear

You should have entries in there that look like these:

Code:

157.55.50.12 - - [02/Mar/2012:00:04:40 +0000] "GET /robots.txt HTTP/1.1" 200 237 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
66.249.72.145 - - [03/Mar/2012:05:55:54 +0000] "GET /robots.txt HTTP/1.1" 200 117 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

The 200 code means that the bot was able to fetch the file OK.

Try doing a search for "robots" within your logs, and see which bots you can find trying to access the file. If there's a 200 code, there's no problem. If there's another code, post back and let us know what it is.

It seems that all of the files fetched have a 200 at the end. that includes the robots fetch. however there seems to be ots such as Gecko matauri, mozilla, rogerbot etc but I cant see Google. Is this something to worry about do you think?

**TechnoBear** · Jul 27, 2012 07:23

What time period did you look at? If it's only a day or two, I wouldn't worry at all. You confirmed yesterday that Google can now access the robots.txt file, and your logs show there's no problem with it, so I'd just keep an eye on it for a few days to check that Google doesn't start reporting errors again. I presume you haven't seen a sudden drop in your Google rankings, or any other cause for concern?

If you like, I can move the thread for you to either Internet Marketing (where you might get more expert advice on Google and its peculiarities) or Web Hosting & Domains, if you're still concerned that there might be a server problem.

**cass27** · Jul 27, 2012 07:35

Originally Posted by TechnoBear

What time period did you look at? If it's only a day or two, I wouldn't worry at all. You confirmed yesterday that Google can now access the robots.txt file, and your logs show there's no problem with it, so I'd just keep an eye on it for a few days to check that Google doesn't start reporting errors again. I presume you haven't seen a sudden drop in your Google rankings, or any other cause for concern?

If you like, I can move the thread for you to either Internet Marketing (where you might get more expert advice on Google and its peculiarities) or Web Hosting & Domains, if you're still concerned that there might be a server problem.

No, I think we have solved it. And i have learnt a lot here too. A crisis can be really powerful in growth if you allow it to be.

I will monitor it over the next few days

Thanks again for all your time with this techno Bear. much appreciated.