Hi everyone, I just moved my wordpress site to a new host and received a error message at webmaster tools during that time that google bot couldn’t access my robots.txt, problaby because we suffered a couple of hours of downtime. After the migration I fetched robots.txt it as google and it says success but the error message “Google couldn’t crawl your site because we weren’t able to access robots.txt” keeps staying there.
This week it the warning changed to a yellow exclamation point. Anyone have any experience in getting this error message to go away? After the date of the errors there are no red dots or blue dots that says failed attempts on my webmaster tool graph.
I should mention that the only thing at my robots.txt currently is
#Begin Attracta SEO Tools Sitemap. Do not remove
sitemap: cdn.attracta.com/sitemap/url.xml.gz
#End Attracta SEO Tools Sitemap. Do not remove
Anyone with any experience with this kind of issues? Is there something wrong with my robots.txt? How long does it usually take for webmaster tools to show that the issue have been solved?
I have attached the graph from my webmaster tool. Any insight would be much appreciated.
The problem is the URL you are using returns a 404 error. I assume the url here “url.xml.gz” should be your website url?
Why don’t you use a simple robots.txt file on your own site?
I can sympathise here. I had that message appear in my GWT for one site for several weeks. (I posted here, but got no replies, so don’t hold your breath. ;)) I’ve no idea why it started; I’d made no changes to the site. I could access the file no bother, but “Fetch as Googlebot” threw an error every time. I was on the point of removing the file altogether for a few days, when suddenly the error message vanished, as mysteriously as it had appeared.
I don’t think that is the problem. That’s simply a link to an external sitemap from within the robots.txt file; the problem is that Google’s saying it can’t access the robots.txt file in the first place.
Maybe on that particular day (April 29) the zipped file was corrupt?
I don’t think that is the problem. That’s simply a link to an external sitemap from within the robots.txt file; the problem is that Google’s saying it can’t access the robots.txt file in the first place.
OK, but if you paste that link into the browser you get a 404 error. Even if Google can find the text file how is that link connected to that website as there is nothing in it to link them?
I recommend the user builds a local robots.text file and forgets about the SEO companies one for now or checks what value they are supposed to use in that link.
Can you link to an external file from a robots.txt file, should the user have downloaded a file and pasted the contents of the file into the robots.txt file rather than pasting a link?
I can not find a site that says you can link to another file from a robots.txt file
You can link to an external sitemap in a robots.txt file, and it (the sitemap) can be in .gz format.
http://www.sitemaps.org/protocol.html#submit_robots
Thanks for the feedback everyone, yes the link goes to an external sitemap, I don’t know wether to remove the robots.txt completely or just wait for the issue to resolve at webmaster tools.
I do not think it will ever work as how does this link relate to your website: sitemap: cdn.attracta.com/sitemap/url.xml.gz there is no mention of a website in the link.
It looks like some generic example and you are supposed to replace the url with something like your_website.com so it will become sitemap: cdn.attracta.com/sitemap/your_website.com.xml.gz
If you can access your robots.txt with “fetch as Google”, I’d leave it and try not to worry about it. If you keep getting new reports of the same issue, and are unable to fetch as Google, then you might want to remove the file until the error clears. But I’m just guessing. 
I never got to the bottom of the same issue on my site, which both arrived and went mysteriously, all of its own accord. :magic:
Rubble: Sorry for the miscommunication there is suppose to be a link there that goes to a sitemap but I just put something generic in there since I don’t know what the forums link policy is.
TechnoBear: Yeah I think i will wait another week since both fetch as google works as well as the tab “blocked URL’s” gives me a success on the robots.txt…just sucks to not know whats going on or if webmaster tools really is that slow to change it’s status.
Again, you have my sympathy. It often feels like that’s the entire story of my relationship with Google. 
How can we protect this robots.txt file uploaded on our root folder. Can we limit or denied the access of robots.txt file for any visitor of the particular website.
You can’t restrict access to the robots.txt file without making it inaccessible to the search bots also. Why would you want to do so, anyway?