I have a blog that has Google Adsense ads.
Yesterday I checked my Google Adsense acct and it said it hasn’t been crawling my site for ads for about a week due to the presence of a robots.txt file in my code. The problem is, I don’t have a robots.txt file in any of my pages.
It then said whether the code is or is not present, add the following two-line code
to the robots.txt file (or create one if I don’t have it) and then upload it to the root directory at my server. I did just that: I created a blank txt file and added the two-line code and then sent it to my server. But this still perplexes me.
My question is: doesn’t the presence of the word, “Disallow:” mean the Google crawlers won’t be able to crawl my site? Of, is it because the word, “Disallow:” is left empty that they will?