Web Security and Robots
I've blocked my site from users (at least I think I have....) as I'm still in development stage . I'm using CloudFlare and according to my reports, I've had tons of robots crawling my site. Can they do this if you need a password just to see the homepage? If it helps, the site is www.myalgebrabook.com.
The robots won't get past the .htaccess login
Thanks for the quick reply!
Then, does this mean that my CloudFlare statistics are based on robots that landed on the page but didn't get through?
Originally Posted by EastCoast
You are right.....htaccess can not get pass through robots.txt file...