Web Security and Robots


I’ve blocked my site from users (at least I think I have…) as I’m still in development stage . I’m using CloudFlare and according to my reports, I’ve had tons of robots crawling my site. Can they do this if you need a password just to see the homepage? If it helps, the site is www.myalgebrabook.com.

Thank you,


The robots won’t get past the .htaccess login

Thanks for the quick reply!

Then, does this mean that my CloudFlare statistics are based on robots that landed on the page but didn’t get through?

Must be.

hello EastCoast,

You are right…htaccess can not get pass through robots.txt file…