SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Evangelist
    Join Date
    Mar 2011
    Location
    Bellingham, WA
    Posts
    450
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Web Security and Robots

    Hello!

    I've blocked my site from users (at least I think I have....) as I'm still in development stage . I'm using CloudFlare and according to my reports, I've had tons of robots crawling my site. Can they do this if you need a password just to see the homepage? If it helps, the site is www.myalgebrabook.com.

    Thank you,

    Eric

  2. #2
    Community Advisor silver trophy

    Join Date
    Nov 2006
    Location
    UK
    Posts
    2,551
    Mentioned
    40 Post(s)
    Tagged
    1 Thread(s)
    The robots won't get past the .htaccess login

  3. #3
    SitePoint Evangelist
    Join Date
    Mar 2011
    Location
    Bellingham, WA
    Posts
    450
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Thanks for the quick reply!

    Then, does this mean that my CloudFlare statistics are based on robots that landed on the page but didn't get through?

  4. #4
    Community Advisor silver trophy

    Join Date
    Nov 2006
    Location
    UK
    Posts
    2,551
    Mentioned
    40 Post(s)
    Tagged
    1 Thread(s)
    Must be.

  5. #5
    SitePoint Member
    Join Date
    Sep 2011
    Location
    Delhi
    Posts
    4
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by EastCoast View Post
    The robots won't get past the .htaccess login
    hello EastCoast,

    You are right.....htaccess can not get pass through robots.txt file...


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •