SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Wizard Anat's Avatar
    Join Date
    Oct 2000
    Posts
    1,281
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    How to limit the bots from crawling my site?

    I don't mean stop them from crawling, just limiting the number of pages crawled each time.

    I have some AWS sites now and bots are hitting them hard. When they do it brings up server loads too much. I wonder if there's a way to limit the number of pages crawled per session or something similar? I do want the sites to be indexed, just no that much all at once.

    Any ideas?
    My Web Publishing Blog: B6S.net - I dofollow but don't spam!
    Follow me on Twitter
    My favorite content writer:
    Steve Snedeker

  2. #2
    Application Developer shabbirbhimani's Avatar
    Join Date
    Apr 2004
    Location
    India
    Posts
    2,272
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    use robots.txt to restrict some areas at some point of time or use a time span in the meta tags to restrict them for some time.

    Best I would suggest redirect them to my site.

  3. #3
    Non-Member betterpc's Avatar
    Join Date
    Dec 2004
    Location
    EU
    Posts
    123
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

  4. #4
    Massimiliano Bruno Giordano sid egg's Avatar
    Join Date
    Aug 2004
    Location
    Canada
    Posts
    1,280
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Robots.txt implements a "crawl delay" feature, google it
    Last edited by moospot; May 28, 2005 at 05:57.
    GamesLib.com - the slickest, most complete and
    easily navigatible flash games site on the web.

  5. #5
    SitePoint Addict JordanTW's Avatar
    Join Date
    Apr 2004
    Location
    Indiana
    Posts
    215
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Put a couple links on your page leading to my sites


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •