SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Guru moonman's Avatar
    Join Date
    Dec 2000
    Location
    The Sea of Tranquility
    Posts
    696
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Scooter Spidering

    My site is being spidered like there's no tomorrow by AltaVista. It's spidering so much, that it's using the vast majority of our bandwidth.

    AV say that they have no telephone technical support, and their email support takes "2 working days". I can't wait 2 days for these people to stop spidering, they are stopping potential visitors.

    Does anyone know of another way to contact AV. I've already told them once this month to stop, and they laid off for 2 weeks.

    Help!!

  2. #2
    We are vigilant icehousedesigns's Avatar
    Join Date
    Dec 2000
    Location
    Io
    Posts
    299
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Unfortunately I don't know of a way to contact them in a timely manner, other than via e-mail. However, you do have choices. You can block their spider via robots.txt

    ----begin robots.txt---
    Code:
    User-agent: scooter
    Disallow: /
    ----end robots.txt---

    Or if your running the Apache web server, there are several options you have there, including MOD_THROTTLE. I'm not familiar enough with IIS to give any advice there.

  3. #3
    SitePoint Guru moonman's Avatar
    Join Date
    Dec 2000
    Location
    The Sea of Tranquility
    Posts
    696
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I've disallowed them from my robots.txt, Scooter blatently ignores it


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •