SitePoint Sponsor

User Tag List

Results 1 to 6 of 6

Thread: robots.txt

  1. #1
    SitePoint Zealot
    Join Date
    Mar 2001
    Posts
    123
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    robots.txt

    Hi,

    My site has been hammered by an Openfind robot the last few days. I'm new to this, but I want to ban the spider using a robots.txt file. I think this is the correct syntax:

    User-agent: Openfind
    Disallow: /

    I've saved the file in UNIX mode (which I read was the right thing to do, but I'm not really sure what it means).

    Anything else I should know? Do I upload it as ASCII?

  2. #2
    SitePoint Enthusiast
    Join Date
    May 2001
    Location
    A parallel universe
    Posts
    59
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    hmm - hope I don't get in trouble for this: but this is a Great resource for learning how to create and validate a robots.txt file here

  3. #3
    SitePoint Guru moonman's Avatar
    Join Date
    Dec 2000
    Location
    The Sea of Tranquility
    Posts
    696
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    My site was hammered by Openfind a few months ago, too the point where we couldn't get out, and no one could get in. I think they are a Japanese company, we phoned them, faxed them emailed them, in the end I used the robots.txt script you mentioned. It stopped them, I then wrote them an email giving them a sturn telling off.

  4. #4
    SitePoint Guru moonman's Avatar
    Join Date
    Dec 2000
    Location
    The Sea of Tranquility
    Posts
    696
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I would now like to point out that some search spiders ignore robots.txt files. Yesterday AV was spidering so much that we couldn't get internet access out, and nobody could get to our site. I disallowed Scooter from my site, it took about 4 hours 'till they checked the robots file again, and then they blatently ignored it!! I have just emailed AV telling them about this, and I can expect a response within 2 working days. If anyone has a phone number for AV help, I would appreciate it.

  5. #5
    SitePoint Enthusiast
    Join Date
    May 2001
    Location
    A parallel universe
    Posts
    59
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    That is true, some do blatently ignore the robots.txt file - while some can take hours or days to actually honor it. There are also spiders that Run from it! Yep, as soon as they find it they turn tail and beat a hasty retreat.

    As for AV responding - that will be a hit and miss. I have heard horror stories about them not responding to vital issues (such as the one you have described) yet also stories about them responding promptly (as I have experienced.)
    Sorry - no # tho.

  6. #6
    SitePoint Guru moonman's Avatar
    Join Date
    Dec 2000
    Location
    The Sea of Tranquility
    Posts
    696
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I got a number for AV, and they were very helpful. Scooter has since slowed down a bit, allowing other people to use my site.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •