SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Thread: Holy Cow!

  1. #1
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,939
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Holy Cow!

    Check this out. I have this little portal site for my teeny tiny rural farming home town (which I thankfully no longer live).

    http://www.westbranch.net

    One of the features is a search engine friendly version of DMOZ, which all in all makes the site huge in number of pages (all dynamic of course).

    It actually gets some traffic when people search for some obscure topic the dmoz page at my site comes up... it gets maybe 5000 page views a day because of that.

    Something has been sucking down my server lately, and I just figured out what. Here are the last two days of traffic on that site:

    Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 74.82% 55038
    Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) 7.88% 5795
    Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) 7.54% 5544
    Openfind data gatherer, Openbot TW/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/tw/openbot.html) 3.77% 2770
    Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) 2.55% 1874
    TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html 1.42% 1044
    That number is the number of hits.

    People find it hard to get Inktomi to spider their site.... 55,000 page views from Inktomi... crazy.

    Considering how I don't make any money off this site I think it's time to consider employing a robots.txt file to block engines other than Google.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  2. #2
    Bananas contain Zinc fonzerelli_79's Avatar
    Join Date
    Oct 2001
    Location
    Scotland
    Posts
    816
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    one of my other sites has dmoz in it. You can get dmoz paegs listed quite easily which is very good

    maybe you should try and make some money with the site
    Last edited by fonzerelli_79; Sep 27, 2002 at 03:39.

  3. #3
    SitePoint Enthusiast
    Join Date
    Apr 2002
    Posts
    77
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Wow

    So simply having the directory on your page added hits to the page. Is there a particular reason why this happens.

    Is the hit rate just as effect if its not on your index page.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •