SitePoint Sponsor

User Tag List

Results 1 to 11 of 11
  1. #1
    SitePoint Addict jamesglewisf's Avatar
    Join Date
    Sep 2000
    Location
    Big D
    Posts
    321
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    [Begin Rant]

    The GoogleBot spider is driving me crazy. It is spidering my forum site day after day after day. It has now spidered 341 pages. I only have 231 threads.

    I'll get all excited thinking I have 11 visitors to my board, and 10 of them will be Google. I've added the robot tag to my showthread pages, but I really don't want to add it to the others. I may have to.

    I'm just wondering if it will ever finish.

    [/End Rant]

    Aaaahhhhhh. I feel better.
    Jim Lewis
    To BE or Not to BE, or: How I Learned to Stop Worrying and Love the Barium Enema
    FrappyDoo Forums

  2. #2
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Heh I was like this for atomz.com

    When they'd index my page I'd get a ton of hits, then I'd look and see that 300 or so of them were from Atomz.com

  3. #3
    AdSpeed.com Son Nguyen's Avatar
    Join Date
    Aug 2000
    Location
    Silicon Valley
    Posts
    2,241
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Can it index dynamic pages (threads/posts)?
    - Son Nguyen
    AdSpeed.com - Ad Serving and Ad Management Made Easy

  4. #4
    SitePoint Addict jamesglewisf's Avatar
    Join Date
    Sep 2000
    Location
    Big D
    Posts
    321
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Originally posted by 123finder.com
    Can it index dynamic pages (threads/posts)?
    Yes. Just check out this search at Google.
    Jim Lewis
    To BE or Not to BE, or: How I Learned to Stop Worrying and Love the Barium Enema
    FrappyDoo Forums

  5. #5
    SitePoint Addict jamesglewisf's Avatar
    Join Date
    Sep 2000
    Location
    Big D
    Posts
    321
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I only use Atomz on my main site, not the boards. But your are right, it does show up in my referral logs.
    Jim Lewis
    To BE or Not to BE, or: How I Learned to Stop Worrying and Love the Barium Enema
    FrappyDoo Forums

  6. #6
    Idea Developer
    Join Date
    Sep 2000
    Location
    Bethlehem, PA
    Posts
    521
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Do all site index pages with ?id=anything
    Professional PHP programing / Hosting
    aim: downtoi3iz icq: 74637813

  7. #7
    SitePoint Addict jamesglewisf's Avatar
    Join Date
    Sep 2000
    Location
    Big D
    Posts
    321
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Originally posted by necrominator
    Do all site index pages with ?id=anything
    Huh? Do=Does?
    Jim Lewis
    To BE or Not to BE, or: How I Learned to Stop Worrying and Love the Barium Enema
    FrappyDoo Forums

  8. #8
    AdSpeed.com Son Nguyen's Avatar
    Join Date
    Aug 2000
    Location
    Silicon Valley
    Posts
    2,241
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Originally posted by jamesglewisf
    Originally posted by 123finder.com
    Can it index dynamic pages (threads/posts)?
    Yes. Just check out this search at Google.
    Yah! It's pretty cool to know that php-generated pages are indexed! However, what about complicated URLs? (like: script.php?var=val&n=d&x=s....)

    And other types of dynamic pages? like asp,pl,cfm?
    - Son Nguyen
    AdSpeed.com - Ad Serving and Ad Management Made Easy

  9. #9
    SitePoint Addict jamesglewisf's Avatar
    Join Date
    Sep 2000
    Location
    Big D
    Posts
    321
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    This URL got spidered:

    http://www.marriedadults.net/forum/s...post&forumid=4

    Grrrrrrrrr. I added a robots.txt file, and I'm still getting spidered. 9 guests right now are all google.
    Code:
    User-Agent: *
    Disallow: /forum/
    /cgi-bin/
    Jim Lewis
    To BE or Not to BE, or: How I Learned to Stop Worrying and Love the Barium Enema
    FrappyDoo Forums

  10. #10
    AdSpeed.com Son Nguyen's Avatar
    Join Date
    Aug 2000
    Location
    Silicon Valley
    Posts
    2,241
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Maybe Google spider forgets how to read robots.txt
    Btw, just not sure if you need the slash at the end for those dirs in robots.txt?
    - Son Nguyen
    AdSpeed.com - Ad Serving and Ad Management Made Easy

  11. #11
    SitePoint Addict jamesglewisf's Avatar
    Join Date
    Sep 2000
    Location
    Big D
    Posts
    321
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    The example that Google gave had the slashes at the end. I really don't know.
    Jim Lewis
    To BE or Not to BE, or: How I Learned to Stop Worrying and Love the Barium Enema
    FrappyDoo Forums


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •