SitePoint Sponsor

User Tag List

Results 1 to 7 of 7
  1. #1
    SitePoint Evangelist ikeo's Avatar
    Join Date
    Oct 2004
    Location
    Austin Texas
    Posts
    591
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Why is Google not indexing my site properly? (log files included)

    These are my log files ...

    66.249.71.53 - - [16/May/2005:02:05:50 -0500] "GET /robots.txt HTTP/1.0" 404 968 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
    66.249.71.53 - - [16/May/2005:02:05:51 -0500] "GET / HTTP/1.0" 200 35254 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
    The site is www.perfectgiftbaskets.com.
    It seems the google bot is coming to the site almost everyday but its not leaving any trace in my log files apart from those two lines.
    Any ideas why its taking off without indexing anything except my front page?

    The Yahoo bot "Slurp" is doing the same thing ... wierd hunh?

    PS: As a side note the MSN bot is tearing it up on the site ... indexing pages like a champ!

  2. #2
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,424
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    There are two Googlebots: one that finds content and one that indexes it. What you've seen is the first one. The second one should (theoretically) be back sometime soon to index some of the pages the first bot found assuming your PR is high enough.

  3. #3
    SitePoint Addict
    Join Date
    Dec 2004
    Location
    San Francisco
    Posts
    263
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    There are two Googlebots: one that finds content and one that indexes it.
    How can I differentiate the Googlebots, which one is the content finder and which one is the indexer?

  4. #4
    SitePoint Evangelist ikeo's Avatar
    Join Date
    Oct 2004
    Location
    Austin Texas
    Posts
    591
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by stymiee
    There are two Googlebots: one that finds content and one that indexes it. What you've seen is the first one. The second one should (theoretically) be back sometime soon to index some of the pages the first bot found assuming your PR is high enough.
    Damn that was quick! Thanks for the help Stymiee.
    Two follow up questions for you if you will indulge me.

    How do you differentiate the first googlebot from the second?
    If it is a content finding googlebot, then why is it coming to my site everyday?

  5. #5
    SitePoint Addict StephenBauer's Avatar
    Join Date
    Apr 2004
    Location
    USA
    Posts
    263
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Isn't the second Googlebot "Referer" entry longer and the term "Googlebot" appears more toward the center of the "Referer" entry?

    (trying to remember my log files)

  6. #6
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,424
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by ikeo
    How do you differentiate the first googlebot from the second?
    I'm not sure. I never bothered to check.

    Quote Originally Posted by ikeo
    If it is a content finding googlebot, then why is it coming to my site everyday?
    You seem to be on Google's radar but not "impotant" enough to be worth digging deeper for. Try to increase your PR (i.e. get more backlinks) and see what happens.

  7. #7
    SitePoint Addict
    Join Date
    Apr 2004
    Location
    Regina, SK
    Posts
    318
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Google digs to a depth based on your PR (or some other ranking that says how important your site is). As your site grows, you'll find it indexing deeper. Basically, just try and get more backlinks and more visitors and you'll see Google paying more attention to you.
    Reduce or eliminate spam on your sites without CAPTHCAs
    - maybe it is possible: http://formantispam.thekerrs.ca/


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •