SitePoint Sponsor

User Tag List

Results 1 to 9 of 9
  1. #1
    SitePoint Member
    Join Date
    Sep 2002
    Posts
    5
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Help with bots please

    Hi all:

    The past few weeks, inktomi and googlebot have been to my site several times, but they don't seem to be indexing anything. Here's what's happening:
    GET /robots.txt HTTP/1.0
    GET / HTTP/1.0
    Then they leave.
    Is this normal? And what are they getting with just "/"?

    I recently changed my .htaccess file because a few people were downloading my whole site. I'm wondering if something in this file is blocking them. I have decent rankings and hope I didn't mess up by changing this file. Here's what I put in it:

    RewriteCond %{HTTP_USER_AGENT} ^(Deweb|Fetch|FlashGet|Getleft|.*HTTrack|KWebGet|
    Mirror|NetAnts|Net.Vampire|pavuk|PSurf|PushSite|reget|Shai|Templeton|w3mir|web.b
    y.mail|WebCopier|WebCopy|WebMiner|WebReaper|WebSnake|WebStripper|webvac|webwalk|
    WebZIP|Wget) [NC,OR]

    All help appreciated

    Thanks

  2. #2
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Whats your robots.txt say?
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  3. #3
    SitePoint Member
    Join Date
    Sep 2002
    Posts
    5
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    User-agent: *
    Disallow: /stats/

  4. #4
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    and whats your site?
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  5. #5
    SitePoint Enthusiast vSector's Avatar
    Join Date
    Jun 2001
    Posts
    39
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    This is very normal. I get it all the time and have heard alot of other people getting it too.

  6. #6
    Back in Action Winged Spider's Avatar
    Join Date
    Jun 2001
    Location
    outside my mind
    Posts
    900
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    It looks like their accessing your main page, and nothing else. Do your inner pages have page rank?


  7. #7
    SitePoint Member
    Join Date
    Sep 2002
    Posts
    5
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I figured it out. A while back I changed the links to my inner pages from the full url to just the file names and forget to change it back. Ugh!

  8. #8
    SitePoint Member
    Join Date
    Sep 2002
    Posts
    5
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Changing the file names back to the full url didn't help. The bots were back and still they only indexed my home page.

    So I did a search on why this might happen. I found out that when you use tables and javascript the main content of your site gets pushed to the bottom of the file.

    Therefore your keywords and inner links may be so deep inside your HTML code that the search engines won't notice them, because most of them don't check the entire document.

    I'm screwed for googles next update. I went from rank 1 - 31 for various keywords to non-existent on www2 and www3.

    But maybe this info will help someone else.

  9. #9
    SitePoint Enthusiast vSector's Avatar
    Join Date
    Jun 2001
    Posts
    39
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hey heres an idea, to make your content seem at the top why not make some blank table at the bottom of your page :-)


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •