SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Evangelist
    Join Date
    Oct 2000
    Posts
    430
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    $HTTP_REFERER and Spiders

    Hi,

    I was wondering if anyone could tell me if a variable is sent in $HTTP_REFERER when a spider/search engine trawls through your site. I believe it is a variable which is sent by a browser, so I'm not sure if SE's etc would send anything.

    Thanks in advance.

  2. #2
    Grumpy Mole Man Skunk's Avatar
    Join Date
    Jan 2001
    Location
    Lawrence, Kansas
    Posts
    2,067
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    The only search engine in my HTTP referer logs for www.tfc-central.co.uk is google, and it's user agent string is:

    Googlebot/2.1 (+http://www.googlebot.com/bot.html)

    I've no idea what the user agents are for all the other search engine bots though.

  3. #3
    SitePoint Evangelist
    Join Date
    Oct 2000
    Posts
    430
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi Skunk,

    Thanks for the information - although it is slightly off track from what I'm trying to get at (maybe I did explain it well). So: -

    If a user goes from one page in your site to another page in your site the string held in $HTTP_REFERER is that of the reffering page. Now if a search engine/spider goes from one page in your site to the next will a similar value for $HTTP_REFERER be held, or will it hold a value like http://www.googlebot.com/bot.html ?

  4. #4
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I don't think bots always send that variable. Also with Google your site doesn't really get spidered. It goes to one page, makes a note of the links, but doesn't actually follow those links until a later time.

    I did notice one rather small spider for some graphics search engine used "-" as the referrer... annoying.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  5. #5
    SitePoint Evangelist
    Join Date
    Oct 2000
    Posts
    430
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I don't think bots always send that variable. Also with Google your site doesn't really get spidered. It goes to one page, makes a note of the links, but doesn't actually follow those links until a later time.
    Do all/most search engines work like this??

    I'm wondering if some search engine will give a $HTTP_REFERER variable which would look something like this http://www.mysite.com/page2.php

    Anyone have any further thoughts ??


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •