$HTTP_REFERER and Spiders

[ChilliBoy]

Hi,

I was wondering if anyone could tell me if a variable is sent in $HTTP_REFERER when a spider/search engine trawls through your site. I believe it is a variable which is sent by a browser, so I'm not sure if SE's etc would send anything.

Thanks in advance.

[Skunk]

The only search engine in my HTTP referer logs for www.tfc-central.co.uk is google, and it's user agent string is:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

I've no idea what the user agents are for all the other search engine bots though.

[ChilliBoy]

Hi Skunk,

Thanks for the information - although it is slightly off track from what I'm trying to get at (maybe I did explain it well). So: -

If a user goes from one page in your site to another page in your site the string held in $HTTP_REFERER is that of the reffering page. Now if a search engine/spider goes from one page in your site to the next will a similar value for $HTTP_REFERER be held, or will it hold a value like http://www.googlebot.com/bot.html ?

[aspen]

I don't think bots always send that variable. Also with Google your site doesn't really get spidered. It goes to one page, makes a note of the links, but doesn't actually follow those links until a later time.

I did notice one rather small spider for some graphics search engine used "-" as the referrer... annoying.

[ChilliBoy]

I don't think bots always send that variable. Also with Google your site doesn't really get spidered. It goes to one page, makes a note of the links, but doesn't actually follow those links until a later time.

Do all/most search engines work like this??

I'm wondering if some search engine will give a $HTTP_REFERER variable which would look something like this http://www.mysite.com/page2.php

Anyone have any further thoughts ??