SitePoint Sponsor

User Tag List

Results 1 to 2 of 2
  1. #1
    SitePoint Enthusiast
    Join Date
    May 2008
    Posts
    65
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    how can I filter googlebot sessions?

    I am using a simple php script to estimate the number of online users by counting the number of apache session files in my tmp directory. Every now and then the # of users spikes up and remains inflated for several days before returning to normal. I checked today during another inflated period, and it looks like the bulk of the sessions are coming from google's crawler (crawl-66-249-71-56.googlebot.com).

    So my questions:

    - Is there a reason there are over 100 sessions coming from the google crawler?
    - Why does this happen periodically over a span of several days and then go away?
    - What can I do to filter this information?

    Thanks.

  2. #2
    Follow Me On Twitter: @djg gold trophysilver trophybronze trophy Dan Grossman's Avatar
    Join Date
    Aug 2000
    Location
    Philadephia, PA
    Posts
    20,578
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by jacksonheights
    Is there a reason there are over 100 sessions coming from the google crawler?
    Because the bot is not sending back the session identifier cookie you sent it, your site is creating a new session for every page it requests. 100 sessions just means Google has requested 100 pages of your site in less than the time it takes for garbage collection to purge the old sessions (usually 24 minutes).

    Quote Originally Posted by jacksonheights
    Why does this happen periodically over a span of several days and then go away?
    Your site is being recrawled, so that Google can keep its search results fresh. It will return periodically forever unless you tell it not to with a robots.txt rule.

    Quote Originally Posted by jacksonheights
    What can I do to filter this information?
    You could not start a session when googlebot is the agent making the request. This would mean changing your website code where the session is initiated.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •