SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Member bigu_c's Avatar
    Join Date
    Nov 2007
    Posts
    20
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Allow Search Engines Bots?

    Hello,

    I'm searching many source and saw this code:

    RewriteCond %{HTTP_USER_AGENT} !(googlebot-image|msnbot|psbot|yahoo-mmcrawler) [NC]

    Do anyone can confirm for me that this code is correct that allow Google Images... can crawl my images?

    Thanks.
    9HWD.com - The HD Wallpapers Network.
    FB covers

  2. #2
    SitePoint Mentor silver trophy
    Rubble's Avatar
    Join Date
    Dec 2005
    Location
    Cambridge, England
    Posts
    2,187
    Mentioned
    71 Post(s)
    Tagged
    3 Thread(s)
    I do not see why you would need it at all. I do not have it and Google has index some of my images.
    Try images:websitename in google and see what you get.

  3. #3
    Certified Ethical Hacker silver trophybronze trophy dklynn's Avatar
    Join Date
    Feb 2002
    Location
    Auckland
    Posts
    14,604
    Mentioned
    19 Post(s)
    Tagged
    2 Thread(s)
    u_c,

    Without the rest of your mod_rewrite (concerning the bots), there's no way to tell. If you're blocking bots, just omit the bot you want to catalog your website.

    Note: Good bots will honor robots.txt directives but bad bots won't. If you're using mod_rewrite to block bots, many/most of the bad ones include the ability to change their bot identification and become whatever "browser" you will admit to your website. IMHO, it is a losing proposition for the webmaster to try to block bots as it's just too easy to get around blocks with the tools available.

    Regards,

    DK
    David K. Lynn - Data Koncepts is a long-time WebHostingBuzz (US/UK)
    Client and (unpaid) WHB Ambassador
    mod_rewrite Tutorial Article (setup, config, test & write
    mod_rewrite regex w/sample code) and Code Generator


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •