SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Enthusiast
    Join Date
    Aug 2006
    Location
    U.K.
    Posts
    64
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Correct robots.txt syntax to disallow Google Adwords bot

    Hi all,

    I have a folder that I don't want Google's adwords bot to see.

    So I have used the standard syntax in robots.txt:

    User-agent: *
    Disallow: /(folder name)

    But I have heard that Google, or the Adwords bot at least, will not obey User-agent: * and will only obey syntax specifically for google.

    So should I use?:

    User-agent: *
    Disallow: /(folder name)

    User-agent: Googlebot
    Disallow: /(folder name)

    User-agent: Adsbot-Google
    Disallow: /(folder name)

  2. #2
    Follow Me On Twitter: @djg gold trophysilver trophybronze trophy Dan Grossman's Avatar
    Join Date
    Aug 2000
    Location
    Philadephia, PA
    Posts
    20,578
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    What is this AdWords bot? I don't think I've ever noticed such a thing. Do you mean the AdSense bot that targets advertising? Either way you'll need to get the user agent correct, and I don't think you want to block Googlebot from your site unless you don't want to be indexed by Google at all.

  3. #3
    SitePoint Zealot
    Join Date
    Jun 2006
    Posts
    133
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

  4. #4
    Follow Me On Twitter: @djg gold trophysilver trophybronze trophy Dan Grossman's Avatar
    Join Date
    Aug 2000
    Location
    Philadephia, PA
    Posts
    20,578
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by brendandonhue View Post
    Thanks for the link, looks like there's still a few topics in the support center I haven't read

    User-agent: AdsBot-Google
    Disallow: /
    Although I wonder why you're willing to pay more for your advertising just to hide pages from an automated bot...

  5. #5
    SitePoint Enthusiast
    Join Date
    Aug 2006
    Location
    U.K.
    Posts
    64
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Although I wonder why you're willing to pay more for your advertising just to hide pages from an automated bot.
    I don't want to hide my entire site from the adwords bot, just a few pages. The pages that I am advertising on adwords will still be visible to the bot and all the other customer facing pages. There are a few tracking pages that I don't want the Adwords bot to see though. Mainly becuase I trust google about as much as I can throw them!


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •