SitePoint Sponsor

User Tag List

Results 1 to 2 of 2
  1. #1
    SitePoint Member
    Join Date
    Oct 2012
    0 Post(s)
    0 Thread(s)

    Post Problem with robots.txt file coding ?

    My question is like very simple but for me a very big cause to think .
    In robots.txt file if i indicate one of my URL having like , so if do use this particular page not to crawl in Google . Then is it possible that only this URL will be considered for not to crawl as it having main category page also.
    According to my it will consider main url like page.

    So will please somebody clarify me how this problems will be removed .
    Last edited by TechnoBear; Nov 30, 2012 at 06:17. Reason: Example URLs delinkified

  2. #2
    Life is not a malfunction gold trophysilver trophybronze trophy
    TechnoBear's Avatar
    Join Date
    Jun 2011
    Argyll, Scotland
    274 Post(s)
    5 Thread(s)
    If I've understood you correctly, you want Google to crawl your "category" directory, but not the single page within it called internetmarketing.php.

    In that case, all you need in your robots.txt file is:

    User-agent: Googlebot
    Disallow: /category/internetmarketing.php

    It will continue to crawl every other page in "category", (provided it can access them, of course).

    If you want to block all search engines, and not just Google, then use:

    User-agent: *
    Disallow: /category/internetmarketing.php

    Remember, this will only stop well-behaved bots whih respect the robots.txt file; it won't protet you from bad bots.


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts