SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Enthusiast
    Join Date
    Jul 2012
    Posts
    49
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    help!! Stop individual folder being crawled and indexed

    Hi guys,

    Not sure whether this is the right section to post this question, but I was wondering whether you could help with stopping google spiders etc crawling and indexing a particular folder on my site.
    I searched around and found this method.

    User-agent:*
    Disallow:/foldername/


    Is this the best way to do this? Do I just upload the file to the root folder?

  2. #2
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,608
    Mentioned
    24 Post(s)
    Tagged
    1 Thread(s)
    Legitimate spiders all generally look for and obey the content of the robots.txt file in the root folder. Bad bots may also read the file but will not obey the instruction.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">

  3. #3
    SitePoint Enthusiast
    Join Date
    Jul 2012
    Posts
    49
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by felgall View Post
    Legitimate spiders all generally look for and obey the content of the robots.txt file in the root folder. Bad bots may also read the file but will not obey the instruction.
    Thanks felgall!

    So, basically this is the best that I can do, just minimise the amount of indexing?

  4. #4
    It's all Geek to me silver trophybronze trophy
    ralph.m's Avatar
    Join Date
    Mar 2009
    Location
    Melbourne, AU
    Posts
    23,620
    Mentioned
    413 Post(s)
    Tagged
    7 Thread(s)
    You could also consider spanking those naughty bots by creating a black hole for them, as described here:

    http://perishablepress.com/blackhole-bad-bots/

  5. #5
    SitePoint Enthusiast
    Join Date
    Jul 2012
    Posts
    49
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by ralph.m View Post
    You could also consider spanking those naughty bots by creating a black hole for them, as described here:

    http://perishablepress.com/blackhole-bad-bots/


    Thanks so much ralph.m


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •