SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Hybrid View

  1. #1
    SitePoint Guru
    Join Date
    Nov 2008
    Posts
    622
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    disallow directory from search engines without writing fullname

    hi

    i want to disallow a particular directory from search engines.

    Code:
    User-agent: *
    Disallow: /folder1/
    
    User-Agent: Googlebot
    Disallow: /folder2/
    The above code works but i dont want to mention the full name of the directory in robots.txt

    what should i do

    vineet

  2. #2
    Non-Member
    Join Date
    Jun 2013
    Posts
    71
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    It is not possible to block any kind of directory without their original name Hence if you want to block particular directory then it is require to mention full name or path. Check below details you will get more ideas about robots.txt : http://www.robotstxt.org/robotstxt.html

  3. #3
    Certified Ethical Hacker silver trophybronze trophy dklynn's Avatar
    Join Date
    Feb 2002
    Location
    Auckland
    Posts
    14,672
    Mentioned
    19 Post(s)
    Tagged
    3 Thread(s)
    vin,

    It is possible:

    1. Do not link to the directory. Security by obfuscation is little security but SE's don't try to find non-linked pages.

    2. As KJ said, you must list the directory in robots.txt to tell honest SE's they're not allowed to follow links to that directory. Bad SE's, however, will merely use that information to take them into the directories you want to protect.

    3. If you must link, then use password protection on the directory. That'll stop SE's in their tracks!

    4. If none of those suit you, create a form whose action is to take visitors to the forbidden land via submit. SE's won't be searching for that as a link so it may work to hide the link.

    Think outside the box: robots.txt is next to worthless so be creative with other solutions.

    Regards,

    DK
    David K. Lynn - Data Koncepts is a long-time WebHostingBuzz (US/UK)
    Client and (unpaid) WHB Ambassador
    mod_rewrite Tutorial Article (setup, config, test & write
    mod_rewrite regex w/sample code) and Code Generator


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •