Prevent robots from accessing directory

I would like to allow Robots to crawl my entire website except for files in my secure directory.

Is this how my robots.txt file should look?

User-agent: *
Allow: /
Disallow: /secure

Should there be a trailing forward-slash after the directory name (e.g. /secure/)?


You don’t need the “Allow” line. Just this:

User-agent: *
Disallow: /secure/

Remember, however, that only well-behaved bots will honor the robots.txt file. Rogue bots will just ignore it.

You can find a lot of this stuff at The Web Robots Pages

Why do you include the trailing slash?

What happens if it is left off like in my example?


Without the trailing slash, it assumes that “secure” is a file name, not a directory name. You could, for example, do this:

User-agent: *
Disallow: /secure/thisfile.html

It is a disallow protocol not an allow protocol using the second forward slash means directory; omitting it would also disallow access to a file called either: /secure.htm or directory /secure/index.htm