Prevent robots from accessing directory

Get Started
1

I would like to allow Robots to crawl my entire website except for files in my secure directory.

Is this how my robots.txt file should look?


User-agent: *
Allow: /
Disallow: /secure

Should there be a trailing forward-slash after the directory name (e.g. /secure/)?

Debbie

2

You don’t need the “Allow” line. Just this:

User-agent: *
Disallow: /secure/

Remember, however, that only well-behaved bots will honor the robots.txt file. Rogue bots will just ignore it.

You can find a lot of this stuff at The Web Robots Pages

3

Why do you include the trailing slash?

What happens if it is left off like in my example?

Debbie

4

Without the trailing slash, it assumes that “secure” is a file name, not a directory name. You could, for example, do this:

User-agent: *
Disallow: /secure/thisfile.html
5

It is a disallow protocol not an allow protocol using the second forward slash means directory; omitting it would also disallow access to a file called either: /secure.htm or directory /secure/index.htm

6