Do not index maintenance pages

Server Config
1

How do I stop maintenance pages from being indexed by search engines?

I can stop them from being used if the user does not have the correct privileges but it would be better if they weren’t indexed to beging with.

Thanks,

Steve

2

[FONT=Verdana]If all the maintenance pages are in the same folder (and nothing else is in there), then you can use robots.txt, with a file containing something along the lines of

User-agent: *
Disallow: /maintenance-folder/

Otherwise, you’ll need to set it on a file-by-file basis.
Include this line in the <head>:

<meta name="robots" content="noindex, nofollow">

(If you want search engines to follow links from that page then leave off “nofollow”).[/FONT]

3

Stevie(s),

robot.txt is notorious for being ignored by bots attempting to index (or scrape) your website. If you’re concerned about this, either:

  1. Move your maintenance scripts out of the webspace

  2. Unlink them from your website (no link from the website = security by obfuscation, i.e., poor to no security)

  3. Password protect your maintenance folder AND use mod_rewrite in that folder to require authentication, i.e., only provide access to your (fixed) IP Address or via an environmental variable only you have.

  4. The best option is a combination of (Apache) password protection on the subdirectory AND use of a login using a strong password hashed for access.

Regards,

DK

4

Thank you both.

I will be using a mixture of meta tags to stop them from being indexed and a separate security on each page that checks that a user is logged in and has the correct privileges should a page become indexed by accident, or if search engines ignore the <meta robots>

Steve

5

Steve,

If the maintenance pages are not isolated in a password protected directory, that’s as good as it gets!

BTW, I’ve been in a conversation with Manuel Lemos, creator of phpclasses.org about hosting classes to break md5 hashed passwords (using a rainbow table lookup hosted by md5cracker.org (or similar)) and the contention of mine is that these lookups shouldn’t be available to script kiddies, his is that it can make for a good check on a hashed password. We’re both correct but the key to security is to use a STRONG password, i.e., one with uppercase, lowercase, digits, special characters and spaces of sufficient length to make it impossible to crack by brute force in less than a few centuries. Of course, that’s if you really need to protect your maintenance pages!)

Regards,

DK

6

David,

yes. I’ve become aware that md5 is not good enough.

as for my own admin pages: at the moment it is not that critical. I provide the facility but I also maintain scripts so that if someone does break in and make a mess, I can just rerun scripts to make everything right again.

When I rule the world there will be no need for any of this security malarky ( in any walk of life ).

Thanks,

Steve