SitePoint Sponsor

User Tag List

Results 1 to 6 of 6

Hybrid View

  1. #1
    SitePoint Enthusiast
    Join Date
    Nov 2008
    Posts
    38
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Question Do not index maintenance pages

    How do I stop maintenance pages from being indexed by search engines?

    I can stop them from being used if the user does not have the correct privileges but it would be better if they weren't indexed to beging with.

    Thanks,

    Steve

  2. #2
    Mouse catcher silver trophy Stevie D's Avatar
    Join Date
    Mar 2006
    Location
    Yorkshire, UK
    Posts
    5,881
    Mentioned
    122 Post(s)
    Tagged
    1 Thread(s)
    If all the maintenance pages are in the same folder (and nothing else is in there), then you can use robots.txt, with a file containing something along the lines of
    Code:
    User-agent: *
    Disallow: /maintenance-folder/
    Otherwise, you'll need to set it on a file-by-file basis.
    Include this line in the <head>:
    Code:
    <meta name="robots" content="noindex, nofollow">
    (If you want search engines to follow links from that page then leave off "nofollow").

  3. #3
    Certified Ethical Hacker silver trophybronze trophy dklynn's Avatar
    Join Date
    Feb 2002
    Location
    Auckland
    Posts
    14,645
    Mentioned
    19 Post(s)
    Tagged
    3 Thread(s)
    Stevie(s),

    robot.txt is notorious for being ignored by bots attempting to index (or scrape) your website. If you're concerned about this, either:

    1. Move your maintenance scripts out of the webspace

    2. Unlink them from your website (no link from the website = security by obfuscation, i.e., poor to no security)

    3. Password protect your maintenance folder AND use mod_rewrite in that folder to require authentication, i.e., only provide access to your (fixed) IP Address or via an environmental variable only you have.

    4. The best option is a combination of (Apache) password protection on the subdirectory AND use of a login using a strong password hashed for access.

    Regards,

    DK
    David K. Lynn - Data Koncepts is a long-time WebHostingBuzz (US/UK)
    Client and (unpaid) WHB Ambassador
    mod_rewrite Tutorial Article (setup, config, test & write
    mod_rewrite regex w/sample code) and Code Generator

  4. #4
    SitePoint Enthusiast
    Join Date
    Nov 2008
    Posts
    38
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Thank you both.

    I will be using a mixture of meta tags to stop them from being indexed and a separate security on each page that checks that a user is logged in and has the correct privileges should a page become indexed by accident, or if search engines ignore the <meta robots>

    Steve

  5. #5
    Certified Ethical Hacker silver trophybronze trophy dklynn's Avatar
    Join Date
    Feb 2002
    Location
    Auckland
    Posts
    14,645
    Mentioned
    19 Post(s)
    Tagged
    3 Thread(s)
    Steve,

    If the maintenance pages are not isolated in a password protected directory, that's as good as it gets!

    BTW, I've been in a conversation with Manuel Lemos, creator of phpclasses.org about hosting classes to break md5 hashed passwords (using a rainbow table lookup hosted by md5cracker.org (or similar)) and the contention of mine is that these lookups shouldn't be available to script kiddies, his is that it can make for a good check on a hashed password. We're both correct but the key to security is to use a STRONG password, i.e., one with uppercase, lowercase, digits, special characters and spaces of sufficient length to make it impossible to crack by brute force in less than a few centuries. Of course, that's if you really need to protect your maintenance pages!)

    Regards,

    DK
    David K. Lynn - Data Koncepts is a long-time WebHostingBuzz (US/UK)
    Client and (unpaid) WHB Ambassador
    mod_rewrite Tutorial Article (setup, config, test & write
    mod_rewrite regex w/sample code) and Code Generator

  6. #6
    SitePoint Enthusiast
    Join Date
    Nov 2008
    Posts
    38
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    David,

    yes. I've become aware that md5 is not good enough.

    as for my own admin pages: at the moment it is not that critical. I provide the facility but I also maintain scripts so that if someone does break in and make a mess, I can just rerun scripts to make everything right again.

    When I rule the world there will be no need for any of this security malarky ( in any walk of life ).

    Thanks,

    Steve


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •