SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Addict SunnaH's Avatar
    Join Date
    Aug 2005
    Posts
    263
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Question Site Indexing and Folder Security

    I have excluded a directory in my robots.txt and site map. But I would like Google to index those pages but if the pages have any secure documents linked on them such as PDF/Word files on my server I do not want that directory access. As an interim solution I am not indexing a large part of my member-only site. How can I secure the folder yet have the member pages indexed? I have read the solution of putting my /folder/ outside the Web accessible folder but i do not know how to configure this. My member pages have an authentication script with checked for cookies and passes variables. Will the Google spider get to these pages regardless? I have an IIS server.
    -=SunnaH=-

  2. #2
    Follow Me On Twitter: @djg gold trophysilver trophybronze trophy Dan Grossman's Avatar
    Join Date
    Aug 2000
    Location
    Philadephia, PA
    Posts
    20,580
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Put the files you don't want Google to access in a folder, and disallow that folder in your robots.txt. It's those files you want to prevent indexing, not the pages that might link to them.

    If you already have authentication set up to prevent direct access to those files without being logged in, then neither Google nor any other search engine can access those files regardless of your robots.txt.

  3. #3
    SitePoint Addict SunnaH's Avatar
    Join Date
    Aug 2005
    Posts
    263
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Just to clarify:

    The pages and the directory of files I do not want Google to spider are declared in my robots.txt file. The pages have an authentication script to check if the login cookie is there only. I wanted to make sure if I allow the Google spider to index the protected pages only that the files in the folder of files which are linked from those pages will not be exposed as well to the Google spider. In other words is it safe to allow the index of the protected pages without access to the files in the folder?
    -=SunnaH=-


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •