SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Enthusiast S.Sameer's Avatar
    Join Date
    Sep 2007
    Location
    UP, India.
    Posts
    25
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    To allow robots to view individual pages in a dir included via PHP or not?

    I am using an Apache mod called mod_rewrite. What this does is just this:

    eg. www.site.com/friendly-seo-link-here

    Now I simply have an index.php page on www.site.com and what that does is just include the right page from a directory called /pages by matching it with whatever comes after www.site.com/

    So for eg. if user clicks on www.site.com/article-1

    It will include www.site.com/pages/article-1.php in the content area while still displaying the URL www.site.com/article-1

    File article-1.php just contains the content which needs to be displayed.

    So my question is do I allow the robots to crawl the /pages/ directory or not?

  2. #2
    In memoriam gold trophysilver trophybronze trophy Dan Schulz's Avatar
    Join Date
    May 2006
    Location
    Aurora, Illinois
    Posts
    15,476
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    No.

  3. #3
    SitePoint Enthusiast S.Sameer's Avatar
    Join Date
    Sep 2007
    Location
    UP, India.
    Posts
    25
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thank you for the answer, Dan Schulz.

  4. #4
    In memoriam gold trophysilver trophybronze trophy Dan Schulz's Avatar
    Join Date
    May 2006
    Location
    Aurora, Illinois
    Posts
    15,476
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    No problem. The general rule of thumb is to basically allow the robots.txt file to let search engines crawl only those pages you want your regular visitors to see (note: I do tend to block my images folder, though that's a personal preference). Especially where duplicate content (or the mere threat of it) is involved.

    Oh, and you can just call me Dan.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •