SitePoint Sponsor

User Tag List

Results 1 to 5 of 5
  1. #1
    SitePoint Zealot animatics's Avatar
    Join Date
    Nov 2003
    Location
    Madison Wi
    Posts
    111
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    How do i restrict certain dynamic urls in robots.txt

    I have an online calendar that the search bots are crawling the heck out of. Normally I would think that was great but the problem i am having is I don't want them to crawl 1,5, 20 years and so on into the future where i have no information yet

    So how can I stop them from crawling a dynamic url

    abc.com/calendar.php?calview=event&event_id=16&prevview=month&year=2008&month=01&day=8&type_id=&clickdate=2008-01-8

    Or /calendar.php?calview=month&year=2008&month=2&type_id=

  2. #2
    Programming Team silver trophybronze trophy
    Mittineague's Avatar
    Join Date
    Jul 2005
    Location
    West Springfield, Massachusetts
    Posts
    17,290
    Mentioned
    198 Post(s)
    Tagged
    3 Thread(s)

    robots.txt

    AFAIK robots.txt only specifies folders and/or files. ie. the GET vars don't make a difference. If so, you can exclude the calendar.php file.

  3. #3
    SitePoint Zealot animatics's Avatar
    Join Date
    Nov 2003
    Location
    Madison Wi
    Posts
    111
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Nevermind I just figured out what I was trying to do

    I just put
    Disallow: /calendar.asp?
    which blocks everything after the ?

    and
    Disallow: /calendar.php?calview=month&year=2009
    which will hopefully stop them from indexing anything past 2009
    without blocking 2008 or calendar.php

  4. #4
    SitePoint Evangelist speda1's Avatar
    Join Date
    Jan 2002
    Posts
    550
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You can also check for the value in the page and spit out a meta nofollow if it exists.

  5. #5
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,424
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by speda1 View Post
    You can also check for the value in the page and spit out a meta nofollow if it exists.
    Or even the new headers supported by Google and Yahoo.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •