SitePoint Sponsor

User Tag List

Results 1 to 7 of 7
  1. #1
    SitePoint Enthusiast
    Join Date
    Dec 2000
    Posts
    35
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Can someone please tell me what "robots.txt" is? I see in my logs that there were 58 times that this page "robots.txt" was not found. I think that this has something to do with the search engine robots. Can someone please tell me what file this is and how to create one?

  2. #2
    SitePoint Wizard
    Join Date
    Jul 1999
    Location
    Chicago
    Posts
    2,629
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Check out this URL: http://www.searchtools.com/robots/robots-txt.html . Also, when there are simple easy-to-search things like this, just use Google...

  3. #3
    SitePoint Enthusiast
    Join Date
    Dec 2000
    Posts
    35
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks so much for the quick reply! That answered my question. Do you have any idea though if not having it can cause any robot to turn away?

  4. #4
    SitePoint Wizard
    Join Date
    Jul 1999
    Location
    Chicago
    Posts
    2,629
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    If you don't have one then robots will just index every single page that it deems to be non-dynamic. So, it's generally a good thing to have a robots.txt but it is definitely not necessary.

    It does take two seconds and may make the difference if there are _some_ robots who will ignore sites without robots.txt. I'd put it in anyways, even as a blank file or with comments.

    # this is a comment, for example

  5. #5
    SitePoint Enthusiast
    Join Date
    Dec 2000
    Posts
    35
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Originally posted by qslack
    If you don't have one then robots will just index every single page that it deems to be non-dynamic.
    Do you mean to say that if my site is dynamic which it is, then I won't get in to the Search Engines?! What should I do then? My site is made with PHP and every time you request a page it enters the database. I'm all lost if that what you're saying is true!

  6. #6

    Join Date
    Oct 1999
    Location
    UK
    Posts
    120
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Search engines tend not to index pages which contain a '?'. Dynamic pages are fine as long as they use some other symbol to indicate the query. Try http://httpd.apache.org/docs/mod/mod_rewrite.html

    Alternatively generate static pages from the database for the search engines.
    Simon Conroy,
    PPC Bid Management and ROI Tracking Solutions
    www.BidBuddy.co.uk www.ROIstats.co.uk

  7. #7
    SitePoint Enthusiast
    Join Date
    Dec 2000
    Posts
    35
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    simon c Thanks so much for telling me this! Because most of my pages include a "?". i.e. http://www.jumpondeals.com/bycategory.php3?cat=17 . So your suggestion is to just take out that "?" and put in something else? Is it easy to change?
    I'm gonna try to talk to my programmer about it, I just hope that he doesn't charge me another $1000 bucks for doing this!


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •