SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Thread: robots.txt

  1. #1
    SitePoint Guru
    Join Date
    Mar 2001
    Location
    Highlands Ranch, Colorado
    Posts
    614
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    robots.txt

    Is there any good tutorial how to configure a robots.txt file? I want to make sure that 'spiders' only find what they are supposed to find and I think I remember that with help of a robots.txt file you can lock 'em out of some directories.

    Additional question: Is there anything different in the process between IIS and Apache in regards to the robots.txt file handling?

    Thanks!

    Christoph
    The-USA dot net - PR4 Web Directory

  2. #2
    Prolific Blogger silver trophy Technosailor's Avatar
    Join Date
    Jun 2001
    Location
    Before These Crowded Streets
    Posts
    9,446
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Give this a whirl...a nice Google search.
    http://www.google.com/search?q=%22robots.txt%22
    Sketch
    Aaron Brazell
    Technosailor



  3. #3
    SitePoint Guru
    Join Date
    Mar 2001
    Location
    Highlands Ranch, Colorado
    Posts
    614
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks! That's what I needed.

    Have a nice day!

    Christoph
    The-USA dot net - PR4 Web Directory


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •