SitePoint Sponsor

User Tag List

Results 1 to 7 of 7
  1. #1
    Non-Member
    Join Date
    Jun 2008
    Location
    Phuket
    Posts
    15
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    What is the robots.txt file?

    What is the robots.txt file?

  2. #2
    Is Still Alive silver trophybronze trophy RetroNetro's Avatar
    Join Date
    Jul 2003
    Location
    Texas
    Posts
    6,883
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

  3. #3
    SitePoint Zealot water2filter's Avatar
    Join Date
    Nov 2007
    Location
    China
    Posts
    113
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    basically speaking,robot.txt is the file to tell search engine,on your site which content is allowed to index,which content to keep private.

  4. #4
    SitePoint Enthusiast mcaronan's Avatar
    Join Date
    Oct 2007
    Posts
    36
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    robot.txt is a text file that instruct robots what they need or not to crawl certain pages in your website.

    Why do people want spiders not to crawl certain pages?
    There are lot of things why, one good example is that your back up your website files in your host. If spiders detected this files, search engines might think that these files are duplicate files.
    Can't afford your dream house? Home Loan Calculator can help you.

    Asian Tourism | Malaysia Property |USA Debt Settlement

  5. #5
    SitePoint Member
    Join Date
    Jul 2008
    Posts
    14
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    to be really basic it tells crawlers where to go and not to go, what to index and what not to index, which crawlers are allowed and not allowed to search the site
    Create a free blog | Sexy Costumes [Womens lingerie]

  6. #6
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,820
    Mentioned
    25 Post(s)
    Tagged
    1 Thread(s)
    Spambots will of course completely ignore the instructions in that file or use it to work out where you don't want bots to go so that they can make sure that they go there.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">

  7. #7
    SitePoint Enthusiast lukemeister's Avatar
    Join Date
    Jun 2006
    Posts
    84
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by felgall View Post
    Spambots will of course completely ignore the instructions in that file or use it to work out where you don't want bots to go so that they can make sure that they go there.
    Spambots are pretty much Internet Gremlins, evil


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •