SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Thread: Robot.txt file

  1. #1
    SitePoint Zealot WEBLAUNCHPHXX's Avatar
    Join Date
    Jul 2007
    Posts
    171
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Robot.txt file

    The robot text is a file that restricts the Search Engine spiders from indexing certain pages of the website.You may prevent your personal/incomplete pages as well as guestbook pages of your site from indexing through this file. Many webmasters use it to avoid spamming. Robot txt codes are listed below.

    HTML meta tags for robots.
    < meta name="robots" content="noindex,nofollow" />

    To allows all robots
    User-agent: *
    Disallow:

    To all robots out
    User-agent: *
    Disallow: /

    To prevent pages from all crawlers
    User-agent: *
    Disallow: /page name/


    To prevent pages from specific crawler
    User-agent: GoogleBot
    Disallow: /page name/

  2. #2
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    You might have better served everyone by sending them here instead: http://www.robotstxt.org/


  3. #3
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,836
    Mentioned
    25 Post(s)
    Tagged
    1 Thread(s)
    The robots.txt file doesn't prevent spamming. Only the legitimate robots read it, the spambots just ignore it.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •