SitePoint Sponsor

User Tag List

View Poll Results: Robots.txt - Is it Worth it???

Voters
11. You may not vote on this poll
  • Yes - It is worth it!!

    5 45.45%
  • No - It is not worth the time and trouble!!

    1 9.09%
  • Probably Helpful but I do not use it

    2 18.18%
  • I do not know

    3 27.27%
  • No Response

    0 0%
Results 1 to 3 of 3
  1. #1
    ********* Addict WP1's Avatar
    Join Date
    Aug 2000
    Location
    New York , USA
    Posts
    407
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hello,

    Looking for some opioions
    [list=1][*]What do you feel about the use of the Robots.txt file for search engine indexing??[*]Is it worth putting energy into creating a robots.txt file???[*]Does it provide an accurate way to get a site indexed by a search engine. That is, indexed the way a Webmaster wants their site to be indexed??[*]Are that many search engines actually utilized the robots.txt file in their indexing programs??[/list=1]
    Thanks in advance for all replies.

    WP
    Webbing Along!!!

  2. #2
    SitePoint Enthusiast Sharon & Roy's Avatar
    Join Date
    Feb 2001
    Location
    Weldon, California, USA
    Posts
    93
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Originally posted by WP1
    What do you feel about the use of the Robots.txt file for search engine indexing?
    All major reputable Search Engines honor the robots.txt file. And yes it does provide an acurate way TO index or to keep FROM indexing your Web Pages by various Search Engines.

    Here is some insightful information ...

    Information on how to implement the Robot Exclusion Text File and the Robots Meta Tag from the World Wide Consortium [W3C]
    http://www.w3.org/TR/REC-html40/appe...s.html#h-b.4.1

    A Standard for Robot Exclusion - How To Format & Examples
    http://info.webcrawler.com/mak/proje.../norobots.html

    The Web Robots Database
    http://info.webcrawler.com/mak/proje...ts/active.html

    List of the Major Spiders
    http://info.webcrawler.com/mak/proje...s/active/html/

    Spider Chart - A List of the Search Engine Spiders
    http://www.searchenginewatch.com/web...iderchart.html

    To check the syntax of your robots.txt file
    http://www.tardis.ed.ac.uk/home/sxw/robots/check/

    View the actual robots.txt file for the World Wide Consortium [W3C]
    http://www.w3.org/robots.txt
    Call For A FREE 1 Hour SEO Consultation 760-378-2539 (Pacific)
    Want To Learn All About Search Engine Optimization? We Teach A 6
    Week Online SEO Class - There Is So Much More To SEO Than Just
    Meta Tags ::: http://www.iwon.net/seo/class-description/

  3. #3
    ********* Addict WP1's Avatar
    Join Date
    Aug 2000
    Location
    New York , USA
    Posts
    407
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi Sharon & Roy,

    THANK YOU!!
    THANK YOU!!
    THANK YOU!!

    I am Exhausted

    Lots of reading to do. I am becoming less and less afraid of the Big Bad <META> tag Bear. Don't want to get over confident though.

    Appreciate you taking the time to provide me with such helpful information.

    Thanks again,

    WP
    Webbing Along!!!


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •