SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Thread: robots.txt ?

  1. #1
    SitePoint Zealot
    Join Date
    May 2004
    Posts
    103
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    robots.txt ?

    can someone explain what should be in robots.txt ?

  2. #2
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,424
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Pages that you don't want to have indexed by search engines or crawled by other bots. If you want every page and file to be crawled, don't make a robots.txt file.

  3. #3
    Application Developer shabbirbhimani's Avatar
    Join Date
    Apr 2004
    Location
    India
    Posts
    2,272
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    User-agent: *
    Disallow: /folder1
    Disallow: /folder2
    Disallow: /images
    .
    .
    .

    Thanks
    Shabbir Bhimani


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •