SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Guru kili's Avatar
    Join Date
    Jan 2004
    Location
    Damn I need a Crap
    Posts
    788
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Googles Robot.txtx file

    An intresting robots.txt file on google.com. I see they have a folder dedicated to Microsoft.
    User-agent:

    User-agent: *Disallow: /searchDisallow: /groupsDisallow: /imagesDisallow: /catalogsDisallow: /catalog_listDisallow: /newsDisallow: /addurl/image?Disallow: /pagead/Disallow: /relpage/Disallow: /sorry/Disallow: /imgresDisallow: /keyword/Disallow: /u/Disallow: /univ/Disallow: /cobrandDisallow: /customDisallow: /advanced_group_searchDisallow: /advanced_searchDisallow: /googlesiteDisallow: /preferencesDisallow: /setprefsDisallow: /swrDisallow: /urlDisallow: /wmlDisallow: /hwsDisallow: /bsd?Disallow: /linux?Disallow: /mac?Disallow: /microsoft?Disallow: /unclesam?Disallow: /answers/search?q=Disallow: /localDisallow: /froogle?Disallow: /froogle_Disallow: /print?Disallow: /scholar?Disallow: /palmDisallow: /completeDisallow: /sponsoredlinks
    It's not how long you live that counts but how you live

  2. #2
    Non-Member Egor's Avatar
    Join Date
    Jan 2004
    Location
    Melbourne, Australia
    Posts
    7,305
    Mentioned
    1 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by kili
    /mac?Disallow:
    What the heck does that mean??

    Quote Originally Posted by kili
    : /microsoft?Disallow:
    Excellente!

  3. #3
    web designer
    Join Date
    Jun 2004
    Location
    Liverpool, uk
    Posts
    178
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

  4. #4
    SitePoint Guru kili's Avatar
    Join Date
    Jan 2004
    Location
    Damn I need a Crap
    Posts
    788
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by mstwntd
    What the heck does that mean??


    Excellente!
    what does what mean?
    It's not how long you live that counts but how you live


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •