SitePoint Sponsor

User Tag List

Results 1 to 9 of 9

Thread: Robot.txt files

  1. #1
    SitePoint Guru kili's Avatar
    Join Date
    Jan 2004
    Location
    Damn I need a Crap
    Posts
    788
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Robot.txt files

    Hi Guys
    my robots.txt file is below

    User-agent: *
    Disallow: /cgi-bin-local/
    Disallow: /www-stats/

    if I add Disallow: /images/ I assume this will stop the spider from indexing my image directory but will it affect anything else i.e. the page display with images on


    Cheers
    Kili

  2. #2
    gingham dress, army boots... silver trophy redux's Avatar
    Join Date
    Apr 2002
    Location
    Salford / Manchester / UK
    Posts
    4,838
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    nope...it will only affect ethical spiders which read and adhere to robots.txt - normal browsers don't care about it.
    re·dux (adj.): brought back; returned. used postpositively
    [latin : re-, re- + dux, leader; see duke.]
    WaSP Accessibility Task Force Member
    splintered.co.uk | photographia.co.uk | redux.deviantart.com

  3. #3
    SitePoint Guru kili's Avatar
    Join Date
    Jan 2004
    Location
    Damn I need a Crap
    Posts
    788
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks Redux an ethical spider now theres a thought. What I'm tryig to do is stop my images being displayed in the google image search anyone know how to do this

    Kili

  4. #4
    SitePoint Enthusiast Ryka's Avatar
    Join Date
    Jul 2001
    Location
    A Globe
    Posts
    34
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by kili
    Thanks Redux an ethical spider now theres a thought. What I'm tryig to do is stop my images being displayed in the google image search anyone know how to do this

    Kili
    If you want to disallow just the Google image bot from indexing your site, put the following into your robot.txt file:

    User-agent: Googlebot-Image
    Disallow: /


    The regular Googlebot will still index your site, but Google images bot will no longer index your pics.

    You can find more info here: http://www.google.com/remove.html#images
    Ryka
    lost in a sea of experts

  5. #5
    SitePoint Member
    Join Date
    Nov 2003
    Location
    toronto, canada
    Posts
    19
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    umm google is an ethical spider, you can turn on hotlink protection with .htaccess that could work

  6. #6
    SitePoint Guru kili's Avatar
    Join Date
    Jan 2004
    Location
    Damn I need a Crap
    Posts
    788
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks Guys as always

    Kili

  7. #7
    Also available in Large Si's Avatar
    Join Date
    Sep 2002
    Location
    Walsall, UK
    Posts
    1,911
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Interesting - I have been working on SEO and robot.txt files for nearly two years now and I never realised there were "ethical" and "non-ethical" spiders.

    It sounds like the start of a dodgy plot for another Internet-based movie (remember that dodgy movie from 1997(?) called "Hackers? now that was funny!)
    Si
    Are you a Photoshop Jedi Master? Prove it!

    Is funky house your bag? You'll love this!

    Voice
    , eyes, ears, body and hands.


  8. #8
    gingham dress, army boots... silver trophy redux's Avatar
    Join Date
    Apr 2002
    Location
    Salford / Manchester / UK
    Posts
    4,838
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by bigsi
    Interesting - I have been working on SEO and robot.txt files for nearly two years now and I never realised there were "ethical" and "non-ethical" spiders.
    unethical/non-ethical spiders are, for instance, used to harvest email addresses. they read robots.txt, and then in some cases, rather than yield the request to stay out of specific directories, they specifically target them...
    re·dux (adj.): brought back; returned. used postpositively
    [latin : re-, re- + dux, leader; see duke.]
    WaSP Accessibility Task Force Member
    splintered.co.uk | photographia.co.uk | redux.deviantart.com

  9. #9
    Also available in Large Si's Avatar
    Join Date
    Sep 2002
    Location
    Walsall, UK
    Posts
    1,911
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Gits!
    Si
    Are you a Photoshop Jedi Master? Prove it!

    Is funky house your bag? You'll love this!

    Voice
    , eyes, ears, body and hands.



Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •