SitePoint Sponsor

User Tag List

Results 1 to 15 of 15
  1. #1
    SitePoint Guru Rebirth Studios's Avatar
    Join Date
    Mar 2003
    Posts
    621
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Backlinks from a Disallowed Website

    I've noticed a significant amount of traffic coming from a site which has set its robots.txt to disallow the whole site.

    Since the site is configured this way, I'm assuming each page linking to us has no link value and therefore we're not getting any juice, just the traffic--is the correct assumption?

    I found some of the pages because they were linked to from another site, which is where I assume they were indexed.

  2. #2
    Galactic Overlord gold trophysilver trophybronze trophy
    HAWK's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    12,540
    Mentioned
    956 Post(s)
    Tagged
    14 Thread(s)
    Yes, you are correct.
    Nofollow will stop spiders from travelling those links, therefore no PR will be passed.

  3. #3
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    If the pages can't be seen by the search engines then they don't exist in their eyes.

  4. #4
    SitePoint Guru Rebirth Studios's Avatar
    Join Date
    Mar 2003
    Posts
    621
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    So even though the page on the disallowed (robots.txt) site is linked from another site, SE's won't index it because when they hit the site, they check the robots.txt before following the inroad, right?

  5. #5
    Galactic Overlord gold trophysilver trophybronze trophy
    HAWK's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    12,540
    Mentioned
    956 Post(s)
    Tagged
    14 Thread(s)
    They index the page but don't follow any of the outbound links on it.

  6. #6
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Actually they won't even crawl any pages blocked by robots.txt.

  7. #7
    Galactic Overlord gold trophysilver trophybronze trophy
    HAWK's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    12,540
    Mentioned
    956 Post(s)
    Tagged
    14 Thread(s)
    Yup, I mean that they index the page with the blocked links on it and stop there.

  8. #8
    SitePoint Member
    Join Date
    Dec 2007
    Posts
    3
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    But like you said you can still get traffic from these sites

  9. #9
    SitePoint Wizard bronze trophy hooperman's Avatar
    Join Date
    Jan 2006
    Location
    Manchester, UK
    Posts
    4,301
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by HAWK View Post
    Yup, I mean that they index the page with the blocked links on it and stop there.
    It's not just the links that are blocked, it's the whole site. Nothing will get indexed.

  10. #10
    SitePoint Zealot WEBLAUNCHPHXX's Avatar
    Join Date
    Jul 2007
    Posts
    171
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by lbouman View Post
    But like you said you can still get traffic from these sites
    Offcource your site will get traffics but not index due to robot.txt file.
    As for example many forums use robot.txt but you can get traffics by your signature links.

  11. #11
    Error 404: Life not found silver trophybronze trophy
    Join Date
    Dec 2007
    Location
    UK Nr Manchester
    Posts
    3,460
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Rebirth Studios View Post
    So even though the page on the disallowed (robots.txt) site is linked from another site, SE's won't index it because when they hit the site, they check the robots.txt before following the inroad, right?
    Yeah, the first thing a spider does is request the robot.txt file, or should do anyway. If they don't find one they default to assuming that everything on the site fair game for indexing.

    The link has no value except the traffic it's sending you.

  12. #12
    SitePoint Guru Rebirth Studios's Avatar
    Join Date
    Mar 2003
    Posts
    621
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks All

  13. #13
    Galactic Overlord gold trophysilver trophybronze trophy
    HAWK's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    12,540
    Mentioned
    956 Post(s)
    Tagged
    14 Thread(s)
    Quote Originally Posted by hooperman View Post
    It's not just the links that are blocked, it's the whole site. Nothing will get indexed.
    So if lets say for example that this page was set to nofollow. This page would be indexed, correct? But none of the links on it would be followed (and therefore none of the pages at the end of the links).

  14. #14
    He's No Good To Me Dead silver trophybronze trophy stymiee's Avatar
    Join Date
    Feb 2003
    Location
    Slave I
    Posts
    23,423
    Mentioned
    2 Post(s)
    Tagged
    1 Thread(s)
    Quote Originally Posted by HAWK View Post
    So if lets say for example that this page was set to nofollow. This page would be indexed, correct? But none of the links on it would be followed (and therefore none of the pages at the end of the links).
    In that specific example that would be correct.

  15. #15
    Galactic Overlord gold trophysilver trophybronze trophy
    HAWK's Avatar
    Join Date
    Aug 2003
    Location
    New Zealand
    Posts
    12,540
    Mentioned
    956 Post(s)
    Tagged
    14 Thread(s)
    Good. That is what I've been (badly) trying to say.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •