SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    SitePoint Addict tempyyyst's Avatar
    Join Date
    Nov 2001
    Location
    Kent, UK
    Posts
    286
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    How deep does Google spider

    Is there an advantage to having a shallow directory structure or will Google spider as deep as it can go?

    e.g. is www.yourdomain.com/page/1/3/4/index.html

    going to be spidered as throughly and as often as

    www.yourdomain.com/page/index.html



    In the example above the page with the deep directory structure can be reached with two clicks from the homepage.

  2. #2
    Level 8 Chinese guy Archbob's Avatar
    Join Date
    Sep 2001
    Location
    Somewhere in this vast universe
    Posts
    3,741
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    In my experience google spider the page you submit and the pages it links to.

  3. #3
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Google spiders all 10,000 of the pages on my literature site, it only adds 2000-3000 of them to it's public index though.

    So it will spider anything it finds a link to.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  4. #4
    One website at a time mmj's Avatar
    Join Date
    Feb 2001
    Location
    Melbourne Australia
    Posts
    6,282
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    I believe that google will usually give a higher page rank to pages that are shallower (ie, pages that it can find within one hop of your index page). Having a good amount of internal links throughout your site both helps google to spider your site, but may improve the pank rank of some pages.
    [mmj] My magic jigsaw
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    The Bit Depth Blog Twitter Contact me
    Neon Javascript Framework Jokes Android stuff

  5. #5
    My precious!!! astericks's Avatar
    Join Date
    Mar 2002
    Location
    Vancouver, BC
    Posts
    1,971
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Originally posted by aspen
    So it will spider anything it finds a link to.
    If i understand this correctly, if you have
    <META name="robots" content="index,follow">

    in the <head> section of every page, it should spider the whole website, though it wont add everything to its public index...

    is this accurate?

  6. #6
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You don't need the meta tag - and it should spider the whole site, if it can find your links.

    However they only add pages they deem important to their public index.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  7. #7
    SitePoint Enthusiast Sharon & Roy's Avatar
    Join Date
    Feb 2001
    Location
    Weldon, California, USA
    Posts
    93
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    The Key To Being Spidered AND Indexed Is To Being Linked (Heavily)

    Originally posted by astericks

    If I understand this correctly, if you have

    <META name="robots" content="index,follow">

    In the <head> section of every page, it should spider the whole Website, though it won't add everything to its public index...

    Is this accurate?
    Hello astericks,

    In a word, NO.

    A search engine friendly page is one that has LOTS of incoming links from other pages already in the search engine's database.

    If a page has only one or two links and they don't produce a sufficient amount of link popularity then the search engines may not list them, even though it has been spidered.

    Think about this ... Not ALL pages that are spidered (a.k.a. crawled) are listed in the search engine databases.

    For example, AlltheWeb actually crawls over 5 billion pages, but indexes only a little over 2 billion pages. The reason is because the other pages do not contain quality material.


    We would suggest you add a link from your Home Page to your Site Map page.

    Your Site Map page will link to EACH of your pages and each of your pages will also link back to it. On the Site Map Page don't just list all your links but also summarize them. Acquire LOTS of Backward Links (inbound links) to your Home Page AND your most important pages from RELATED pages and sites. This will insure that your pages get listed (as long as they are quality and not spam).

    If you need more specific help, just ask.
    Call For A FREE 1 Hour SEO Consultation 760-378-2539 (Pacific)
    Want To Learn All About Search Engine Optimization? We Teach A 6
    Week Online SEO Class - There Is So Much More To SEO Than Just
    Meta Tags ::: http://www.iwon.net/seo/class-description/

  8. #8
    My precious!!! astericks's Avatar
    Join Date
    Mar 2002
    Location
    Vancouver, BC
    Posts
    1,971
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hey Sharon & Roy,

    thank you for the reply and for the tips. SEO is not really my concern since the only website I manage does not depend on search engines for traffic, though quite surprisingly, we do get a lot of referrals from search engines, esp. google.

    However, to satisfy my curiosity and most probably for future use, I'm currently reading almost all the website related articles at sitepoint...so, expect quite a few questions from me in the near future.

    Again, thanks
    asT.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •