SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    SitePoint Enthusiast MonsterZero's Avatar
    Join Date
    Jul 2002
    Location
    Worth, IL, USA
    Posts
    96
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Question Show Me an Example of a "Crawler Page"

    Below is an excerpt from a search engine optimization article at Webmonkey. The author provides a link to a sample "crawler page" but it's not working. The server is not responding. Thanks in advance for all views and replies. MonsterZero.
    ------------------------

    Make a Crawler Page

    It isn't necessary to submit every page on your site to the search engines. Just make sure they can find all the pages that matter by hopping links from your front door. To do that, make a "crawler page" that contains nothing but a link to every page you want search engines to crawl. Use the page's TITLE info as the link text this helps improve your site score. For an example, check out Artloop's crawler page.

    Basically, the crawler page is a site map that lists all the pages on your site it may be a bit too big for humans to read through, but it will be no problem for a search engine. Add an obscure link to the crawler page on one of your site's top-level pages, using a small amount of text. MSN used to use 1x1 images for this trick, but the Google geeks warned us to avoid such obviously invisible tags. "Why not just label it 'site map?'" one asked. Search engine spiders will find it as soon as they get to your site, and suck down all the pages it finds on it.

  2. #2
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Not necessary, also an obscene amount of links on one page raises flags at google.

    If a person can browse from your home page to a page deep within your site, then a spider can too.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  3. #3
    SitePoint Enthusiast MonsterZero's Avatar
    Join Date
    Jul 2002
    Location
    Worth, IL, USA
    Posts
    96
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Question

    Thanks for reply but what if I like site maps in my site? Lots of links crammed into a single site map document? Am I in danger of my site being rejected by Google because of that single site map doc?

  4. #4
    ********* Callithumpian silver trophy freakysid's Avatar
    Join Date
    Jun 2000
    Location
    Sydney, Australia
    Posts
    3,798
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    No - but the value of those links will be diminished because of the amount of links on the page. In other words they just won't be weighted as highly by google. Aspen and others would know more about this voodoo.

  5. #5
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You can have a sitemap, just don't overdo it.

    100 links? Maybe okay. 500 links? Playing with fire. 1000 links? Where's the hose?

    Worst that'll happen is your sitemap page will not be indexed by Google, they'll spider it but they're neither count the links nor add the page to their index. I tried this once with about 1000 links (my site's got over 10,000 pages) and thats what happened.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  6. #6
    SitePoint Enthusiast Sharon & Roy's Avatar
    Join Date
    Feb 2001
    Location
    Weldon, California, USA
    Posts
    93
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Here Are A Few Site Map ("Crawler Page") Examples

    Originally posted by MonsterZero

    Thanks for reply but what if I like site maps in my site? Lots of links crammed into a single site map document? Am I in danger of my site being rejected by Google because of that single site map doc?
    Hello MonsterZero,

    In a word ... NO.

    Just keep your Site Map page(s) under 80k in size each and you'll be just fine for Google.

    Although Google does index up to 101k of your page, we would not advise you to create pages any larger than 80k.

    It is also good to keep in mind that other search engines may index a smaller portion of your page than Google or even a smaller portion than 80k of your pages.

    Therefore, it is wise to check the other engines you want to be included in for page size limits. (We are not currently aware of such a size chart anywhere. If anyone knows, they might post it for you, so you can check.)

    Show Me an Example of a "Crawler Page"
    Here are some examples of the top ranked Site Map ("Crawler Pages") pages on Google ...

    (Note: Most of these pages have very high PageRank scores.)

    ------------

    1113 Links - PageRank8
    http://www.ncbi.nlm.nih.gov/Sitemap/
    248k (NOTE: Google only indexes up to 101k)

    945 Links - PageRank5
    http://www.ousu.org/system/sitemap
    159k (NOTE: Google only indexes up to 101k)

    846 Links - PageRank7
    http://www.harcourtschool.com/sitemap/
    109k (NOTE: Google only indexes up to 101k)

    636 Links - PageRank7
    http://www.findlaw.com/toc.html
    67k

    545 Links - PageRank8
    http://www.npwrc.usgs.gov/sitemap.htm
    60K (NOTE: Mostly Just Links)

    486 Links - PageRank7
    http://www.miami.edu/UMH/CDA/UMH_Mai...3033-3,00.html
    113k (NOTE: Google only indexes up to 101k)

    477 Links - PageRank8
    http://www.accuweather.com/adcbin/sitemap
    72k

    464 Links - PageRank5
    http://www.wn.com/s/sitemap/index.html
    60K

    351 Links - PageRank7
    http://www.urban.org/template.cfm?Se...p&NavMenuID=60
    136k (NOTE: Google only indexes up to 101k)

    323 Links - PageRank7
    http://www.ausport.gov.au/sitemap.htm
    139k (NOTE: Google only indexes up to 101k)

    307 Links - PageRank (Current page is not ranked by Google)
    http://www.nasdaq.com/sitemap/sitemap.stm
    80k

    239 Links - PageRank6
    http://www.azfamily.com/sitemap/
    137k (NOTE: Google only indexes up to 101k)

    298 Links - PageRank8
    http://www.cmp.com/sitemap/
    64k

    274 Links - PageRank (Current page is not ranked by Google)
    http://www.latimes.com/services/site...emap.htmlstory
    67k

    91 Links - PageRank9
    http://www.intel.com/intel/nav/sitemaps.htm
    72k

    273 Links - PageRank8
    http://pages.ebay.com/sitemap.html
    42k

    272 Links - PageRank7
    http://the.honoluluadvertiser.com/current/help/sitemap
    71k

    195 Links - PageRank7
    http://www.pe.com/sitemap/
    79k

    230 Links - PageRank7
    http://www.dallasnews.com/sitemap/
    83k

    275 Links - PageRank7
    http://www.marconi.com/html/sitemap/
    76k

    255 Links - PageRank7
    http://www.shrm.org/sitemap/
    85K

    179 Links - PageRank8
    http://www.fbi.gov/sitemap.htm
    45K

    299 Links - PageRank6
    http://www.king5.com/sitemap/
    79K

    ------------
    Call For A FREE 1 Hour SEO Consultation 760-378-2539 (Pacific)
    Want To Learn All About Search Engine Optimization? We Teach A 6
    Week Online SEO Class - There Is So Much More To SEO Than Just
    Meta Tags ::: http://www.iwon.net/seo/class-description/

  7. #7
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,937
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Just because the toolbar reports a PageRank doesn't mean the page is in Google. If the toolbar finds a page on a domain where the domain is indexed but not the page it will guess a PageRank based on the domain.

    Also the problem is when you have only links, having content interspersed with links means you can have more links on that one page.

    For instance the first page you link to has a ton of content with the links, so that page is listed.

    The second page is all links, try to find backwards links or a cached snapshot of that page, you'll notice that Google has not indexed it.

    So like I said, if you make a page with all links and you get close to 1000 links you're in trouble.

    Of course when I made my sitemap it was well over a year ago, and things could have changed since then.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  8. #8
    SitePoint Enthusiast MonsterZero's Avatar
    Join Date
    Jul 2002
    Location
    Worth, IL, USA
    Posts
    96
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks a lot for help, everybody. Sharon & Roy's advice for keeping my HTML dosc under 80K in file size caught my attention.

    Hmmm....What about other search services? What's the max file size per page if I want to avoid problems with most search engines, not just Google?


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •