SitePoint Sponsor

User Tag List

Results 1 to 15 of 15
  1. #1
    SitePoint Enthusiast
    Join Date
    Apr 2005
    Posts
    51
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Search Engine doesn't index past 1st page..

    Google and Yahoo have indexed my website. However, it doesn't seem like they've indexed past my first page.

    I used a search engine spider simulator (http://www.searchengineworld.com/cgi-bin/sim_spider.cgi) to figure out what's going on, and it seems like the spider sees all the links on my site as just file names and not full URLs to crawl further.

    How can I get Google and Yahoo to index the rest of my site?
    Last edited by clickagent75; Mar 20, 2006 at 15:05.

  2. #2
    Word Painter silver trophy Shyflower's Avatar
    Join Date
    Oct 2003
    Location
    Winona, MN USA
    Posts
    10,053
    Mentioned
    142 Post(s)
    Tagged
    2 Thread(s)
    Your site needs readable optimized content for the Spiders to pay attention to it. Most SEO experts agree that pages with less than 200 words of optimized content are generally ignored by search engine spiders.
    Linda Jenkinson
    "Say what you mean. Mean what you say. But don't say it mean." ~Unknown

  3. #3
    SitePoint Guru scotlandforvisit's Avatar
    Join Date
    Aug 2004
    Location
    Scotland
    Posts
    687
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    similar problem but my pages DO have more than 200 words of text on them. The spider sim can see all the pages (I even have a spider map on the site) so why can google only see one?
    The newest sites in the online empire!Scotlands Culture
    Baby Names

  4. #4
    SitePoint Addict ServerUnion's Avatar
    Join Date
    Jan 2005
    Location
    madison, WI
    Posts
    225
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    From what I can tell, it takes 2 items: Time and backlinks. Try to increase backlinks with relevant phrases per content and wait. Might try to get some backlinks to the sub pages and also increase the readable content on the site.

    Good luck.

  5. #5
    SitePoint Zealot toddieg's Avatar
    Join Date
    Jan 2005
    Location
    san diego
    Posts
    186
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Time is the answer here.

    also, make sure you have a sitemap linked from every page.. once google gets the sitemap, it will most likely start indexing the rest of your site.

  6. #6
    Non-Member betterpc's Avatar
    Join Date
    Dec 2004
    Location
    EU
    Posts
    123
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Add:

    <meta name="robots" content="index, follow">

    Between <head> and </head>.

    Maybe it will help.

  7. #7
    Word Painter silver trophy Shyflower's Avatar
    Join Date
    Oct 2003
    Location
    Winona, MN USA
    Posts
    10,053
    Mentioned
    142 Post(s)
    Tagged
    2 Thread(s)
    Scotlandforvisit:

    A backlink is a link from another site to your site. For instance, if I create a link from my site to Site Point, I have added a backlink to Site Points coffer.

    Patience is a virtue in waiting for the search engines. That's for sure! However, I noticed on your site, that you have cloaked certain text elements (such as spider map). Cloaking text is a no-no for search engines. Their purpose is to find relevant content for their end users. Sites that use ploys such as cloaking text, link farming, and other elements that are there only to increase page rank, may not be fully spidered and in some instances, may even be banned and previously spidered pages pulled from the Search Engine Index.

    You also have a substantial use of javascript which search engines don't read, although I didn't find any content within your scripts. So I don't think the JS is a problem.

    In addition, your keyword density is probably too great. Anything over 7% the engines generally consider as spamming and don't index the page.

    As far as the Index follow meta tag, the spider is a robot and its job is to follow the pages within the confines of the algorithm. That particular meta tag is generally considered obsolete these days, although some smaller, local engines may still use it.
    Linda Jenkinson
    "Say what you mean. Mean what you say. But don't say it mean." ~Unknown

  8. #8
    SitePoint Enthusiast
    Join Date
    Apr 2005
    Posts
    51
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Not sure exactly what cloaking text is..can you give an example? I've been using dreamweaver to create my site. Will that have anything to do with it?

    Once I have an understanding of what cloaking is, i can figure out how to resolve it.

    Thanks for the replies. And looking forward to more advice (i.e. cloaking)

  9. #9
    Word Painter silver trophy Shyflower's Avatar
    Join Date
    Oct 2003
    Location
    Winona, MN USA
    Posts
    10,053
    Mentioned
    142 Post(s)
    Tagged
    2 Thread(s)
    Quote Originally Posted by clickagent75
    Not sure exactly what cloaking text is..can you give an example? I've been using dreamweaver to create my site. Will that have anything to do with it?

    Once I have an understanding of what cloaking is, i can figure out how to resolve it.

    Thanks for the replies. And looking forward to more advice (i.e. cloaking)
    You don't have any cloaked text in your site. What your site needs is optimized content. My reply about cloaked text was in response to Scotlandforvisit's question. If you don't know about cloaking text, it's really an area you don't want to learn about if you want your site indexed.

    What it is - ie. white text on a white background, black text on a black background, etc. It's there, but the user (the visitor you have worked so hard to get) can't see it.
    Linda Jenkinson
    "Say what you mean. Mean what you say. But don't say it mean." ~Unknown

  10. #10
    SitePoint Guru scotlandforvisit's Avatar
    Join Date
    Aug 2004
    Location
    Scotland
    Posts
    687
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Shyflower

    Thanks

    I have removed the "cloaked text" - the only reason I used it was I didn't think a real visitor would want to visit a spider map. However having it as a link in normal colours isn't really going to make much difference to the layout.
    The newest sites in the online empire!Scotlands Culture
    Baby Names

  11. #11
    Word Painter silver trophy Shyflower's Avatar
    Join Date
    Oct 2003
    Location
    Winona, MN USA
    Posts
    10,053
    Mentioned
    142 Post(s)
    Tagged
    2 Thread(s)
    Quote Originally Posted by scotlandforvisit
    Shyflower

    Thanks

    I have removed the "cloaked text" - the only reason I used it was I didn't think a real visitor would want to visit a spider map. However having it as a link in normal colours isn't really going to make much difference to the layout.
    I looked at your layout. I don't think that is the problem. What you could do is put that link in your Navigation for your site. Instead of calling iti "spider map" call it site map. That way it will be accessible to your visitors and the search engines will be happy.

    I'm not sure, but you may need to resubmit your site after you have made the change.

    I just got the current issue of Developer's Shed today. It has an excellent article on "Playing by Google's Rules". Has mostly the "don'ts" but check their archive and other articles for dos.
    Last edited by Shyflower; May 6, 2005 at 05:58.
    Linda Jenkinson
    "Say what you mean. Mean what you say. But don't say it mean." ~Unknown

  12. #12
    SitePoint Evangelist djdykes's Avatar
    Join Date
    Feb 2005
    Location
    Chester, Cheshire
    Posts
    565
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    i am having the same problem when running my site address through this spider sim, the spider is not picking up any of the links, which is worrying as they are all to my content!

    any suggestions? my site is structured

    site -> category -> content

    HELP! !

  13. #13
    Word Painter silver trophy Shyflower's Avatar
    Join Date
    Oct 2003
    Location
    Winona, MN USA
    Posts
    10,053
    Mentioned
    142 Post(s)
    Tagged
    2 Thread(s)
    A dozen people may tell me I'm wrong about this, but I think these "spider simulators" are a scam. Search engine spiders search the web according to the algorithms in their programs (at least that's what I've been told). As far as I know, none of the majors has released their algorithms to a third party.

    The best way to see if your site has been indexed and what pages of your site have been indexed is to check your stats and not rely on a third party.
    Linda Jenkinson
    "Say what you mean. Mean what you say. But don't say it mean." ~Unknown

  14. #14
    SitePoint Evangelist djdykes's Avatar
    Join Date
    Feb 2005
    Location
    Chester, Cheshire
    Posts
    565
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    ^^ fair point...

  15. #15
    SitePoint Enthusiast
    Join Date
    Apr 2005
    Posts
    51
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Update...

    So Yahoo is great at following every link on your website. It even followed parameterized links from my estore for products stored on my database (i.e. has parameterized URLs).

    However, Google still refuses to index past my first few pages. Quite slow actually to catch on that I have more than just the first page. I even doubt they would go as far ast to crawl my parameterized product URLs.

    Any helpful hints to get Google to crawl parameterized URLs for shopping carts?



    Thanks!
    Last edited by clickagent75; Mar 27, 2006 at 14:57.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •