help getting indexed/bots not seeing links

[fluff]

hi, my site has over 40,000 pages, but google only sees like 84. basically the recipes listed (its a recipes site) on the front page have been indexed. now theres catagory links on the right side of the main page, and everything links to each other at the bottom of those main catagory pages. but those catagory pages arent being seen by the bots.
site is:
sharedrecipes dot org
grrr this place wont let me post the link cause im new to posting here...

i was trying to make a sitemap by a couple different generators, but apparently they only are seeing what the google bot sees. any help? cause making a sitemap by hand with 40,000 pages would take months i would think...is there missing coding or something?

[Mittineague]

Hi fluff, welcomer to the forums (posting-wise).

The only thing I spotted that's messed up is the bloglines link. The robots meta looks OK, the links don't have nofollow, or "messy" GET variables, and the robots.txt isn't disallowing them. If it was just Google, I'd ask how long the site has been up, as it takes time. But if the XML sitemap generators aren't finding the pages either, I can't see why not.

The only thing I can think of is maybe the class attribute of the link tags? That is, if the bots grab attribute[0] assuming it's href, but it's class instead, that could cause a problem.

[fluff]

thanks for answering, but alot of this stuff is over my head, this would be the first sitemap ive even made for one of my sites.
the site was made in september so it is pretty new, but still the sitemap generators also had problems recognizing there were more links as well. guess i will have to keep hunting for the answers.

[Mittineague]

September is kind of new for Google. So that could be just a matter of waiting. As for the sitemap generators, look for a "depth" (or something similar) option. If it was set for 1, then it would only find links on the first page, not links on those pages to other pages not found on the first page.

[fluff]

thanks, i messed around with the generator options till it finally went through and it seemed to have indexed most pages, i think restrictions on the generator site itself stopped from the full index, but almost all of it seemed to of went through. submitted to google and seeing how i did, no doubt its wrong but still i finally got it to work lol.
thanks again

[Mittineague]

Of the various tags

<url>
  <loc>http://www.mittineague.com/dev/co.php</loc> 
  <lastmod>2008-11-11</lastmod> 
  <changefreq>monthly</changefreq> 
  <priority>0.8</priority> 
</url>

as long as the 'loc' is good, it should help Google out immensely. The others are more of a "suggestion" anyway, giving "hints", that may or may not influence the crawl rate, so you can fine tune them when you get the chance.

[TheAtHomeCouple]

What platform is your site built on? Wordpress by chance?

The only thing I can suggest to get some of the other posts indexed is to start submitting your recipes to do-follow social bookmarking directories for some link juice and traffic. Visit SocialMarker.com to do this quickly to multiple sites...

Submitting an article to EzineArticles.com usually helps with quick indexing...

Be sure that every single page you have on your links to another page within your site, so the spiders always have somewhere to go... A lot of people over look this, and it's very important. Try and link within the content itself, or using a "related posts" type of plugin...

Hope this helps - not sure what the problem is!

[fluff]

thx i will try that. my forum seems to index just fine, bots are always on it. my main site has over 40,000 pages, and everything links to each other in some way. when a catagory is clicked there are links at the bottom of the page to the next dozen recipes, each recipe having its own link on the page.
next pages links are only seen as numbers (click on the number 2 which is the next set of recipes, 3 for next and so on)

seems that the catagory pages end in .html
http(grrrr wont let me post link)://sharedrecipes.org/Main_Dish.html

but clicking into the next page and so on it doesnt list as .html anymore
http(cant post my links)://sharedrecipes.org/Main%20Dish/page93/

also it seems that the only page that has www on it is the main page...

when i search on google for indexed pages on my site using www only like 150 links show, when i do it without www 2200 or so show up (which is basically all that i was able to set in the sitemap i so badly created)

does any of this affect the way bots see or wont see or index the pages?

i should probably add:
when looking up info on my site only a copycat website shows up
(cantpostlinks)google.com/search?q=infowww).sharedrecipes.org&hl=en