SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    SitePoint Member
    Join Date
    Jul 2008
    Location
    massachusetts
    Posts
    4
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    help getting indexed/bots not seeing links

    hi, my site has over 40,000 pages, but google only sees like 84. basically the recipes listed (its a recipes site) on the front page have been indexed. now theres catagory links on the right side of the main page, and everything links to each other at the bottom of those main catagory pages. but those catagory pages arent being seen by the bots.
    site is:
    sharedrecipes dot org
    grrr this place wont let me post the link cause im new to posting here...

    i was trying to make a sitemap by a couple different generators, but apparently they only are seeing what the google bot sees. any help? cause making a sitemap by hand with 40,000 pages would take months i would think...is there missing coding or something?

  2. #2
    Programming Team silver trophybronze trophy
    Mittineague's Avatar
    Join Date
    Jul 2005
    Location
    West Springfield, Massachusetts
    Posts
    17,290
    Mentioned
    198 Post(s)
    Tagged
    3 Thread(s)
    Hi fluff, welcomer to the forums (posting-wise).

    The only thing I spotted that's messed up is the bloglines link. The robots meta looks OK, the links don't have nofollow, or "messy" GET variables, and the robots.txt isn't disallowing them. If it was just Google, I'd ask how long the site has been up, as it takes time. But if the XML sitemap generators aren't finding the pages either, I can't see why not.

    The only thing I can think of is maybe the class attribute of the link tags? That is, if the bots grab attribute[0] assuming it's href, but it's class instead, that could cause a problem.

  3. #3
    SitePoint Member
    Join Date
    Jul 2008
    Location
    massachusetts
    Posts
    4
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    thanks for answering, but alot of this stuff is over my head, this would be the first sitemap ive even made for one of my sites.
    the site was made in september so it is pretty new, but still the sitemap generators also had problems recognizing there were more links as well. guess i will have to keep hunting for the answers.

  4. #4
    Programming Team silver trophybronze trophy
    Mittineague's Avatar
    Join Date
    Jul 2005
    Location
    West Springfield, Massachusetts
    Posts
    17,290
    Mentioned
    198 Post(s)
    Tagged
    3 Thread(s)
    September is kind of new for Google. So that could be just a matter of waiting. As for the sitemap generators, look for a "depth" (or something similar) option. If it was set for 1, then it would only find links on the first page, not links on those pages to other pages not found on the first page.

  5. #5
    SitePoint Member
    Join Date
    Jul 2008
    Location
    massachusetts
    Posts
    4
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    thanks, i messed around with the generator options till it finally went through and it seemed to have indexed most pages, i think restrictions on the generator site itself stopped from the full index, but almost all of it seemed to of went through. submitted to google and seeing how i did, no doubt its wrong but still i finally got it to work lol.
    thanks again

  6. #6
    Programming Team silver trophybronze trophy
    Mittineague's Avatar
    Join Date
    Jul 2005
    Location
    West Springfield, Massachusetts
    Posts
    17,290
    Mentioned
    198 Post(s)
    Tagged
    3 Thread(s)
    Of the various tags
    Code XML:
    <url>
      <loc>http://www.mittineague.com/dev/co.php</loc> 
      <lastmod>2008-11-11</lastmod> 
      <changefreq>monthly</changefreq> 
      <priority>0.8</priority> 
    </url>
    as long as the 'loc' is good, it should help Google out immensely. The others are more of a "suggestion" anyway, giving "hints", that may or may not influence the crawl rate, so you can fine tune them when you get the chance.

  7. #7
    SitePoint Member TheAtHomeCouple's Avatar
    Join Date
    Sep 2008
    Location
    Toronto
    Posts
    6
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    What platform is your site built on? Wordpress by chance?

    The only thing I can suggest to get some of the other posts indexed is to start submitting your recipes to do-follow social bookmarking directories for some link juice and traffic. Visit SocialMarker.com to do this quickly to multiple sites...

    Submitting an article to EzineArticles.com usually helps with quick indexing...

    Be sure that every single page you have on your links to another page within your site, so the spiders always have somewhere to go... A lot of people over look this, and it's very important. Try and link within the content itself, or using a "related posts" type of plugin...

    Hope this helps - not sure what the problem is!

  8. #8
    SitePoint Member
    Join Date
    Jul 2008
    Location
    massachusetts
    Posts
    4
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    thx i will try that. my forum seems to index just fine, bots are always on it. my main site has over 40,000 pages, and everything links to each other in some way. when a catagory is clicked there are links at the bottom of the page to the next dozen recipes, each recipe having its own link on the page.
    next pages links are only seen as numbers (click on the number 2 which is the next set of recipes, 3 for next and so on)

    seems that the catagory pages end in .html
    http(grrrr wont let me post link)://sharedrecipes.org/Main_Dish.html

    but clicking into the next page and so on it doesnt list as .html anymore
    http(cant post my links)://sharedrecipes.org/Main%20Dish/page93/

    also it seems that the only page that has www on it is the main page...

    when i search on google for indexed pages on my site using www only like 150 links show, when i do it without www 2200 or so show up (which is basically all that i was able to set in the sitemap i so badly created)

    does any of this affect the way bots see or wont see or index the pages?

    i should probably add:
    when looking up info on my site only a copycat website shows up
    (cantpostlinks)google.com/search?q=infowww).sharedrecipes.org&hl=en
    Last edited by fluff; Dec 3, 2008 at 12:49.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •