We made completely new website almost 1,5 month ago (www.medemis.ee), with even new URL struckture. But still our new pages are not indexed (only one www.medemis.ee/fin/palvelut/laser-ja-fotoepilaatio).
First, there was redirect problem that didn’t allow Google to crawl any pages. Spider was sent from every page to kampaania.medemis.ee page and wasn’t able to crawl our site (it is redirect we need to have). Now we have another solution: First time client appears on the website (www.medemis.ee), a cookie will be set and they will be directed to subdomain campaign page (kampaania.medemis.ee). Should they return to the original website and the cookie is still there, the client is not redirected anymore. This is done through Javascript, instead of PHP, thus hopefully allowing spider engines to index the original information without problems of redirection.
This was fixed before last Google crawl. At least cache:www.medemis.ee shows that last crawl was 30.10. But few days ago there was another pages indexed from new URL struckture http://medemis.ee/est/teenused/ilusystid/higistamisvastane-ravi/ and it was first on Google searched with keyword „higistamisvastane ravi“. Now it’s completely gone. I tried to test crawling with http://tools.seobook.com/general/spider-test/ and everything worked then with Xenu everything seem to work, SEO browser showd right page but cache got still kampaania.medemis.ee not www.medemis.ee. Even Yahoo/Bing got all pages indexed what we need. I eve tried to test some penalty possibilities but no new messages in Google Wembaster Tools and this only page what is indexed, ranks well. I have also added sitemap to Google Webmaster Tools and so on. Can anybody even quess what could be wrong?
Or is there more possibilities to get more Google to crawl your site more often. I set WT crawl rate faster, site has been on PR5+ sites and go some quite good backlinks.