You Have to Try This on Google.
Google is now offering a new glimpse at exactly how their spider views your Website.
1. Login to your Google account (AdWords, Gmail or Analytics)
2. Enter your Website URL at the bottom of this page.
3. Verify your Website by creating a uniquely-named, but empty HTML file and uploading it to your server. Click on the “verify” link to view instructions on how to do this. More info about verifying your site.
4. Click the “View stats” link
So what exactly can you find out?
- The most popular queries that your Website shows up for
- The top queries from which you get clickthroughs
- Crawl statistics: Pages succesfully crawled, pages blocked by robots.txt, pages that generated HTTP errors or were unreachable
- The PageRank distribution within your site
- Various indexing stats (pages indexed, etc.)
This is an awesome troubleshooting tool for all Webmasters, and something that hopefully the other search engines will copy soon.