Google is now offering a new glimpse at exactly how their spider views your Website.
Try this:
1. Login to your Google account (AdWords, Gmail or Analytics)
2. Enter your Website URL at the bottom of this page.
3. Verify your Website by creating a uniquely-named, but empty HTML file and uploading it to your server. Click on the “verify” link to view instructions on how to do this. More info about verifying your site.
4. Click the “View stats” link
So what exactly can you find out?
- The most popular queries that your Website shows up for
- The top queries from which you get clickthroughs
- Crawl statistics: Pages succesfully crawled, pages blocked by robots.txt, pages that generated HTTP errors or were unreachable
- The PageRank distribution within your site
- Various indexing stats (pages indexed, etc.)
This is an awesome troubleshooting tool for all Webmasters, and something that hopefully the other search engines will copy soon.
There isn’t anywhere at the bottom of this page to enter a URL… ?
November 16th, 2005 at 7:41 pm
Look for: “If you don’t have a Sitemap…”
November 16th, 2005 at 7:51 pm
Matt – I think you missed something, and pdxi may be right. It looks like you have to be logged into Google to get to the page you mentioned. Otherwise, you get redirected to a login page.
November 16th, 2005 at 7:58 pm
You’re right – it seems that even a GMail account will work though. I’ve updated the instructions.
November 16th, 2005 at 8:02 pm
One hang up on their clever verification process. If you use something like the Google tool “Blogger”, you can’t upload a file. :O
November 16th, 2005 at 8:13 pm
Hmmm….
For “Query stats” I received – “Data is not available at this time. Please check back later for statistics about your site.”
Guess I’ll check back – because that does not compute… ;)
Thanks!
November 16th, 2005 at 9:32 pm
Help Matt!
I’ve signed up and found my web site (http://www.lukesplace.com)
(when I go to stats and click on the first link (Indexed pages in your site) I get 183 results showing different pages of the site. Does that mean that the google search engine has indexed 183 pages on my site (Is that good?)
How do I use this utillity to its fullest potential?
November 16th, 2005 at 9:41 pm
I noticed this earlier today, but it seems that some of the things it lists, like most popular queries, don’t agree with my stats.
November 16th, 2005 at 9:48 pm
Luke those are good questions, and I’d like to know whats the best way to utilize this. I actually got it up and running in about 3 minutes. Pretty straightforward. Thanks.
November 17th, 2005 at 1:51 am
Round Two…
Ok – so know Google see something, but umm – so what, I knew that.
The “Top search queries” and “Top search query clicks” gives no raw data (e.g. number of clicks per XXX, number of queries per XXX, etc.)
The top queries could have a million queries per second and the second place one could have TWO a YEAR…
Also, under “” I’m getting “URLs timed out” even thought Google’s “Distribution” bar says 5% of the sites URLs are timing out…
Guess when they say BETA they’re not kidding…
Oh well, any data Google give out is well not data.
November 17th, 2005 at 2:08 am
I need some sleepzzz ;)
November 17th, 2005 at 2:10 am
I thought this would be cool when I read about it in the forums yesterday, but your post convinced me to sign up.
The stats are really cool, I can even preview the PageRank of my new website with this new interface, some clever people just have to find out how to interpret these new pagerank visualisations.
November 17th, 2005 at 4:52 am
http://www.icis.com
November 17th, 2005 at 6:24 am
Neat. A lot of work for me to do considering the query stats for work’s site!
November 17th, 2005 at 9:23 am
You said that this would enable us to see the Pagerank distribution? I signed up and can’t see where that data is. Help?
November 17th, 2005 at 9:33 am
This is cool! Great webmaster resource. Just tested on my site and works great ;)
Thanx Matt.
November 17th, 2005 at 10:10 am
this is incredible! thanks for showing this
November 17th, 2005 at 10:29 am
how strange… lets me log in, but then won’t let me past the CAPTCHA… says my username/password credentials are incorrect!?!
November 17th, 2005 at 10:34 am
The top query stats are from Google’s click tracking on the SERPs. The tracking scripts aren’t in place permanently, so it’s just a snapshot. The PageRank distribution is interesting, it seems it’s based on the real PageRank used in rankings, not outdated toolbar stuff. As for the crawler problem reports, I monitor them since 8/30/2005 and they are accurate (more info). Unfortunately they don’t show the source of broken links yet.
November 17th, 2005 at 10:46 am
Google Deine Nachbarschaft
Man kann über Google denken, was man will. Ist ja ein freies Land hier. Aber, dass die Jungs die geilsten nutzlosen Features haben, die es gibt, das kann man wohl mit Fug behaupten. Nach dieser Anleitung bei rikman, die er…
November 17th, 2005 at 12:23 pm
[...] read more | digg story [...]
November 17th, 2005 at 12:58 pm
[...] SitePoint Blogs » You Have to Try This on Google. Great SEO advice using Google Sitemaps. by Black Rim Glasses | posted in running Trackback URL | Comment RSS Feed Tag at del.icio.us | Incoming links [...]
November 17th, 2005 at 2:23 pm
lukeurtnowski: It depends on your site… If you know how many pages you have on your Website, you can compare it against the number that Google has indexed, and arrive at a percentage. If Google is missing a significant number of pages, you might consider submitting a full “site map” to Google. Details of how-to do this are at: https://www.google.com/webmasters/sitemaps/docs/en/submit.html
craig34: Look under stats -> Page analysis for a neat little graph.
Since the service is still in beta, you might have to revisit the page to see the numbers as I’ve noticed it occasionally goes offline.
November 17th, 2005 at 2:38 pm
[...] Matt Mickiewicz has unearthed a new feature at Google that tells you: the most popular queries that your blog shows up for, the top queries from which you get clickthroughs, crawl statistics, pages blocked by robots.txt, pages that generated errors or were unreachable and even the PageRank distribution within your site. You first need to upload an an empty HTML page to your blog’s root directory before this works. What a great hack. Google has also posted full instructions. [...]
November 17th, 2005 at 2:39 pm
See the Google Sitemaps how-to at A Consuming Experience for detailed information re: making this work.
November 17th, 2005 at 3:00 pm
[...] via Sitepoint.com weitere Artikel zu: Google, GoogleSitemap [...]
November 17th, 2005 at 3:10 pm
[...] Check it out. [...]
November 17th, 2005 at 3:12 pm
Google Sitemaps and Verification
Of course here’s where things get tricky because verification is a problem here in blogspot land!!
I’d like to be able to verify, please!!
The irony is that the official Google Sitemaps blog is a blogspot blog, & so presumably has a limited sit…
November 17th, 2005 at 3:13 pm
[...] You Have to Try This on Google te explica como saber. (link via livemarks) [...]
November 17th, 2005 at 3:21 pm
Round Three…
Well I check back and Google had deleted the information the had and replaced it again with “Data is not available at this time. Please check back later for statistics about your site.”
Also, I tried Google Analytics (e.g. Urchin Hosted for Free)
http://www.google.com/analytics/
And ran into the same issue of reports not being generated real-time nor are they archive… Pretty freaken lame even if it is “free.”
I mean the point of web services is that they are “on-demand” not when ever we get around to it. If their load is so high, then they could atleast say when the next generation will be rendered and cache the old data…
Oh well.
November 17th, 2005 at 3:34 pm
Nice post.. any help with getting listed better on google I will do.
November 17th, 2005 at 5:33 pm
Regarding Pagerank Distribution… Question is, what does Google consider High, Medium and Low pageranks? In the sitemaps interface, looks like I have a page or two showing High, but the highest PR pages (according to the toolbar & datacenter checks) I have on this relatively new site are PR3. Hope that is not considered High. Would love for these results to be somewhat forward looking and indicate I have a PR5+ showing up near term, but I’m not holding my breath.
November 17th, 2005 at 5:54 pm
[...] Matt Mickiewicz wrote about a new Google feature that gives you all kinds of neat statistics, regarding the way Google indexs your site. This feature, is actually a sub-feature of Google SiteMaps, The service that is supposed to help Google index your site. So what’s all the fuss about? Using this feature reveals the following: [...]
November 17th, 2005 at 6:14 pm
I agree – I mean google as XXX number of you pages indexed & PRzero to PR10, seems like they should just list all of the pages and their PR and have a report that summarize which fall into what PR… in mean they know you are the webmaster of the site, not a competitor.
November 17th, 2005 at 6:24 pm
By the way, WTF with all the SPAM post…
Also – this is an english site – please stop commenting in German, Hebrew, & etc… (even if I can read it…)
November 17th, 2005 at 6:28 pm
[...] I was poking my head around Digg.com today and ran accross a story saying they had added some new functionality to Google Sitemaps. Now considering I’m a Sitemaps advocate with all of my clients I had to go take a look and see what all the fuss was about. Upon verifying your site by FTPing a empty html file to your root directory you are able to access som statistics for your site. These include query stats, crawl stats, page analysis and index stats. Inside of these base options you can see how well your pages rank, what your top queries were and where google ran into problems indexing your site. Right now the information is vague but anything helps when it comes to Google in my opinion. I hope to see expanded functionality and more detailed information in the future. [...]
November 17th, 2005 at 7:53 pm
For me, the really valuable info is the top search queries compared to the top search query clicks – the two lists should be as similar as possible, I believe.
For example, for my site, ‘vodcast’ is at the top of both lists, meaning my site is listed high when searching for ‘vodcast’ and when people search for ‘vodcast’, they often click on my site. However, ‘video podcast’ is in my query clicks list but not in my search queries list, effectively meaning my site is relevant to people searching for ‘video podcasts’ but I’m not appearing high enough in Google’s results. In other words, I need to optimise my site for the term ‘video podcast’.
To summarise, terms that are in the right (clicks) list but not in the left (queries) list are terms you need to optimise your website for.
November 17th, 2005 at 9:43 pm
Very Cool!
- MountainUU
-
November 17th, 2005 at 10:57 pm
very nice
November 17th, 2005 at 11:09 pm
[...] Google Site maps visualized? [...]
November 17th, 2005 at 11:13 pm
[...] Google is now offering a new glimpse at exactly how their spider views your Website. [...]
November 18th, 2005 at 1:06 am
Excellent. Really worked well, thanks for the tip.
November 18th, 2005 at 8:36 am
[...] I need to try this, on my blogs and see what happens, when I get home tonight. Here is a quick guide to set up your site on Google Sitemap. After a quick (less than 5 minutes) setup, you’re ready to see interesting stats like what keywords your site shows up under and much more. read more | digg story [...]
November 18th, 2005 at 10:56 am
Check out google Sitemaps
This is an awesome troubleshooting tool for all Webmasters, and something that hopefully the other search engines will copy soon.
…
November 18th, 2005 at 3:57 pm
What if i have a blog that i want to check the stats for? How will i upload that Googlexxxx.html file then? Is it possible to use Google SiteMap for blogs on Blogspot? Gurus….any suggestions?
November 18th, 2005 at 5:05 pm
[...] See what Google knows about your site with Google Sitemap [...]
November 19th, 2005 at 5:59 am
[...] SitePoint Blogs » You Have to Try This on Google. (tags: design tips traffic howto google blog statistics webdev) [...]
November 19th, 2005 at 9:56 am
SS, see A Consuming Experience for lobbying blogger to ask for verification. Right now, there’s no way to make it work. They need to integrate sitemaps with blogspot more effectively….
November 20th, 2005 at 10:09 am
[...] So now you can claim sites at Google, sorta similarly to the way you can at Technorati – this is what Google thinks of This Damn Blog: [...]
November 20th, 2005 at 9:36 pm
[...] SitePoint Blogs » You Have to Try This on Google. How the Google spider reads your site content (tags: Google Development Search Productivity) [...]
November 22nd, 2005 at 3:39 pm
Tried it and it actually works. Consistent with the SERP and PR data that I have been checking these past few months.
November 23rd, 2005 at 3:16 am
One problem with verification can be your handling of ‘404′ pages. (Page not found). For all of my sites, my 404 handler is a redirect to the root page of the site, and issuance of a 200 return code. (OK)
During the verification, Google checks for the blank page, and then also checks for a non-existing page to see if you have this feature enabled.
No major problem if you have access to the server — just temporarily set it to issue normal 404 failures, get verified, and then switch it back the way it was.
December 9th, 2005 at 4:54 am
Allow me to join the clamour of voices shouting the same thing … What about Blogger … Anyone got any ideas? Can use site maps (see this link) but can’t verify so can’t get the spidery goodness of the full strength stats … Ideas?
December 12th, 2005 at 1:23 am
I learned of this through the O’Reilly newsletter. It’s too busy at the moment for me to get results right away but I’m sure the Google goodness will be nice.
December 21st, 2005 at 7:56 am
How we analize the traffic through google analytics…How will we get that code which we’ll have to put into our own site?
March 31st, 2006 at 6:21 am
PLz tell me the working process of google analytics..
March 31st, 2006 at 6:23 am
i didnt work for me :-(
also the links are down
June 24th, 2006 at 8:31 pm