My site is currently being developed and has a holding page up to catch people emails, so I can notify them when the site is up for beta testing.
However, my main audience for the beta testing are people who would be searching on the “song meanings” search term through google.
What I would like to do is let google crawl my site so it gets listed in the search results, but prevent other visitors from seeing the full site at the moment, presenting them with the holding page still.
Technically you can find a way, but you’re not allowed. Serving different content to Google (the real pages) than to visitors (the holding page) is against their webmaster guidelines. It can get you banned from the index. You’re either open to the public or not. Plus, if you allowed Google to index the site, then anyone can view the pages using the “Cached” link under the search result. Your holding page won’t actually block them.
They can also access it simply by changing the useragent in their browser to be identical to that used by Google. That’s really easy to do in Internet Explorer and Firefox which allow the useragent to be set to anything.
Even easier… when you go to the website, click the cached result from Google’s index, it’ll take you to a simplified version of the page with all the information in-tact with absolutely no way for the website to prevent you from accessing that indexed content externally. It’s what I do if I find something I need in Experts Exchange (to which I’ve no idea why Google have not banned them for what is a clear violation of their usage policy). Essentially namtax, what’s been said above is correct, you can’t have it both ways… if search engines can index it, they can cache it, therefore anyone searching can view it (without your site even being involved).
Dan had already mentioned that. I was just pointing out that setting up a way for Google to index the password protected pages means that you are providing a way for anyone to access those pages without a password even if Google hasn’t indexed the pages yet.
It’s definitely a case of opening the floodgates, though I suppose the less tech inclined people may overlook the ability to bypass the protection in that way
Oh yeah, have seen that first click free thing, I may use that…but I was more looking towards blocking out all content to users, but suppose this is a good option considering the advice I have recieved above.
Ok, it looks like im going to have to open up the site to everyone then, probably better in the long run anyway…
