I work with Visual Website Optimizer and I'll attempt to provide some explanations here (pardon me if I get too salesy)
The nefarious WYSIWIG editor
Just a little background, an A/B test is when you create two versions of an element and divide visitors 50/50 between them. Simple example: Red "Submit" button and green submit button, and then see which one results in more "submits".
Split URL testing = You create two versions of a page where the goal page is the same. For example, your website is www.example.com and you have a new theme at www.example.com/new so you split users 50/50 between your original site and the new theme. Then, you measure which one leads to increased sales, which you track by when a customer reaches the Thank You page (let's assume www.example.com/checkout/thank-you.php). The best part about a Split URL test? You don't need to go through a WYSIWIG editor.
About that, here's some detail: the WYSIWIG editor is there because most often, it is marketers who handle A/B testing. However, we know that some people don't like them and so have the Advanced Code Editor. Using that, you can make changes in HTML, CSS and JS without any wysiwig irritation. Also consider that when you're doing a lot of testing for many clients, you'll appreciate that you don't have to dive into the code everytime a variation has to be created.
The deal with Google Search Rankings and A/B testing
Google cleared the air about this through their blog post http://googlewebmastercentral.blogspot.in/2012/08/website-testing-google-search.html
Most important bits from that post are:
Cloaking—showing one set of content to humans, and a different set to Googlebot—is against our Webmaster Guidelines, whether you’re running a test or not. Make sure that you’re not deciding whether to serve the test, or which content variant to serve, based on user-agent. An example of this would be always serving the original content when you see the user-agent “Googlebot.” Remember that infringing our Guidelines can get your site demoted or removed from Google search results—probably not the desired outcome of your test.
If you’re running an A/B test with multiple URLs, you can use the rel=“canonical” link attribute on all of your alternate URLs to indicate that the original URL is the preferred version. We recommend using rel=“canonical” rather than a noindex meta tag because it more closely matches your intent in this situation. Let’s say you were testing variations of your homepage; you don’t want search engines to not index your homepage, you just want them to understand that all the test URLs are close duplicates or variations on the original URL and should be grouped as such, with the original URL as the canonical. Using noindex rather than rel=“canonical” in such a situation can sometimes have unexpected effects (e.g., if for some reason we choose one of the variant URLs as the canonical, the “original” URL might also get dropped from the index since it would get treated as a duplicate).
Use 302s, not 301s.
Only run the experiment as long as necessary.
The amount of time required for a reliable test will vary depending on factors like your conversion rates, and how much traffic your website gets; a good testing tool should tell you when you’ve gathered enough data to draw a reliable conclusion. Once you’ve concluded the test, you should update your site with the desired content variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup. If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if you’re serving one content variant to a large percentage of your users.
Tools of the Trade
Free tools are:
- Google Content Experiments - This is Google's free solution. Currently, it only allows Split URL testing. See http://support.google.com/analytics/bin/answer.py?hl=en&answer=1745216 on how to start a new experiment
- MaxA/B - I noticed that you do a lot of work using WP, so I think this should work really well for you. It's a free WP plugin for Split URL testing (even though it calls itself MaxA/B). Here's the link http://wordpress.org/extend/plugins/maxab/
- A/Bingo - A/Bingo is a free RoR solution most famously used by Khan Academy. Link: http://www.bingocardcreator.com/abingo
- Genetify - Free JS based A/B testing. Project home page is https://github.com/gregdingle/genetify/wiki
Paid tools abound. WhichMVT.com has a good comparison.
Why pay so much?
Because this stuff will make your clients bucket loads of money. No seriously, A/B testing tools increase the rate at which websites convert their business goals by a huge margin. You should see the Visual Website Optimizer case studies to see how simple changes have huge impact.
Also, if you'll care to, please have a look at an article I recently wrote for Webdesigntuts+ titled "A Web Designer’s Introduction to A/B Testing". It can be found at http://webdesign.tutsplus.com/articles/general/a-web-designers-introduction-to-ab-testing/
I hope this helps. Feel free to ask any questions. I'll happily answer them without any of the sales mumbo jumbo people in my line of work usually resort to.