Madlib-style forms increase conversion by 30%? Well, maybe …

A while back, while writing about Google’s Social Graph API, I mentioned Jeremy Keith’s audio-sharing web app, Huffduffer, and its unconventional signup form. If you haven’t seen it, here’s what it looks like:

As you can see, unlike the traditional web signup form, it’s asking you to fill in the blanks of a short paragraph or sentence with the required information. At the time it was released, it was seen more as a novelty and a curiosity than an innovation in web form design, but recently an A/B test of a similar form against a more traditional form by the team at Vast.com, as blogged by Luke Wroblewski, should make you think twice about it. The test apparently showed a 25-40% increase in conversion from the storytelling forms as opposed to the traditional, stacked field style form. You can see an example of the new madlib form here on Vast.com.

Blogging about the results, Jeremy Keith says:

That seems to be a statistically-significant result, even accounting for Cennydd’s reality-check on A/B testing.

The link is to a fantastic piece by Cennyd Bowles on the fact that basic statistics are so often overlooked when conducting A/B tests. A quick excerpt from that article (though I strongly recommend you read the whole thing):

Last week I tossed a coin a hundred times. 49 heads. Then I changed into a red t-shirt and tossed the same coin another hundred times. 51 heads. From this, I conclude that wearing a red shirt gives a 4.1% increase in conversion in throwing heads.

This example is only partly in jest: many A/B tests make similar conclusions from similar sample sizes. What I don’t see, however, is what Keith sees as this being “accounted for”: Luke’s post makes no mention of sample size, or the duration over which the test was run, or whether the number stated is an absolute percentage or a relative one.

So, for the moment, what can we take from this? It’s not certain that this kind of form will work wonders for every kind of site, and it’s also not certain whether or not its performance might be due solely to its novelty (in which case the advantage would dwindle as more sites began employing similar forms). On the other hand, looking at the side-by-side screenshots in Luke’s post, the new-style form does feel more personable and appealing, less cold and machine-like, so maybe there’s something to it.

What do you think? Will we start seeing an abundance of story-style forms on the Web? Outside of signup forms, what else can you see them being used for?