Firefox Plans Built-In Usability Platform

Though still in the concept stage, Mozilla is planning an interesting new platform called Test Pilot that will allow every Mozilla Labs project to push out usability tests directly to participating users. The platform will begin life as a Firefox extension with the goal of gathering a 1% representative sample of Firefox users — which is a tall order given the Firefox has many millions of them.

Mozilla Labs’ Head of User Experience Aza Raskin talked about Test Pilot this week in a blog post. “Its not just Firefox that needs a usability lab. Thunderbird needs one. Seamonkey needs one. Every Mozilla Labs project needs one. Test Pilot is a platform — starting as a Firefox extension — on top of which anything can be put through usability-testing boot-camp,” he wrote. That’s necessary, says Raskin, because currently Mozilla only has access to scattershot usability research collected via surveys, anecdotes, user feedback, etc.

Test Pilot would allow Mozilla to collect feedback from real users, in actual use scenarios and Raskin sees it as part of a massively scalable, open usability lab being built at Mozilla Labs. Test Pilot won’t constantly record data, says Raskin, but only just enough information to answer the question posed by the researcher. Raskin outlined how the software will work from the user end:

The first time the Test Pilot add-on is run, it will ask a few simple non-personally-identifiable questions to put you into a demographic bucket, e.g. technical level, locale, etc., and to let you opt in to additional anonymous instrumentation.

We’ll only collect aggregate anonymized data, publish all results under open-content licenses, and review every test to make sure your privacy is held sacred. Once in a while you may be asked to participate in a short survey based on your demographic. If you’ve opted into allowing additional anonymous instrumentation, an experiment may request some of that information for aggregated study.

Though it will begin with Firefox, eventually the plan is to have Test Pilot baked into every Mozilla Labs project. Mozilla plans to offer open access to test results, and will allow anyone to create research tests and run them once they pass review.

Open source blogger Matt Asay thins that Test Pilot will offer a major competitive advantage to Mozilla. “If Mozilla can successfully gather input from nontechnical, average users, it will have secured the holy grail of computing: deep customer feedback,” he writes. “What will Mozilla do with that feedback? Why, conquer the world, of course.”

But we have a few of important questions about Test Pilot that Raskin’s blog post didn’t answer. First: when is it coming? Test pilot was actually announced back in March 2008, and at the time the first version was promised to ship within a “few weeks.” Clearly, that didn’t happen if it is still in the concept stage 10 months later. But the fact that Raskin is talking about it again seems to indicate that a release of Test Pilot is either near, or it is at the very least still under active development.

Second: How many people will actually use it? That’s a big if in terms of the usefulness of Test Pilot. Even if the platform is well built and succeeds at capturing meaningful feedback data, it won’t be worth anything if no one uses it. Can Mozilla really convince 1% of its massive install base to join their user feedback panel?

Third: Will Test Pilot be made available to non-Mozilla developers? Because of the open source nature of Mozilla’s software, it seems like a safe bet that the Test Pilot platform will be open to other open source developers. But nothing specifically was mentioned about that in Raskin’s post.

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • http://freebsdhowto.com/ cipixul

    I am gonna use it for sure and considering the last performance with “as many downloads as possible in one day” they are gonna make it in my opinion :)

  • http://www.tyssendesign.com.au Tyssen

    I’ll probably give it a go too.

  • Steve Williams

    Great to see usability being taken seriously. I hope that people do not use the plug in as a pass or fail on usability and perform actual usability test on their websites.

  • http://www.studio-gecko.com/ XLCowBoy

    I think I would go as far as installing it into my client’s machines. After a good explanation, I don’t think they would mind at all.

  • http://manwithnoblog.com tuna

    I echo Steve’s concern, having a tool to assist with the usability testing process does not automatically mean that you really know what you are testing and that you understand what the subsequent results of the testing mean. Nice to have but doesn’t replace the bulk of the usability work that should be done.