User testing in Second Life

Oz-IA ran over the weekend in Sydney and it was a fantastic event.

There were some really interesting presentations, and while I didn’t get to all of them, without doubt the most interesting for me was User Research in Virtual Worlds, a presentation I highlighted earlier as being one I wanted to see.

Gary Bunker and Gabriele Hermansson, both from Hyro, spoke for close to an hour about their experiences in setting up and running an experimental focus group within Second Life.

Now that virtual worlds (such as SecondLife and World of Warcraft) are becoming vastly more popular, Hyro set out to build a research platform that would allow us to research users within those worlds, not only for their experiences there but also for their needs outside of it. We wanted to know if we could use virtual research – focus groups, interviews and user testing – in a practical way in design projects requiring complex user input. We also wanted to understand how user behavior would change between real-world and virtual forums.

Going into the presentation I was somewhat skeptical about how they might have conducted the evaluation, but by the end of the session I was really very engaged and intrigued.

The Focus group in Second Life was conducted as a trial to test its feasibility for future use, and as such a second focus group was conducted in parallel, in the physical world. This allowed them to benchmark the findings gathered in Second Life against those that were recorded during the traditional focus group session.

The Session in Second Life was run similarly to how we would conduct a focus group ‘in real life’. Participants were recruited (as they teleported in), screened and scheduled for the session just as we normally would.

During the session, participants signed a code of conduct and consent forms, and received instructions on how the session would be run. ‘Virtual Participants’ were recruited from both Australia and the UK, with the focus group being recorded with a media camera and chat logger.

Things that worked

  • There was a high level of feedback
  • Participants were comfortable
  • There was a good level of interaction with the participants
  • The focus group had an international reach, which was a requirement of the testing
  • Findings of the online session matched those recorded in the offline session

Things that didn’t work so well

  • Online sessions took about one and a half times as long (i.e. 1.5 hrs online and 1hr offline)
  • There were multiple conversation threads running at the same time, which were difficult to track
  • The response time of some participants was slow
  • It was confusing if participants weren’t identified directly by name during discussion, as it was at times difficult to indicate who the focus group facilitator was addressing

Lessons Learned

  • Fewer participants (maximum 4) would be easier to manage
  • There was a need for more facilitators than usual (minimum two)
  • More time was required than usual
  • Using coloured text instead of everyone having the same colour would be useful, particularly between facilitators and participants
  • Payment to participants must be immediate in the virtual world or people get nervous
  • It’s essential (as in the real world) that the screener isn’t leading, as it’s impossible to know what the participants are really like
  • Get the right people – open, talkative, respectful and interested
  • Facilitators must be able to type quickly, multitask and know the platform
  • Give a focused response – use the participants name when addressing them
  • Summarise the participants feedback so that they know you have heard and understood them (eye contact isn’t possible online)
  • Be clear that you’ve finished talking about a particular topic and that you’re moving on to the next one.

From my perspective, the biggest barrier to entry would be setting up the virtual space and building the focus group facilities online. As an independent consultant, that’s a massive effort that I’m unlikely to invest in. Even for smaller companies it’s a huge amount of work.

But it is interesting… *very interesting*.

It’s an example of how technology is allowing us to change the way we work. No doubt there are a lot of questions about the validity of the participants recruited for the Second Life Focus Group. But I’m a firm believer that most of what happens online can happen offline and vice versa. I’ve interviewed a few doubtful characters myself over the last few years, despite rigorous screeners. So Hyro’s report that results from the virtual focus group were very similar to that of the real life focus group is a good sign and perhaps not too much of a surprise after all.

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • Andrew Boyd

    Hi Lisa,

    I loved the presentation and love the concept. I wish that they’d had the cash for a larger trial .’. a more representative sample – but the concept and their presentation of it was world class.

    Cheers, Andrew

  • http://www.mikehealy.com.au cranial-bore

    Online sessions took about one and a half times longer

    One and a half times longer, or one and a half time as long?
    The former would be two and a half times as long.

  • Lisa Herrod

    Oh thanks cranial-bore, that occurred to me as I was writing it and I wasn’t sure… Corrected ;)

  • http://boyohazard.net Octal

    I believe Second Life now supports voice chat. Not sure if that’s of any use but it might save the poor facilitators from having to type so fast!

  • Lisa Herrod

    hi Octal,

    yes you’re right, it does support voice now. But apparently it still isn’t that great for groups of people. A bit like Skype. It’s ok for a few but once you hit 6 or more it gets a bit patchy. Also, because there were participants from both the UK and Australia, I think that reduced the audio quality as well.

  • Sheila Webber

    Hi, thanks for the post (which I just came across). You can log typed chat automatically in SL, so no transcription needed! In that respect, using Voice can be more problematic (even more difficult than usual to tell who is talking because of low quality) and you have to transcribe. A mix of people using Voice (which not everyone can/wants to do) and chat is not a good idea in a focus group as it’s difficult to keep a balance – people in Voice mode forging ahead and not seeing text comments/questions etc.

    I haven’t conducted focus groups, but I have facilitated discussions, and it definitely is useful to have a person dealing with the sort of practicalities you don’t have in RL (people suddenly disappearing, IMing you for teleport coordinates, newbies unsure how to perform certain movement/positioning/communication activities)

    I think the investment is in learning about how to function in SL yourself, plus learning how to facilitate this kind of event – not really in the “space” issues since there are now all sorts of spaces and venues that could be used for small scale discussions.

  • Jack

    Regular focus groups are run in Second Life at the iAsk Center. For more info go to http://www.iAskCenter.com
    Many of the comments in this article ring true for the many groups we have run. 4 or 5 participants are the most for a Second Life group. Proper moderation however can make these groups very productive. We have many clients who are amazed and enlightened by these groups.