Oz-IA ran over the weekend in Sydney and it was a fantastic event.
There were some really interesting presentations, and while I didn’t get to all of them, without doubt the most interesting for me was User Research in Virtual Worlds, a presentation I highlighted earlier as being one I wanted to see.
Gary Bunker and Gabriele Hermansson, both from Hyro, spoke for close to an hour about their experiences in setting up and running an experimental focus group within Second Life.
Now that virtual worlds (such as SecondLife and World of Warcraft) are becoming vastly more popular, Hyro set out to build a research platform that would allow us to research users within those worlds, not only for their experiences there but also for their needs outside of it. We wanted to know if we could use virtual research – focus groups, interviews and user testing – in a practical way in design projects requiring complex user input. We also wanted to understand how user behavior would change between real-world and virtual forums.
Going into the presentation I was somewhat skeptical about how they might have conducted the evaluation, but by the end of the session I was really very engaged and intrigued.
The Focus group in Second Life was conducted as a trial to test its feasibility for future use, and as such a second focus group was conducted in parallel, in the physical world. This allowed them to benchmark the findings gathered in Second Life against those that were recorded during the traditional focus group session.
The Session in Second Life was run similarly to how we would conduct a focus group ‘in real life’. Participants were recruited (as they teleported in), screened and scheduled for the session just as we normally would.
During the session, participants signed a code of conduct and consent forms, and received instructions on how the session would be run. ‘Virtual Participants’ were recruited from both Australia and the UK, with the focus group being recorded with a media camera and chat logger.
Things that worked
- There was a high level of feedback
- Participants were comfortable
- There was a good level of interaction with the participants
- The focus group had an international reach, which was a requirement of the testing
- Findings of the online session matched those recorded in the offline session
Things that didn’t work so well
- Online sessions took about one and a half times as long (i.e. 1.5 hrs online and 1hr offline)
- There were multiple conversation threads running at the same time, which were difficult to track
- The response time of some participants was slow
- It was confusing if participants weren’t identified directly by name during discussion, as it was at times difficult to indicate who the focus group facilitator was addressing
Lessons Learned
- Fewer participants (maximum 4) would be easier to manage
- There was a need for more facilitators than usual (minimum two)
- More time was required than usual
- Using coloured text instead of everyone having the same colour would be useful, particularly between facilitators and participants
- Payment to participants must be immediate in the virtual world or people get nervous
- It’s essential (as in the real world) that the screener isn’t leading, as it’s impossible to know what the participants are really like
- Get the right people – open, talkative, respectful and interested
- Facilitators must be able to type quickly, multitask and know the platform
- Give a focused response – use the participants name when addressing them
- Summarise the participants feedback so that they know you have heard and understood them (eye contact isn’t possible online)
- Be clear that you’ve finished talking about a particular topic and that you’re moving on to the next one.
From my perspective, the biggest barrier to entry would be setting up the virtual space and building the focus group facilities online. As an independent consultant, that’s a massive effort that I’m unlikely to invest in. Even for smaller companies it’s a huge amount of work.
But it is interesting… *very interesting*.
It’s an example of how technology is allowing us to change the way we work. No doubt there are a lot of questions about the validity of the participants recruited for the Second Life Focus Group. But I’m a firm believer that most of what happens online can happen offline and vice versa. I’ve interviewed a few doubtful characters myself over the last few years, despite rigorous screeners. So Hyro’s report that results from the virtual focus group were very similar to that of the real life focus group is a good sign and perhaps not too much of a surprise after all.
Lisa Herrod
