The Absolute Beginner’s Guide to User Testing

This article is a companion piece to my newly launched incrementalux podcast and is based on my learning from interviewing people much smarter than I am.

In Pursuit of the Perfect Product

While user-testing has always had a place in the web development process, perhaps it’s the explosion of smartphones and internet-enabled devices that has really brought user-testing into the spotlight.

In the recent past, several design methodologies have focused on feedback from the end user. Lean UX places measuring feedback as one of its core tenets. User centered design puts the user,.. well, in the center.

Being a UX designer involves a lot of testing and gathering feedback. There are now many products in the marketplace that enable us to analyze and review our users. But, before you wear out that credit card on a raft of user testing software and services, you’ll need (at least) an absolute beginner’s understanding of the what, when, and how of user testing.

That’s what I want to tackle today!

Let’s start with the easiest question to answer.

When to Test

User testing is most effective when utilized before any effort goes into development. It helps to uncover issues on a conceptual or thinking level, before you begin coding or building. It helps to do a lot of testing before you commit to building a solution.

Test early, test often is a mantra that has been hammered into all of our heads. But, testing early alone is not sufficient.

Meaning, the moment you have anything tangible, you begin testing with a user. The first napkin sketch or whiteboard diagram is a great place to begin your testing.

Paper prototypes allow you to draw and test out an entire flow. Here the user mimics their interactions with paper sketches. You, as a designer, get to test out the entire workflow and identify gaps in mental models of your users v/s the mental model of the system you built.

This phase of testing is both rapid and iterative. As soon as you identify issues, it’s time for the next version to be quickly sketched and tested. Once you finish with throwaway prototypes like sketches and whiteboard diagrams, you can move on to more tangible wirefreames or photoshop mockups.

How to Test

The 5 second test

Goal: A simple yes/no test to establish whether an idea is viable

The quickest and easiest way to gather feedback on designs is the 5 second test. Show your design to your team members, friends or spouses for 5 seconds and ask them questions like:

• What do you think this design is about?
• What is the primary action you could take?
• What did you like about it?
• What did you dislike about it?

This is a great way of quickly gathering feedback on whether the design is “hit or a miss”. Whether the primary purpose of a screen is being conveyed instantly or not is a big deal in the attention economy, where users are bombarded with visuals, information, data and even bullshit. However, it’s very binary and does not go into finer details.

Guerilla testing

Goal: Gather feedback on a design from random people at common places

‘Get out of the building’ is a fairly common phrase heard in the design community.

Guerilla testing is a fancy name for showing up at a coffee shop and randomly selecting people and asking them (politely) for feedback on designs.

This is a great way to get feedback from a variety of sources. Perfect for customer-facing applications that require feedback from a varied demographic.

The downside to guerilla testing is when people give you irrelevant feedback so they can get free coffee, in exchange. This can be mitigated by observing the users and their behavior. Using the notes from observation in combination with their verbal responses helps understand issues with the design better.

Longitudinal Study

Goal: Gather feedback from the same person repeatedly, over a period of time

A lot of focus is given to initial impressions and metrics around user tasks like register, sign up, pay. This focus transcends what your role is in the industry.

• Product managers focus on user onboarding
• Marketers focus on landing page and conversion rates
• Developers focus on first load time or time to first byte

We have worked on it so much that we have landing page elements down to a science. But what happens beyond that?

Binge watching on Netflix

There are several use cases for testing with users who engage with an application for longer periods of time. Journal studies request users to log their experience within applications over a period of time. This way you are not only gathering feedback for “first impressions”. You are also optimizing for repeated uses.

A longitudinal study is when you gather feedback from the same person over a period of time for the same application. You’re literally journaling a customer’s lifetime within the application.

Think of how your experience differs between the first time you used a service like netflix and how you use it currently. The initial experience is all about acquainting ourselves with the interface but later on it’s all about binge watching movies. The expectations and requirements of the user changes over usage and time. So does the feedback associated with the user.

Gather feedback on feature changes

A longitudinal study is also a great way to gather feedback on changes to an existing product and how users’ perception of the app changes based on new additions and deletions to the core product.

• How does the new google logo make people feel?
• How do people respond to it as soon as it’s released?
• How About a month later?
• What happens when you remove a feature?
• How do they rate their experience after such a change?

System usability scale

Goal: A standard metric/scale to gather and measure feedback

Speaking of ratings, one of the best ways to gather feedback is to use a standard scale. The system usability scale is a great tool to gather subjective feedback.

• How does the user feel about the design?
• Did they find it easy?
• Did they feel confident accomplishing their tasks?
• Would they recommend it to a friend?

Using a standard scale allows for a good reference point and say X% is the benchmark. If people rate it above X%, it means the change remains, whereas negative responses lower than X% would mean, we will kill this change.

What to Test

Usually, you test users on tasks that are core to the website’s functionality. For e-commerce websites, that’d be discovering and purchasing products. Some of the tasks could be like:

• Search for a jacket in your favorite color and purchase it
• Find trousers from your favorite brand
• Complete a checkout for 3 items you find interesting

How do you qualitatively measure the feedback on these items? Some of the standard metrics used are:

• Time on task – How long did it take the user to complete the task?
• Success on task – How often were users able to successfully complete the tasks?
• Failure rate – How often did they fail?

Last words

User testing is a powerful method to test the effectiveness of your screens. It is usually recommended that you use a combination of qualitative and quantitative techniques to measure user feedback.

P.S. If you are interested in learning more about user testing and user research, you should check out this episode on incrementalux podcast.