The Absolute Beginner’s Guide to User Testing

Share this article

This article is a companion piece to my newly launched incrementalux podcast
and is based on my learning from interviewing people much smarter than I am.

In Pursuit of the Perfect Product

Mobile devices While user-testing has always had a place in the web development process, perhaps it’s the explosion of smartphones and internet-enabled devices that has really brought user-testing into the spotlight. In the recent past, several design methodologies have focused on feedback from the end user. Lean UX places measuring feedback as one of its core tenets. User centered design puts the user,.. well, in the center. Being a UX designer involves a lot of testing and gathering feedback. There are now many products in the marketplace that enable us to analyze and review our users. But, before you wear out that credit card on a raft of user testing software and services, you’ll need (at least) an absolute beginner’s understanding of the what, when, and how of user testing. That’s what I want to tackle today! Let’s start with the easiest question to answer.

When to Test

User testing is most effective when utilized before any effort goes into development. It helps to uncover issues on a conceptual or thinking level, before you begin coding or building. It helps to do a lot of testing before you commit to building a solution. Test early, test often is a mantra that has been hammered into all of our heads. But, testing early alone is not sufficient.
User testing iconic graphic
Source: Usertesting
Meaning, the moment you have anything tangible, you begin testing with a user. The first napkin sketch or whiteboard diagram is a great place to begin your testing.
Sketches
Source: Zurb
Paper prototypes allow you to draw and test out an entire flow. Here the user mimics their interactions with paper sketches. You, as a designer, get to test out the entire workflow and identify gaps in mental models of your users v/s the mental model of the system you built. This phase of testing is both rapid and iterative. As soon as you identify issues, it’s time for the next version to be quickly sketched and tested. Once you finish with throwaway prototypes like sketches and whiteboard diagrams, you can move on to more tangible wirefreames or photoshop mockups.

How to Test

The 5 second test

Goal
: A simple yes/no test to establish whether an idea is viable The quickest and easiest way to gather feedback on designs is the 5 second test. Show your design to your team members, friends or spouses for 5 seconds and ask them questions like:
  • What do you think this design is about?
  • What is the primary action you could take?
  • What did you like about it?
  • What did you dislike about it?
This is a great way of quickly gathering feedback on whether the design is “hit or a miss”. Whether the primary purpose of a screen is being conveyed instantly or not is a big deal in the attention economy, where users are bombarded with visuals, information, data and even bullshit. However, it’s very binary and does not go into finer details.

Guerilla testing

Goal: Gather feedback on a design from random people at common places
‘Get out of the building’ is a fairly common phrase heard in the design community.
Gorilla Guerilla testing is a fancy name for showing up at a coffee shop and randomly selecting people and asking them (politely) for feedback on designs. This is a great way to get feedback from a variety of sources. Perfect for customer-facing applications that require feedback from a varied demographic. The downside to guerilla testing is when people give you irrelevant feedback so they can get free coffee, in exchange. This can be mitigated by observing the users and their behavior. Using the notes from observation in combination with their verbal responses helps understand issues with the design better.

Longitudinal Study

Goal: Gather feedback from the same person repeatedly, over a period of time
A lot of focus is given to initial impressions and metrics around user tasks like register, sign up, pay. This focus transcends what your role is in the industry.
  • Product managers focus on user onboarding
  • Marketers focus on landing page and conversion rates
  • Developers focus on first load time or time to first byte
We have worked on it so much that we have landing page elements down to a science. But what happens beyond that?
Ordering priorities on a typical page layout
Ordering priorities: Source: Formstack

Binge watching on Netflix

There are several use cases for testing with users who engage with an application for longer periods of time. Journal studies request users to log their experience within applications over a period of time. This way you are not only gathering feedback for “first impressions”. You are also optimizing for repeated uses. A longitudinal study is when you gather feedback from the same person over a period of time for the same application. You’re literally journaling a customer’s lifetime within the application. Netflix logo Think of how your experience differs between the first time you used a service like netflix and how you use it currently. The initial experience is all about acquainting ourselves with the interface but later on it’s all about binge watching movies. The expectations and requirements of the user changes over usage and time. So does the feedback associated with the user.

Gather feedback on feature changes

A longitudinal study is also a great way to gather feedback on changes to an existing product and how users’ perception of the app changes based on new additions and deletions to the core product.
Google logo screen cap
Google’s new logo
  • How does the new google logo make people feel?
  • How do people respond to it as soon as it’s released?
  • How About a month later?
  • What happens when you remove a feature?
  • How do they rate their experience after such a change?

System usability scale

Goal: A standard metric/scale to gather and measure feedback Speaking of ratings, one of the best ways to gather feedback is to use a standard scale. The system usability scale is a great tool to gather subjective feedback. A set of typical user test questions
  • How does the user feel about the design?
  • Did they find it easy?
  • Did they feel confident accomplishing their tasks?
  • Would they recommend it to a friend?
Using a standard scale allows for a good reference point and say X% is the benchmark. If people rate it above X%, it means the change remains, whereas negative responses lower than X% would mean, we will kill this change.

What to Test

Usually, you test users on tasks that are core to the website’s functionality. For e-commerce websites, that’d be discovering and purchasing products. Some of the tasks could be like:
  • Search for a jacket in your favorite color and purchase it
  • Find trousers from your favorite brand
  • Complete a checkout for 3 items you find interesting
How do you qualitatively measure the feedback on these items? Some of the standard metrics used are:
  • Time on task – How long did it take the user to complete the task?
  • Success on task – How often were users able to successfully complete the tasks?
  • Failure rate – How often did they fail?

Last words

User testing is a powerful method to test the effectiveness of your screens. It is usually recommended that you use a combination of qualitative and quantitative techniques to measure user feedback. P.S. If you are interested in learning more about user testing and user research, you should check out this episode on incrementalux podcast.

Frequently Asked Questions (FAQs) about User Testing

What is the importance of user testing in product development?

User testing is a critical aspect of product development. It provides insights into how users interact with a product, what they like or dislike about it, and how it can be improved. User testing helps to identify usability issues that may not be apparent to the developers or designers. It also helps to ensure that the product meets the needs and expectations of the users, thereby increasing user satisfaction and product success.

How often should user testing be conducted?

The frequency of user testing depends on the nature of the product and the stage of development. However, it is generally recommended to conduct user testing at various stages of product development, from the initial concept stage to the final product stage. This helps to identify and address usability issues at an early stage, thereby reducing the cost and time of product development.

What are some common methods of user testing?

There are several methods of user testing, including usability testing, A/B testing, surveys, interviews, and focus groups. Usability testing involves observing users as they interact with the product, while A/B testing involves comparing two versions of a product to see which one performs better. Surveys and interviews are used to gather feedback from users, and focus groups involve a group of users discussing their experiences with the product.

How can I recruit participants for user testing?

Participants for user testing can be recruited in several ways. You can use your existing user base, recruit through social media or online forums, or use a user testing service that provides access to a pool of participants. It’s important to select participants who represent your target user group to ensure that the results of the testing are relevant to your product.

How can I analyze the results of user testing?

The results of user testing can be analyzed in several ways. You can use qualitative methods, such as observing user behavior and listening to user feedback, or quantitative methods, such as analyzing metrics like task completion rates and error rates. It’s important to analyze the results in the context of your product goals and user needs to identify actionable insights.

What are some common challenges in user testing and how can they be addressed?

Some common challenges in user testing include recruiting participants, managing the testing process, and analyzing the results. These challenges can be addressed by using a structured approach to user testing, using tools and services that facilitate user testing, and having a clear understanding of your product goals and user needs.

How can user testing improve the user experience?

User testing can improve the user experience by identifying usability issues and user needs that may not be apparent to the developers or designers. By addressing these issues and needs, you can make your product more user-friendly, intuitive, and satisfying to use.

Can user testing be conducted remotely?

Yes, user testing can be conducted remotely. This can be done through video conferencing, screen sharing, or using a remote user testing service. Remote user testing can be a cost-effective and convenient way to conduct user testing, especially when the participants are located in different geographical locations.

What is the role of a moderator in user testing?

A moderator plays a crucial role in user testing. They guide the participants through the testing process, ask questions to gather feedback, and ensure that the testing is conducted in a structured and unbiased manner. A good moderator can help to obtain valuable insights from the user testing.

How can I ensure the validity of user testing results?

The validity of user testing results can be ensured by using a representative sample of users, conducting the testing in a controlled and unbiased manner, and analyzing the results in the context of your product goals and user needs. It’s also important to conduct multiple rounds of testing to confirm the findings.

Vinay RaghuVinay Raghu
View Author

Vinay Raghu is a UI/UX developer passionate about experimenting with the latest frontend technologies. He is an avid open source contributor and loves meetup groups. You can learn more about him at viii.in.

AlexWux
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week
Loading form