Responsive Web Design: Real User Testing

Russ Weakley
Tweet

Have you ever tried testing your Responsive Website with real users? If not, it is definitely worth considering. You may be surprised by the results.

User Testing vs Technical Testing

Before we begin, some clarification about different types of testing.

This article will not focus on technical testing, where the aim is to determine how the site performs on a wide range of devices, or in different circumstances such as with different bandwidths.

Instead, we are going to focus on user testing. The aim of this type of testing is to determine whether real users are able to successfully use the product as required. In this case, the product is your website or web application.

You can conduct user-testing to determine if users are able to navigate around your site, search for content, fill in a form, purchase a product or perform any other site-related tasks.

Why test your site with real users?

All of us have our own opinions about how products should work. When we test them ourselves, no matter how experienced we are, our views are always biased to some degree.

Only actual users can really tell you whether your product meets their needs.

For this reason, it is important to test any product as early as possible in the process and as frequently as possible throughout the process.

Task-based Testing

Task-based testing is a very powerful user testing process. Users are asked to perform a series of tasks while interacting with the actual product. Each task is monitored and scored to determine if the user is able to successfully complete the task or not.

If users are not able to complete a task, the product may have a potential problem that needs to be addressed.

The actual tasks are determined by the required outcomes. If you want to test the purchasing process within an online shopping website, then tasks could include finding a specific product, adding the product to a cart, and completing the purchase process.

The best results come from testing a small set of users (around 5 users) and running as many small tests as you can afford.

Controlled Testing or Free Exploration

The testing process can be conducted in a variety of ways.

At one end of the spectrum, the entire process is tightly controlled. Users are not allowed to interact with the product until they are given instructions by the tester. This means that you are able to receive detailed feedback from the users at each point in the process before moving on. Users can be asked about their expectations before they click to a new “screen” or a new function.

At the other end of the spectrum, users can be allowed to explore the product as they wish. They are prompted to provide feedback as they interact with the product.

Each method has strengths and weaknesses. The “controlled testing” method allows you to closely observe each interaction. The “free exploration” method allows you to observe users performing in a more natural manner.

Using Different Media

User testing can be undertaken at any stage in the planning, design or development phases, and can employ different media:

  • paper-based wireframes
  • onscreen wireframes
  • onscreen prototypes
  • fully functional website

Before Testing Responsive Websites

Before jumping into user testing with responsive websites, it’s important to know who your current and potential future users may be, and how they are likely to access the website. For example, are they likely to visit the site on desktop and mobile? Mobile only? Mobile and tablet?

Each of these options can affect how you choose to conduct the user testing. For example, if users frequently use both dektop and mobile, it may be worth conducting user testing on wide screen and mobile as part of the process. If users frequently use desktop only or mobile only, you may want to run user testing with two separate groups.

Responsive Website user-testing challenges

Responsive websites present challenges that are quite different to those faced by traditional wide screen user testing. For example:

  • Users may be disconcerted that the mobile version of the site looks different to the wide screen version
  • Users may not be familiar with small screen interactions such as scrolling, swiping etc
  • Users may be familiar with specific mobile apps and expect your website to operate in a similar way.
  • Users may be familiar with specific platforms (iPhone or Android) and this may affect their expectations.
  • Users may not be familiar with linearized content, which is common for responsive websites when displayed on small screens.

Some Lessons Learned

1. Make the user feel “at home”

When conducting responsive website user testing, it is important to provide devices that are familiar to the user. We try to provide a range of different devices, such as iPhone, Android and a small set of tablet devices. In some cases, we also allow users to perform the testing process on their own device, if they prefer. This ensures they aren’t struggling with the device at the same time that they are trying to “learn” your product.

2. Users want to “play” on mobile devices

When conducting user testing on mobile devices, we have found that users immediately want to “play” – they want to grab the device and click around the website or web application.

The very first mobile user testing session we ever did was a disaster. We had intended to follow a “tightly controlled” process where the user was only able to look at one screen at a time. Instead, the user picked up the device and immediately began jumping around the site.

This can be solved by using the “free exploration” method, or giving the user clear guidelines before commencing.

3. Observing detailed interactions

Small screen devices mean smaller and more intimate interactions. This means that you have to closely observe the user at all times. Videoing them and replaying it later for analysis may help with this.

There is actually a range of different methods that can be used to capture the testing process.

You can record iPhone and iPad devices using a combination of Air Flow and screencast software such as Camtasia or Screenflow.

You may also want to record “over the shoulder” or “head-cam” video of the user as they move through the site. This allows you to see the users’ interaction experience from their point of view on top of the screen, rather than just recording the data from the screen itself.

You can also hook up the device to a Data Projector. We have found this useful when we have other people observing the testing process. These observers can view the process live, but from a different room.

Conclusion

It’ s very important to test your responsive websites with users – just as you would any other product.

This is the only way you will be able to see how real users interact with your site. It takes some planning and effort, but it is definitely worthwhile.

Need more?

If you are based in Australia, you can come to the Max Design series of workshops touring capital cities during March 2013, where we address key aspects of Building Responsive Websites. 

Free Chapter! HTML5 & CSS3 for the Real World

Get a free chapter of SitePoint's new book, the second edition of our popular HTML5 & CSS3 for the Real World and receive updates on our latest offers.

  • Tim

    So, I expected this article to actually be about testing people and showing the results or videos or something. The majority of this info we already know. What we don’t know is how people are actually using the sites. Please link to videos showing this. Show why they are clicking where they click. Have the users talk aloud explaining their thought process as they are browsing the mobile site. This article was a huge letdown.
    As someone who has seen usability tests in the past of people viewing regular websites, “paper-based wireframes” absolutely do not work. Users cannot grasp the correlation between what is on paper and an actual live site. Avoid that step or you are just wasting time. Users cannot comprehend anything other than the final working product.

  • http://maxdesign.com.au/ Russ Weakley

    Hey Tim, sorry that the article was a huge let down for you. :)

    The article was written as a quick introduction to user testing for those who are unfamiliar with the concept – and an attempt to explain why testing, especially on responsive sites, is important.

    WHY VIDEO?

    We sometimes video the results for two reasons:

    1. So that we can review the testing process more closely. This is especially important for mobile use as users sometime quickly flip through a process. Video allows us to observe these interactions with pause and play.

    2. To show clients or other tea members.

    PAPER MOCKUPS

    We have found paper mockups to be a useful tool along with other methods.

    We will often test on paper mockups very early in the process. To make sure that users understand that these are “websites”, we try to make these mockups look roughly like web pages.

    Paper mockups allow us to test high level concerns such as whether users understand primary navigation labels, wording for key content areas and whether users can navigate around a site.

    We never use paper mockups for detailed interaction testing – such as how site functions will work.

    We are also aware that some aspects of this type of testing can be highly misleading (such as users being able to see entire pages rather than having to scroll on a website).

    However, paper testing has some huge advantages. We can mockup up a concept and quickly test this. It allows us to do rapid iterative testing very early in the process – before the design phase, before the build phase.

    This is not to say that prototype or final site testing are less valuable. We just find that paper testing is invaluable for some types of information.

    PAPER AND MOBILE

    One thing we have found, while we can run paper testing on wide screen interfaces, paper testing works well (within the limitations mentioned able).

    However, for mobile testing paper mockups do not work well at all. As you mention, it is too much of a stretch for users to understand. More importantly, so much of mobile interaction is gesture and touch based. This cannot be replicated with paper mockups.

    HTH
    Russ