Without data, you’re just another person with an opinion.
— W Edwards Demming
Analytics is an often overlooked area of UX. This book will help you understand how an analytics-first approach to UX will enable you get great results. I’m not suggesting that you use analytics to replace your other UX methods, but instead that you combine analytics with other UX methods to create an excellent user experience.
In the first chapter of this book, we’ll look at why analytics data is important when it comes to evaluating user experience. We’ll then move on to how to check that your analytics data is accurate, and how to go about analyzing it. The second half of the book focuses on how to use analytics to find problems, how to use your data to aid your user research, and how to use analytics tools to evaluate and report on the success of your design changes.
By the end of this book, you’ll have a good idea of how to use your website analytics to make informed decisions about the design of your website or app.
The Importance of Analytics for UX
Good UX is based on evidence. This is often qualitative (qual) evidence, such as observations from usability testing or findings of field research. Using only qualitative evidence, though, means overlooking a huge amount of quantitative (quant) data that may be available to you. Quantitative data is primarily numerical and can be measured. This is often used in the form of website analytics, and can be even more compelling than qualitative evidence. Later on in this chapter, we’ll look in more detail at how the two forms of analysis are defined and what they entail.
It’s not a case of using one form of research and analysis: qual and quant go hand in hand to give a rounded picture of the overall user experience.
The diagram below shows how analytics can inform other user research methods:
As the diagram shows, the landscape is made up of both qualitative and quantitative methods. Both make up crucial parts of user experience research and design. They give different forms of insight that are crucial for getting a rounded picture of the user experience of a website or app.
Put simply, the quant side of things will generally tell you what users are doing on a website; the qual side of things will help you to find out why those things are happening.
Advantages of Using Analytics in Your UX Process
Web analytics should be a vital part of your UX process. However, you may not be aware of all the reasons why you should include a quantitative approach to your work. In this section, I’ll cover five advantages of using analytics information to shape your UX recommendations.
- Analytics provides facts that are hard to argue with.
The data that comes from website analytics software is not subjective in the same way that qualitative data often is. While forms of qualitative research are open to interpretation, the metrics used in website analytics are more black and white. This leads to fewer arguments about the findings.
As this quant data is definitive, it also makes it perfect for measuring the impact of changes. If a design solution to a problem is implemented, it can be measured by analyzing the changes in user behavior before and after the change was made. This makes it easier to prove your point and show the value of your work.
- Analytics gives fast results.
Analytics can often give an almost instant answer to a question. Other forms of user research can take days or weeks to produce results, but accessing quantitative data can be as easy as opening your analytics package and navigating to the right report. As mentioned previously, I’m not advocating using only a quantitative approach, but if time is short, it may not even be possible to use other UX methods. You can get a good overview of user behavior on a website within minutes using the right analytics tools and techniques.
- Analytics offers unique insight.
There is a huge amount of information available from your web analytics package. In Google Analytics alone there are around 80 different standard reports made up from over 400 different dimensions and metrics, and that’s before you event start to create your own reports. No other UX methods can give you as much data.
Also, as website analytics are constantly collected, you have a full overview of how a website is being used over time. Other methods tend only to show a snapshot of how users are engaging with a site over a particular period. Analyzing data that has been consistently collected over time gives much more insight than one-off studies.
- Analytics combines well with other UX methods.
As you’ll find out later on in this book, analysis of your analytics data can fit in at various points during your UX process. Data sourced from your analytics can pinpoint problem areas on your website that can then become a focus for usability testing, heuristic assessments, or any other form of qualitative research.
Demographic information can also be taken from analytics to help build up your personas and to provide the basis for user research or usability testing recruitment. If you’re evaluating user journeys, website analytics can play a central role in showing you how people are currently navigating your website. Finally, as touched on before, analytics data can be used to measure the impact of any design changes you make. Analytics data can enhance almost every aspect of your UX work!
- Analytics offers an effective way to present your findings.
Presenting findings of UX work to stakeholders can be difficult. Getting across the results from long-term field studies, for example—in a way that stakeholders can relate to—can be a challenge. Analytics data naturally fits into presentations. This relates to point 1—that analytics data is hard to argue with. The “black and white” nature of analytics means that you have “hard” data to show stakeholders and clients. This can be particularly useful for demonstrating the difference that your UX efforts have made, and even showing the ROI of your work. Also, clients and stakeholders love a graph!
Arguments Against Using Analytics
Hopefully I’ve convinced you that analytics should play a key role in your UX work. Just in case you still have any doubts, though, here are some of the common arguments against using analytics, along with an explanation of why they’re all wrong.
- Analytics devalues other UX work.
Some people are concerned that using quantitative data dehumanizes users and takes away from the purpose of UX. This would be true if quantitative data were used on its own. But as long as it’s used to support qualitative methods, there’s no negative impact of using analytics data. Quantitative analysis will tell you what, but not why. That’s why it can’t be used in isolation. It serves to inform other UX work rather than to replace it.
- Quantitative analysis takes a certain mindset.
I’ve heard people claim that they don’t have the right type of mind for analytics, and that they’re “not good with numbers”. I don’t believe that’s the case at all. I wouldn’t consider myself to be naturally analytical; I never did well with mathematics at school, and it’s not an area I’ve always wanted to work with. Analyzing metrics relating to how people use a website isn’t rocket science. The analytics software does most of the number crunching for you. Nobody’s going to ask you to come up with complicated formulas or have an in-depth knowledge of statistics, and you don’t have to be a full-time data analyst to get great insight from your data. The real skill is in knowing where to look and how to interpret reports. If you can navigate a website and pull out the key information from it, you can do web analytics.
Once you understand the basics, you’ll be surprised how easy analyzing your website analytics actually is. The people I meet in the UX industry tend to have a broad skill set across a range of different disciplines, so adding analytics knowledge to that shouldn’t be too challenging.
- Learning analytics is too difficult.
Following on from the previous point, developing a good understanding of web analytics really isn’t that difficult. Learning some basic principles will get you a long way. If you already have a good idea of the data you want to find, then a basic understanding of how to use a web analytics package will be all you need to get started. There is also lots of support for analytics packages available online. Google Analytics has the excellent Google Analytics Academy, a free resource containing hours of instructional videos. Then, of course, there’s this book, which will show you how use your analytics knowledge to aid your UX process.
- It takes too long to set up.
Some people believe that it takes too long to set up web analytics, integrate it and gather enough data. Even if you’re starting completely from scratch, you should be able to set up analytics software on your website very quickly. It’s often as simple as spending five minutes setting up an account, and then a couple more minutes adding a line of code to your website. You can set up analytics on a website in the morning and have access to visitor data by the afternoon. A more advanced, bespoke setup may take a bit longer—but no longer than many other forms of analysis. Setting up a usability testing session, for example, can take days. Also, once your analytics software is set up correctly, it will constantly deliver useful data with very little maintenance.
- It only shows data on website/app visitors.
The fact that website analytics only contains data from people who visit your website or app is a limitation. You won’t be able to find any data in your reports about potential users, and, as a result, you’ll only get part of the picture. This brings us back to the first point, though. This book is not about convincing you that you should only use quantitative data. Instead, it’s about how you can use quantitative data to inform your qualitative work.
Defining Qualitative and Quantitative Data
Hopefully the previous pages have convinced you that analytics will help you improve your UX process, and you’re now keen to get started. Before you dive into your analytics, let’s have a look at what we mean when we talk about quantitative and qualitative data. The table below gives an overview of the two terms.
Quantitative data is anything that can be “quantified”. This type of data is normally presented in the form of numbers, and is used for discovering facts. Examples of quantitative data are things like the number of visits to your website, the amount of time people spend on your website, and the amount of purchases users make on your ecommerce website.
Qualitative data cannot be expressed as a number. In UX terms, qualitative data often centers around understanding behaviors or attitudes based on observations. This could be from user research, and is more open to interpretation than quantitative analysis. Other forms of qualitative analysis include stakeholder interviews, immersive research and heuristic assessments. Qualitative data is generally used to find out why something is happening.
Both types of data are valid types of measurement, and both should be used during the UX process. The real skill is knowing which type or method to use to achieve your particular UX goal. Remember, though, that quant and qual compliment each other, so rather than using one or the other, use both.
Quantitative UX methods are used to get an understanding of what is happening with a website. They seldom tell you much about the motivations and motives of your users, but they can be used to get a good understanding of what they are doing on your website. Some typical examples of quantitative methods include:
- web analytics
- form analytics
- A/B testing
- eye tracking
- click testing
- surveys (can be quant and qual)
These methods will help to get a great understanding of what people are doing on your website. The data provided by these quantitative methods will help to build an understanding of where you might want to focus your efforts on improving your website.
Both quant and qual are vital for a rounded UX approach. This book will focus on how you can use your quant data to inform your qual work. It’s all about taking an “analytics first” approach to UX.
Qualitative UX methods take many forms, and are often used for building an in-depth picture of your users. They are typically used to help you find out why something is happening. Some typical examples of qualitative methods include:
- usability testing
- ethnographic/field studies
- focus groups
- diary studies
- screen/session recording
- user feedback
- stakeholder interviews
- immersive research
- design workshops
- heuristic assessments
These methods will help to get a great understanding of your website users. The level of detail provided by these qualitative methods will help to build empathy with your users, and will give a good understanding of why they behave the way they do. This book is not about these types of methods, but it will reference them to show how they can be informed by quantitative data.
A Look at Some of the Analytics Tools Available
The final section of this chapter covers some of the different analytics tools that are available. This is not intended as an in-depth review of each option, but instead gives an overview of the pros and cons of each. Remember, the tool you use is less important that how you use it.
A lot of analytics tools provide similar reports to each other. Website analytics tools, for example, will normally show you the similar data in a similar format. The majority of the techniques covered in this book can be achieved using a combination of the tools that are covered below.
Website Analytics Tools
The most commonly used type of quantitative tools are website analytics tools. These tools will tell you, amongst other things, how many visits your website has received and what pages those users viewed.
There are several advantages to using Google Analytics as your analytics tool. One of the most compelling reasons to use Google Analytics is the fact that it’s free. There is a premium, paid-for version available, but this is only recommended for websites that receive millions of visitors a month. You’re paying for the tool to handle more data and for dedicated technical support. The free version will be the best option for the majority of us, and it provides almost all the functionality of the premium version, so you’re not really missing out on anything by using the free version.
Google Analytics is very easy to set up, too. Anyone can register for an account, and the setup process just involves adding a simple snippet of code to every page of your website. While customization of your account is recommended, just adding the code to your site will mean that you have access to the majority of the standard Google Analytics reports. You can get the basics set up and begin tracking visitors within about ten minutes of creating an account.
Though no customization is required to use Google Analytics, it’s recommended for getting the most out of your data. There are many customization options that will help you improve your reporting and ensure it’s relevant for your requirements. Google Analytics is easy to customize, allowing users to set up their own goals, dashboards and custom reports. You can even import data from other sources and analyze it alongside Google Analytics data.
As touched on previously, Google Analytics is the most widely used analytics service in the world. This means that it’s very well supported. Google itself provides excellent tutorials and documentation, and many independent analysts, including me, regularly blog about how to get the most from Google Analytics. If you want to focus on one website analytics tool, Google Analytics is a good choice, as you’re likely to encounter it on many other sites. It’s like choosing to learn a language that’s spoken in lots of countries.
As the tool is constantly being updated, it means useful new features are launched regularly. One example of this is the recent introduction of the User Explorer reports, which allow you to analyze the behavior of individual (anonymous) users over time. This is a very exciting development for us UXers.
There are few disadvantages of using Google Analytics. One of the main disadvantages is actually a side effect of one of the advantages. The fact that Google Analytics is updated regularly means that useful new features become available. However, updates to the Google Analytics interface and reports mean that you need to use it regularly to stay on top of these changes. If you don’t access your Google Analytics account regularly, you may find yourself lost when you log in after some time away and find yourself presented with unfamiliar-looking reports.
At the other end of the scale, larger websites may run into data sampling issues. Data sampling is an analysis technique that analyzes a smaller set of your data in order to identify larger patterns and trends. Google Analytics uses data sampling to speed up the performance when dealing with a large amount of data. This can lead to accuracy issues and inconsistent results.
Another issue for some is that you need to allow Google access to your data. While this is not a problem for the majority of website owners, there will be cases when this is not desirable.
The use of ad-blocking software has also led to users blocking their data from appearing in Google Analytics. Some ad-blocking tools will block Google Analytics by default, while many others can be set up to block Google Analytics tracking at the click of a button. This problem is not unique to Google Analytics, but it’s an issue that you should keep in mind when analyzing your data.
Adobe Analytics (previously Site Catalyst) is a set of tools for predictive and real-time analytics. It has a lot of similarities to Google Analytics in terms of its reporting functionality. It’s less commonly used than Google Analytics; around 360K sites use it globally, though it’s still used by a lot of high-traffic, high-profile websites.
Adobe Analytics is sometimes seen as the better choice for websites with a lot of traffic. Google Analytics will sample data if there’s a large amount of traffic. Sampling data can lead to inaccurate reports, which can potentially be a big issue.
The tool is almost infinitely customizable. This means that, when it’s set up correctly, it can give you almost all the data you’ll ever need. To create the optimum setup in Adobe Analytics, three components have to be configured: business objective implementation, technical implementation, and end-user understanding/training. If you can crack all three, then Adobe Analytics can be an immensely powerful analytics tool.
Adobe Analytics also integrates out of the box with other Adobe marketing tools, making it an attractive choice for companies who are already using Adobe products.
One of the main drawbacks of Adobe Analytics is the cost. It offers no free version, unlike Google Analytics, and the cost of accessing the tool is likely to be tens of thousands of dollars.
Another big issue for Adobe Analytics is that there’s no out-of-the-box interpretation of the data. Reports—particularly the custom insight variables—differ from implementation to implementation. This means even if you master the setup of one Adobe Analytics account, you may be completely stumped by another. Adobe Analytics is a powerful tool, and with this comes the need for training and external support.
Adobe Analytics requires a fair bit of configuration. This means that setting up the tool for the first time can be a lengthy and complicated process.
Other Notable Web Analytics Tools
Yandex Metrica is an interesting tool. It’s currently free to set up. As well as giving standard website analytics, it also offers heatmapping and session recording, where users’ visits are recorded and can be watched back on video.
Heatmapping and Session Recording Tools
There are blurred lines between the different types of analytics tools. Often tools will fall into more than one category and cover a range of services. The tools below focus primarily on heatmapping. Heatmapping covers a range of different click, scroll and mouse movement reports.
Heatmapping tools use colors to show graphical representations—primarily of where users have clicked on a page. They’re useful for giving an “at a glance” indication of which elements on a page have received the most clicks. The heatmap below shows where users have clicked on a page. (Reports of this type are also referred to as “click maps”.)
Scroll reports are another type of heatmap, and are really useful for indicating what content is just not being seen: they track how far down your pages users are scrolling.
Crazy Egg works by taking a screenshot from a specified URL (or a collection of URLs), collecting all the clicks on that page, and projecting it as a heatmap on top of the screenshot. A lot of valuable data—such as the referrer, country, device type, etc.—are added to that click data. Heatmaps can be set up for specific device types, so you can see separate heatmaps for mobile, tablet and desktop users.
Scroll mapping is also available in Crazy Egg. Scroll maps can be used to show how much time users spend looking at sections of your page while they scroll.
Crazy Egg is very easy to set up on your website. All you have to do is insert a simple asynchronous script in the footer of the pages you want to track.
As touched on in the introduction, there are several different types of heatmap available within Crazy Egg, which means it can give you a lot of insight.
It’s a very widely used tool as well, so there’s plenty of support available for it. Crazy Egg also integrates with A/B testing tools like Optimizely. It works with HTTPS, iframes and Flash objects, making it particularly versatile.
A new “snapshot” needs to be set up for every page that you wish to analyze. Only when this has been set up can the data start to be recorded. This means that you won’t have the same instant access to data as you can get with some of the other tools.
Crazy Egg only focuses on heatmapping. It offers in-depth heatmaps, but doesn’t offer any other functionality. This may be a good thing, as it can focus on one area and provide a great heatmapping service. If you want session recording and form analytics, though, you may want to try another tool instead.
Hotjar offers a range of different tools. Heatmapping is one of these. You and can also see form analytics and session cam recordings of your users. Hotjar is a relative newcomer to the analytics market, having come out of beta in 2015.
As with a lot of analytics tools, Hotjar is easy to set up by just adding a line of code to your pages.
It’s a bit of an “analytics Swiss army knife”, as it includes heatmaps, session recordings, form analytics and several other tools. Despite this, it’s a relatively inexpensive option compared with similar enterprise-level tools.
Hotjar can be used on multiple websites, making it a good option for agencies likely to want to analyze several different sites.
It also integrates with A/B testing tools, which is particularly cool for seeing the heatmaps, session recordings, and survey responses from audiences who saw different test variations.
Hotjar can be reliant on sampling data, which can lead to questionable data. Like Crazy Egg, it’s not “always on”, meaning that you have to manually set up tracking on individual pages.
The fact that Hotjar includes so many different tools means it can be overwhelming. It may also lead you to try to do too much and potentially waste valuable time over analyzing—though this isn’t a fault of the tool!
Being a jack of all trades, it doesn’t offer the level of targeting and analysis for each of the features that you’re likely to get from more focused tools.
Clicktale is a similar tool to Hotjar. It’s described as a “customer experience analytics tool”. Serving over 2,000 customers worldwide, including Fortune 500 businesses, Clicktale is one of the fastest-growing companies in this space.
Clicktale is an enterprise-level tool designed for larger websites. The tool is “always on”, meaning that it continually pulls in data for every page it’s set up on.
Clicktale has an even wider range of tools than Hotjar, giving an extremely detailed account of all user behavior.
As with Hotjar, there’s a risk that Clicktale could provide more options that you need to analyze.
Cost could also be a barrier, as Clicktale is an enterprise-level tool with an enterprise-level subscription fee.
Other Notable Heatmapping Tools
There are lots of tools on the market that can provide you with heatmaps, session recordings and other analytics services.
Some of the more popular ones include Lucky Orange, Clicky, SessionCam, Inspectlet and Mousestats. Having used all of these (or at least had demos of them), I’ve noticed a lot of similarities, making it hard to recommend any one individual tool.
Before you decide which heatmapping tool to use, you should first think about what you’d like it to do. Some tools have some great-sounding features, but you need to consider how often you would use them, and whether they would help or slow down your analysis process.
Split Testing Tools
Split testing is a method of conducting controlled, randomized experiments on two or more variations of a design, to see which variation generates the highest number of conversions. We’ll cover split testing in detail in Chapter 6, but the section below gives a brief overview of some of the market-leading tools.
Optimizely is an advanced split testing platform that’s easy for beginners to get to grips with. The setup just requires the addition of a small code snippet to your pages, after which you’re ready to set up your first split test. Customers are able to set up tests using a WYSIWYG editor or a code editor, depending on how comfortable they are with developing their own tests.
A 30-day free trial is available to everybody, so anyone can give the tool a try to see the functionality it offers.
The platform allows you to integrate with several other tools such as Google Analytics, Adobe Analytics and Crazy Egg.
Optimizely isn’t the cheapest tool available, but it’s one of the most popular due to its ease of use, advanced features and great support and community.
Adobe’s split testing tool offers similar functionality to Optimizely. The tool is less accessible, as no free trial is available. It offers a strong enterprise-level solution, however, and integrates seamlessly with other Adobe Marketing Suite products. This makes creating audiences for tests, and analyzing results, easy for companies that are already using the other Adobe Marketing Suite tools.
Visual Website Optimizer (VWO)
VWO is one of the most affordable split testing tools currently on the market. While low prices make the tool attractive to smaller companies, VWO is also used by leading brands like Microsoft, Virgin Holidays and Toyota. It’s easy to use for both developers and non-developers, meaning there’s a low barrier to getting tests set up.
The platform makes use of several built-in features, including a heatmapping tool that’s available for all experiments.
Google Optimize 360
As a new-comer to the market, Google’s Optimize 360 tool has a lot of catching up to do. The platform focuses on its native integration with Google Analytics. This provides you with the behavioral insights you need to create effective tests and to deliver optimized customer experiences.
The tool is easy to use, and its integration with Google Analytics means it’s simple to set up, run and report on data-driven tests.
Other Useful Analytics Tools
This final section covers other types of analytics tools that can be useful for the UX process. New tools are being developed constantly, though, so don’t be surprised to see new contenders emerging.
UsabilityHub offers a range of different tools to help you make design decisions. There are five different testing tools available: five second test, click test, question test, navigation test and preference test. These tests provide a mixture of qualitative and quantitative data. The quantitative tests are the click, navigation and preference tests.
Click tests work in a similar way to heatmapping tools. Users are asked to click on a screenshot rather than a fully functioning webpage. They’re asked a question to begin with—such as “Where would you click to get in contact with us?”—and their responses are recorded on a heatmap. This test also times the average response, to show how quickly users were able to find your information.
Navigation tests work in a similar fashion, but they use multiple screenshots instead of one. Users are given multi-step tasks to follow, and clicking on the first screenshot will navigate them to a second, and so on.
Finally, the preference test is a simple way of finding out which design users prefer. You can upload two images, often screenshots, and ask users which they prefer. This gives simple, direct, quantitative feedback—though keep in mind that asking users what they want is not always the best option!
Formisimo is a form and checkout analysis tool. It gives powerful data on how users are engaging with forms on your website. Users’ actions are recorded on all form pages, showing which fields are causing problems. Formisimo records where users drop out of the form-filling process, and also which fields are taking users the longest to complete.
Because Formisimo only analyzes form usage, it’s simple to use but gives vital, focused data. If your website analytics suggests that forms may be causing you an issue, a tool like Formisimo will give you more detailed data to analyze, better helping you to pinpoint the problem.
Woopra is a real-time customer analytics service that can track how individuals use your website or app. It tracks your new and unidentified website and mobile app users from their first touch anonymously, in a similar way to how the User Explorer reports in Google Analytics. Once the user chooses to sign up, subscribe or do something similar, all of their previous anonymous activity is merged into the same, identifiable customer profile. These customer profiles can contain all the information you have about a user, including details such as their name, company and email address.
Woopra is primarily used by marketing and sales teams, but it provides data that can be really useful for making UX decisions. It can help you understand user interaction on a deeper level than other analytics tools.
Using Tools Together
The tools covered in the previous section will all give you data about your user behavior. It’s rare that any one tool will provide you with all of the data that you need. Combining data from different tools will give you more insight than focusing on data from a single source.
For example, you may start off by using a website analytics tool like Google Analytics to find out where problems are occurring on your site. This might identify a page, or group of pages, where you’re seeing some worrying trends. You can then delve deeper into the problem using a heatmapping or form analytics tool to get a more precise idea of what specifically is causing the problem.
Some tools can be integrated, too. A good example of this is integrating your split testing platform with your website analytics. Your split test results will give you information on conversion performance, while your analytics will enable you to go deeper into the data and look at broader user behavior for the variations in your tests.
Analytics Tools Summary
With so many different tools on the market, you could be forgiven for feeling a bit overwhelmed by the options. However, most of these tools do similar things and operate in a similar fashion, so if you’re familiar with one tool, it shouldn’t be too difficult to get to grips with another.
This book will primarily use examples from Google Analytics. That’s because it’s the most popular tool, it’s free, and it’s easy to set up, so there’s no barrier to entry. It’s important to remember, though, that the principles are more important than the tool itself. This book isn’t a how-to guide or a textbook. Instead, it focuses on the best approaches to using analytics data in the UX process.
You may not have a choice of which analytics tool you use, but the tool is not as important as the technique. This book will arm you with the analytics knowledge you need to take your UX work to the next level.