“Do you have a good hotel recommendation in York?” asked one of my best friends this morning. “Nah, “ I replied, as my experience from our stay last year was pretty lackluster. During a one-day stopover, I booked a room late in the afternoon at a quaint hotel.  We checked-in, dropped our luggage in the room and went for a walk in the city. Upon our return later that night, we noticed a strange and unpleasant smell coming from the bathroom. But we were tired and had an early wake up call the next morning so we didn’t have the will to complain; instead we shut the bathroom door and went to bed. This is how we “resolved” the smell issue. We woke the next morning, checked out and went on driving towards Scotland. None of the checkout staff asked if we were happy with our stay and we did not bother mentioning the unpleasant smell.

Why am I telling you this? It’s because here lays one of the biggest challenges for many companies as they attempt to address usability issues with their products. Many managers believe analysing user support calls will yield the usability issues facing their users and then they’ll fix them.

Sounds reasonable? It is natural to think so. The data is available as most companies keep support call logs so all you need is smart analysis. With minimal effort, you can get a big list of usability issues to resolve. Unfortunately, this is not the case. The problem with such an approach is that many users do not provide accurate feedback or any at all (as in my example above). People often think complaining is inconvenient and are skeptical as to the service they’ll receive after doing so. But, as a company, the fact that a customer does not log a complaint does not necessarily equal satisfaction. Your customers might be resolving their problem the same way we did – hiding it or ignoring it. Slowly but steadily those dissatisfied customers may drift away until they are gone forever. It’s the users who do not renew contracts or upgrade to the latest version and when you find out it’s usually too late. Support analysis will give you valuable data insight and knowledge but will not paint the full picture of consumer sentiment around your product or service.

Your overall customer base can be placed into three buckets:

• Customers who absolutely adore your offering and care about providing feedback, good or bad. They write add-ons, blog about you and praise your brand publicly. You usually get a lot of information from them with little effort.
• Potential users who are not your customers. You attempt to grow your user pool by acquisition strategies but certain people are just out of your focus, which is fine. You might, however, be surprised to learn that some were a customer at one point.
• And then there are your silent customers. They simply use your product or service without giving away too much information. The majority of your customers will comprise this group and are the most likely to drift away without you even noticing until it is too late.

Coming from the world of support and user services, I know how important it is for users to provide input and even I did not file a complaint. Plumbing issues can occur in any hotel and it could have resided in my memory as a “great little hotel that upgraded us to a magnificent suite after we told them about the odor problem.” Instead it’s the hotel I did not recommend to my friend.

So, what should the hotel have done to keep me a happy customer? What should you do to prevent users from drifting away? First and foremost, acknowledge user support and services as one of the organisation’s most important activities. This is how customers see you and is a major way for them to establish a positive opinion about your brand.

When you put your mind to it, you can develop practical steps to ensure the following:

• Make it easy for customers to provide input, whether positive or negative. A good example is “Love Film” DVD rentals service (owned by Amazon). When “Love Film” sends a DVD, a customer will also receive a pre-paid return envelope and feedback form already populated with comments / problems commonly reported by other users. Instead of having to freehand, “This DVD is scratched”, a user simply has to check the box.
• When you do receive a user comment, show it’s important to you. Make the user feel you have taken their feedback to heart and provide fast, courteous and impeccable service. Apologise and compensate!
• Most importantly, be proactive. Do research. Perform user testing of your products not only before releasing them but also over time because circumstances can change the way your product is used (E.g. people, technology, social norms, context of use, etc.). You must invest in user research; test user journeys over and over again during the product life cycle. Do not rely on users to provide you with voluntary feedback.

The above steps serve as a way to start a conversation with customers and continue the dialogue, creating brand loyalty. Don’t let your customers drift away. Instead, create an experience that makes them recommend your nice little hotel in York to their friends.

Mobile retailing has exploded in the past year, with almost one third of shoppers having used an m-commerce site, and the number of people completing a transaction quadrupling.

However, while mobile payments have become an important part of Japan’s economy since it was introduced in 1999, the rest of the world is yet to widely embrace m-commerce. With the 2012 London Olympics looming, the major mobile operators are planning to introduce NFC (near filed communication) widely to make purchasing transport and tickets easier.

One to One Insight has conducted independent research, surveying almost 1,000 people to understand customers’ attitudes towards mobile payments and ultimately their propensity to use mobile payments. Dr. Philip Rhodes is your host for this second episode of OneToOne.tv. We invite you to join him as he discusses consumer concerns about mobile payments and addresses whether m-commerce is the future of payments.

Last weekend I helped to run a small community festival, celebrating cultural diversity in our neighbourhood. We set up marquees in the local school ground, and invited traders to present their wares – everything from ice cream and hotdogs to hand-woven woollen blankets.

My job was to sit behind a desk by the main entrance labelled “information”, to answer questions and hand out paper guides to those who requested them. It all seemed very self-explanatory.

But a bizarre thing kept happening… over and over again, people would approach my desk and say “what’s this then?” or “what do we do here?”, even just an inquisitive “Hello?”. I found myself repeatedly explaining, “This is information – what would you like to know?”.

I had a good long think about this, to understand the situation from the point of view of those enjoying the festival. Why were they confused and what could we have done to clarify things?

Here are some of my thoughts:

Convention

As we know, people don’t like to think unnecessarily. On a hot sunny day, they wander into a gate marked ‘Entrance’ and what’s the first thing they expect? To be stopped, counted, charged, given a ticket or wristband… some sort of authority is supposed to step in and ratify their entry through that gate.

Similarly there are very strong conventions on websites. People know by now that a company logo is on the top left and links to the homepage. They know that underlined text is a link to something… On eyetracking studies we see that customers gaze in the place they expect something to be, creating a hot spot in conventional areas even if that area is blank on the page they’re using.

So this was a free community festival. We had no interest in stopping people. We had a counter under the table to make sure we didn’t exceed site capacity, but there were no tickets, wristbands or bag searches. So, like the eyetracking studies on a webpage, people thought “there’s something missing here – I should stop and wait for it”, and when nothing happened they habitually sought out that expected authority.

Sheep Mentality

It’s still considered a slur on our personalities, but countless studies have demonstrated this to be very valid indeed. In a situation that people are not especially passionate about, they will happily follow the crowd and do what they know other people will do.

Amazon famously took advantage of this when they introduced ‘People like you bought…’, now copied across the web with great commercial success. And at this festival, it was a simple matter of following the person in front of you. When the first person approached me to ask a question, everyone else behind him followed and approached without knowing why.

Labelling

So those are probably the main insights into human psyche but often the most effective solution is the most simple. How well did we really signpost the stall? We had a big sign saying “Information”, sure, but it was propped up against the low-level table. So as soon as a crowd started to form in front of it, it was unreadable to anyone else entering the site.

And how descriptive is that word “Information” anyway? Does it tell them all they need to know? Considering what we’ve learned about convention, perhaps it would be more effective to have a clear sign on the gate saying “Free festival – come on in!” or similar.

Does your website, Facebook welcome page, or mobile app have a problem like this? Have you labelled a button or link with a completely logical word or phrase, only to find that customers don’t see it or don’t get it? I know it’s happened to me before – and you don’t find out until you watch it happen to someone in real life! But then, that’s exactly what usability testing is all about, isn’t it?

A couple weekends back I went to the Glastonbury Music Festival. It was in England. In June. In a field. It rained. It got muddy.

This is a festival with a main site area of nine square miles and a population of around 170,000. It contains 35 main stages hosting countless performers, plus a seemingly endless selection of stalls and shops, over a period of 5 days. Just a few of the headline acts this year were U2, Coldplay and Beyoncé Knowles.

I watched with interest last week as people struggled to get into and around the festival site. Groups of friends donned wellington boots and waterproof ponchos as they heaved bags, crates, tents and poles along muddy paths and across miles of farm land. There were exhausting tales of four-hour queues to get into the site, followed by yet more time trudging round and round the site trying desperately to find a spot to camp. Customers pitched tents, their hair clinging wetly to their skin, and dragged crates supermarket beer through the mud, all before sitting on a damp and dirty groundsheet and sampling their room-temperature wares.

Now, this immediately struck me as odd. After all, in this business we constantly preach the importance of making your customers’ lives as easy as possible. We tell you that they don’t want to work hard, that everything must be laid out for them, that they’re fickle people who will drop out as soon as you make it difficult. Why should Glastonbury be different?

Well we’ve hit upon a contradiction. There is such a thing as making your experience too easy, too pleasant, too… bland. If a customer doesn’t have to think at all, they will simply switch off. Any tiny difficulty will suddenly earn significance and the customer will not be engaged enough to bother with it. They’ll actually be more likely to drop out of the process than their harder-working counterparts!

Before you think I’ve gone mad, let me explain some rules…

The rules of making your customer work

1. Brand. Before your customer will work for you, he/she needs to know who you are. Glastonbury has been running for 41 years – it has an international reputation and is the UK’s largest music event of each year. Everyone’s heard of it and the core customers, the ones who attend year after year, are absolutely dedicated to it. They use it to help define their personalities. They plan their lives around it. It is even their religion. When you talk to people about why they go to Glastonbury they’ll tell you it’s “for the vibe”. Is that really something that a company can create, or is it just a customer perception created at least in part by the sense of accomplishment that follows struggling to get there?

2. Investment. The customer has to have invested in your product before they start work. The people I watched queuing up in the rain for Glastonbury had each spent around £200 ($320) on a ticket months and months before the event itself, and there were no refunds. These people had already decided that the product was worth investing in and had handed over their hard-earned cash, so their minds were set – they were going to get value from their tickets.

3. Value. And that leads us nicely onto the most important rule. The customer has to know exactly what they’re going to get at the end of this, and that it’s worth all their effort. You wouldn’t suffer through mud and rain if you weren’t going to get to see Beyoncé at the end of it. Equally, you wouldn’t fill out a lengthy web form if it wasn’t going to give you some sort of reward. If a customer has a goal in sight and they know how to reach it, then there’s no stopping them.

4. Competition. Sense has to prevail – if your competitors are offering the same thing but making it simpler to achieve, then there’s just no point in trying. You absolutely must have an edge that justifies the effort, or you’ll lose people at the first hurdle. Glastonbury works because there’s nothing in the UK comparable to it. It’s the biggest, it’s been running the longest, it books the most exciting acts. If an identical festival sprung up next door with a concierge service included in the price, we’d be telling a different story. And if your business has a twin, you must make sure yours is the easier to engage with.

The benefits

So, even if you have everything in the list above, why would you want your customers to work hard? Isn’t it still a negative thing? Well, there’s no customer more loyal than one who’s invested effort into your brand. It’s part of a cycle – if someone has gone out of their way to learn about you and your product, if they’ve spent time interacting with your brand, then they’re familiar and comfortable with it and willing to do the same again. They will overlook some mistakes, forgive little flaws. They feel a sense of accomplishment, they’ll stick with you above your competitors and they’ll spread the word to their peers. And who doesn’t want a core group of customers with that strong sense of loyalty and evangelism?

If Glastonbury wasn’t such hard work, maybe it would never have achieved such success.

The July-August 2011 Edition of the Harvard Business Review highlighted an interesting perspective regarding the color pink and if it is helping or hurting the breast cancer movement. Dr. Stefano Puntoni, an Associate Professor of Marketing Management at the Rotterdam School of Management, Erasmus University, authored an article titled “Defend Your Research: The Color Pink Is Bad for Fighting Breast Cancer“. Puntoni and his research partners—Dr. Steven Sweldens of Insead and Dr. Nader Tavassoli of London Business School—conducted 10 studies over the course of three years to determine if “pink and other gender cues would make campaigns against women’s diseases, such as breast and ovarian cancer, more effective”.  Surprisingly, they found the opposite was true.

The above infographic is an attempt to illustrate the results of the research conducted by Puntoni, Sweldens, and Tavassoli.  It indicates that gender cues (female colors, symbols, and voices/visuals of things associated with women) may be counterproductive to the goals of breast cancer fundraising.  The reason: a psychological reaction called defensive response.

Puntoni writes:

In psychology, there’s a lot of literature on defensive responses. How do we deal with threatening ideas, with things that are existentially difficult to comprehend? What happens is, these set off very strong denial mechanisms. By adding all this pink, by asking women to think about gender, you’re triggering that. You’re raising the idea that this is a female thing. It’s pink; it’s for you. You could die. The cues themselves aren’t threatening—it’s just a color! But it connects who you are to the threats. – Puntoni “In psychology, there’s a lot of literature on defensive responses. How do we deal with threatening ideas, with things that are existentially difficult to comprehend? What happens is, these set off very strong denial mechanisms. By adding all this pink, by asking women to think about gender, you’re triggering that. You’re raising the idea that this is a female thing. It’s pink; it’s for you. You could die. The cues themselves aren’t threatening—it’s just a color! But it connects who you are to the threats.”

What does this mean for for the organizers of breast cancer fundraising who rely on the Pink Ribbon, “Think Pink” campaign slogan, and general use of the color pink as a brand asset?  There is no doubt that since 1991, when the Susan G. Komen Foundation first handed out pink ribbons to participants in its New York City race for breast cancer survivors, that the use pink has increased awareness and strengthened solidarity for the cause. Puntoni’s study suggests that organizers may need to go further to maximize their marketing efforts. Simply relying on the association with color may be leaving more on the table then originally thought.

Indeed, Puntoni and his research partners are getting early evidence that acknowledging the fear seems to offset the defensive triggers brought about by the us of these specific gender cues.

“We’ve seen that the negative effect on perceived risk can be eliminated by helping the audience build a buffer against the threat posed by breast cancer by, for example, boosting their self-esteem by asking them to think about times they helped others.”

Needless to say, this research may be opening a whole new field of study around gender cues and their perceived vs. actual effectiveness on behalf of brands. As it relates to breast cancer and the causes that support its victims, these early findings may serve as important insights to uncovering more effective messaging strategies for driving greater awareness of the disease, resulting in even more human and financial support to find a cure.

You may, or may not, know that I live in a Welsh city famed for comedy chav hip-hop, British heritage and the Ryder Cup – an eclectic mix that I’m proud to be part of. If you’re not familiar with it (looking at you YANKS!), below is video to help put things in perspective.

Last year, Network Rail teamed up with Newport City Council and the Welsh Assembly Government to build a new £22 million railway station that would give visitors the impression of arriving at an exciting, up-and-coming metropolitan deserving of an international reputation.

They unveiled their prize in October 2010 and it looks like this:

Wow! That’s £22 million very well spent – it looks like aliens have landed and brought the future with them! It’s got a ‘bubblewrap’ roof that uses the same technology as the Eden Project! Corr!

Let’s forget for a minute the fact that the roof has leaked constantly since opening, that it still leads to the same old dodgy platforms and that it’s a full 200 yards further away from my house. These things aren’t important.

What is important is the experience that you go through from arriving at this station until leaving it. Has this been designed for the people? Is catching your train a smooth action, in which you are led seamlessly on from point to point?

Let’s find out.

We’ll start with a simple mental model for a persona likely to frequent the station. I’m going to focus for now on those who are in Newport and want to leave (it’s going to take some imagination), in this case a commuter who works in nearby Cardiff:

To meet the requirements of our commuter, we will need to provide:

So it’s fairly simple now to design a station that contains all these things, allowing our commuter to move easily from one to the next without having to cross his own path or those of others.  So how does Newport’s new railway station stack up? Let’s take a look at its layout:

Okay, so how does it stack up?

Well, issue number one has to be the clock. There’s no clock anywhere in the terminal building. It isn’t above the ticket office, it isn’t hanging on a wall – it’s not even included in the train arrival display. There is no clock this side of the ticket barriers. So, straight away, our commuter is unable to make use of the train arrival times because, without knowing the current time, he can’t calculate how long he’s got until the train leaves. The next train to Cardiff might be at the platform right now, its doors slamming and the whistles blowing, and he has no way of knowing.

Incidentally, I happened to bump into a man in a suit who appeared to be involved with Network Rail and he asked the attendant why there was no clock.  The attendant answered, “Everyone has a watch, or at least a mobile phone”. True point, but how accurate is our commuter’s timekeeping device? Does he have faith in it being perfectly synchronised with Network Rail’s? And does he want to root around his pockets looking for it when he first arrives at the station? I suspect not.

Next comes the ticketing which is actually very well catered for. It is easy to find the ticket windows and machines, and our commuter can step straight from the doors into a queue. Very neat.

However, this is where we fall into trouble again because, now that our commuter has his ticket, he wants to keep moving through the barriers and off to the platform. But which platform is it? He still doesn’t know the time – will he make it to the next Cardiff train which is on Platform 1, or should he start climbing the stairs to Platform 2? He has no help at all – the ‘train information’ on Platform 1 is too far away to see, and the first available clock is all the way up the stairs. Our commuter has to take a gamble and dash over the bridge to Platform 2.

Our commuter arrives on Platform 2 and realises he has a few minutes to spare. Perfect! Perhaps, a nice cup of coffee and something to read on the way to work? Err – no. Not unless you’re willing to take a five minute march down one platform to the obscure coffee shop hidden at the end, followed by a good ten minute trek back up the platform, up the stairs, over the bridge and across to the opposite end of Platform 1. And let’s face it – nobody has that sort of time in the morning. So instead he crosses his arms, leans against a wall and waits grumpily for the train.

So there we have it… there’s a lot of good to say about Newport railway station. It is very modern-looking. It does have a large, wide bridge that’s never crowded and it does have two entry buildings right next to purpose-built car parks. But it will never give people the great experience that it should, and that’s a shame because it really doesn’t need to change very much.

Just shows you – a little bit of usability testing in your design project can make all the difference!

It’s a subject that’s been studied extensively. In the year 2000, researchers at Columbia University investigated how the amount of choice available to a person affects their ability to make and feel good about their decisions. Best known of these experiments involved asking customers of a delicatessen to sample different flavours of jam. Customers in Group A were offered an exciting array of 24 various jams to taste, while those in Group B were shown a more limited 6 flavours. The result was that, although more people were attracted to the large display, just 3% of Group A chose a pot of jam to buy. Meanwhile 30% of the restricted Group B walked away pleased with their purchase.

The research even extends into sociology, where theories are abound that human beings given too much freedom and flexibility in their daily lives actually struggle to choose a path and are less satisfied with the one they finally take, fearing that it may not have been the best available.

As fascinating as all that is, how can we relate it to an online business? Well it’s all about how you display your products and services. Retailers know that, given a range of similar products, very few will buy the cheapest and only a small number will buy the most expensive – the vast majority will choose a comfortable middle-ground between price and quality. In fact, it’s been proven that introducing a high-end product to your portfolio can dramatically increase the popularity of your next most expensive option.

We’ve also seen repeatedly in AB and multivariate tests that a webpage with too much information, too many buttons or too much general noise can confuse the customer and damage user flow through the site.

So if all this is true, how do we make sure that people are able to reach a conclusion? Show a customer too few choices and they’ll feel constrained, but show them too many and they’ll have no idea which way to turn. Could you introduce new options to widen a very small range? If you have a lot of options can you hide them, displaying only your most disparate products until the customer chooses to search further? The balance is different for every organisation but, with a little experimentation, it won’t be hard to find.

Senior Customer Experience Consultant, Gurvinder Sandhu takes an in-depth look at the user experience of the Game mobile app.

What is it?

The Game app was created to allow users to easily browse and purchase video games from the store. The app is currently on version 2.01.

Purpose?

The App store suggests that the app can be used to locate the nearest Game store, shop for games, view game screenshots, watch videos, read user reviews, and members can check their reward points.

Download?

The Game app is free to download however will require Wi-Fi or 3G to browse and purchase video games.

Overview

Overall the Game app offers limited functionality and features. The features which are present are basic and are not up to the standard Game users would expect from one of the premier retail stores for purchasing video games.

So what does it do?

When the app is started up, the main screen (Fig 1) allows users to scroll through various options such as ‘shop’, ‘reward card’, ‘videos’, and ‘store finder’.

fig 1: Main Screen

Within ‘shop’ (Fig 2), users can scroll in a similar fashion as to the home screen and select any console type, be it Xbox360, Nintendo DS or a PC to filter the different products.

Fig 2: Shop

Within a console, the ‘top sellers’ and ‘pre order chart’ are shown (Fig 3). The ‘top sellers’ chart would be more informative if displayed in a numerical list to show the top ten and also by allowing users to filter by date e.g. top 10 of all time and the current top 10.

The ‘pre order chart’ (Fig 4), although useful, would be more informative if displayed in list form, by month so that users know which month a certain game title is due out.

fig 3: Top sellers

fig 4: Pre order chart

The shopping process works relatively well, although the ‘add to basket’ button may possibly be overlooked as it is quite small and at the bottom of the page. To purchase, users are required to either ‘login via Game’ or alternatively ‘login via MoBank’. This could potentially be a deterrent for users to purchase off the Game app as MoBank isn’t a well known means of purchasing.

Mobank is a free mobile banking service that works securely with existing bank accounts, similar to PayPal but developed for purchases using the iPhone and iPod Touch.

The search function works well however some users may not know exactly what title they are searching for, and may want to browse the full offering from Game; users cannot currently do this unlike on the Game website.

The ‘video’ (Fig 5) section works slightly differently to that of previous sections; in this section users do not scroll to select a console, but have to instead select a small console icon at the top of the screen to view a few games which have video content available. The video icons on the game pages are not immediately obvious and they also hide the cover images of the game (Fig 6).

fig 5: Video

 fig 6: video icon

The ‘store finder’ (Fig 7) opens up into Google maps format to allow users to search by postcode, however this does not work well if a Game store is not near to the location entered. The app does not tell the user where the nearest Game store is. The ‘store finder’ would work better if users could search by a postcode to see a list of Game stores ordered by proximity to the searched location, and then be given the choice to see the Game store of choice on Google maps.

The ‘reward card’ section allows users to view the number of points they have through the Game reward scheme. A suggestion is also provided on what these points can be used to purchase. A greater list of suggestions (currently only one suggestion is made) would be a welcome addition especially since people have different budgets.

What doesn’t it do?

The key issues with the Game app are listed below;

1. Users cannot browse the entire product offering from Game. If the user doesn’t know of a specific game then the ‘shop’ section could be redundant for some.

2. The ‘top sellers’ would be more informative if displayed in a numerical list to show the top ten and also by allowing users to filter by date e.g. top 10 of all time and the current top 10.

3. The ‘pre order chart’, would be more informative if displayed in list form, by month so that users know which month a certain game title is due out.

4. The video icons in the ‘video’ section are not obvious and also obscure the game cover images.

5. The ‘store finder’ section does not notify the user of where the closest Game store is to their entered location. Conventionally store finders work by the user entering in their postcode to see a list of Game stores ordered by closest to the entered location. This is usually followed by the ability to see the location of the store on a map.

What others have to say

The average rating given to the Game app is 2.5 stars out 5.

Reviews on the App Store mentioned the following;

• Users described the content of the app as being ’vague’ and said that they expected better from Game.
• The shop section was described as being limited and users said that it was good if you knew what you were looking for.
• Users mentioned that the app did not display all the Game stores.
• Users were critical of the app not displaying enough options and suggested the inclusion of a list of categories by which they could filter to their desired product.
• Users expected more freedom when browsing the shop section, similar to the Game site.

Final thoughts

The app has generally been received negatively due to the limited functionality and features on offer to the users. The main improvement for users would be to be able to browse the full offering from Game rather than have to search for specific titles or only be able to see a select few titles in the ‘top sellers’ and ‘pre order chart’.

One to One Insight believes that if the above mentioned issues, particularly the ability to browse the entire store are included, then the app will provide a better experience for users and possibly encourage more users to purchase through the app.

The sudden downpour of rain in London an hour before the One to One Social Seminar couldn’t deter over 30 people leaving their offices to brave the rain and join us at the Design Council in Covent Garden.

Dr. Philip Rhodes,  kicked things off explaining how online communities can act as an invaluable research tool if managed properly.  Philip highlighted the benefits of online communities, such as iterating ideas in real time and gave examples of communities that work, such as Dell’s Ideastorm.  Philip concluded his section of the seminar by outlining the ten rules to ensure your online community is successful including developing a thick skin and learning from the negative!

Paul Hedgeland,  took to the lectern to discuss social media strategy and how to target people inside their own social networks.  Paul highlighted that the best way to target the 600 million users of Facebook by creating multiple pages and how to use Twitter to drive people to your Facebook page and website.

Discussions continued in the reception area after the seminar finished (and long after the rain had ceased) with people from companies such as Unibet, the BBC, Virgin Media and The British Medical Association keen to pick the minds of the One to One team.

Thank you to everyone who joined us!

Why measure time?

When conducting user experience studies, it is useful to measure certain aspects of the products performance, such as number of errors and level of user satisfaction. Time is a metric which seems like a no brainer because it is absolute, easy to measure, and easy to interpret. Timing how long a user takes to complete a process or activity gives a concrete figure which is sometimes easier for clients to accept than qualitative findings. Often a client may wonder how they can decide if a product has ‘passed’ or ‘failed’ a user test, or how performance can be increased to upturn KPI’s. Time is also a metric which fits into the ‘efficiency’ category, which, along with ‘effectiveness’ and ‘satisfaction’ makes up the ISO9241-11  standard for measuring usability.

Limitations of timing users

However, there are several considerations to make before time is to be used as a valid usability metric:

Methodology

If you are going to use ‘think aloud’ methodology, timing the participant is out of the question as the result will be skewed by moderator questions, time to think about and respond to these questions, and time to re-focus back to the original task at hand. Timing sessions requires using alternative methods

Context

Time taken has different meanings depending on the context. Some activities, like game playing, are used as a ‘time killer’, so although interesting, time spent is not a useful usability metric. Information seeking is equally hindered by times when the user is confused, interested, or distracted causing them to spend longer. Arguably the most useful context in which to time a user is when they are completing a registration process and moving through fields; here we can be fairly certain that the user does not want to be in this process, but have been forced to move through it to progress to a subsequent goal (banking, blogging etc). Therefore the time spent should be as low as possible.

Number of participants

Nielson recommends testing with 20 users when measuring time or other quantitative metrics to ensure that the confidence level is acceptable. In practice though, timing even a few users can provide a frame of reference when comparing multiple sites or versions of a site

Actionable data

If you know that it took users a mean time of 4 minutes to complete a process…is that good…bad…better than average? How much should you aim to reduce that by? What was the factor that caused them to take that amount of time? Having a comparison is necessary (either your site over time, or your site versus competitor sites.)

Practical advice

If the study is a benchmarking or performance assessment type, a time metric would be useful to indicate general efficiency. Whereas if the goal of the study is to uncover why the interface is performing badly, or to uncover usability problems and how to fix them, timing the session may prove less fruitful. In either case, the decision whether to time a usability session must be made very early in the process such as shortly after the kick off meeting; this way all involved parties are clear about how to prepare the test materials and what is expected of the reports.

If the session is to be timed, consider using retrospective think aloud which asks the user to explain their thought processes after the event (they remain silent whilst using the interface, either with or without a moderator present). Alternatively, ‘stimulated retrospective think aloud’, also uses a memory aid such as replaying a video of the actions they took on screen, to elicit the participants’ thoughts on what they did. This approach is more thorough but takes more time and resources.

Ultimately it should be understood that the usability session is a qualitative event and therefore the implicit and explicit actions of users can lead to more insight and actionable recommendations than a measure of how long users spent to complete a task.