Here’s some thoughts on some usability testing methods I’ve used and when they are useful.
Cognitive Walkthrough
Ok, this isn’t usability testing. It doesn’t involve users. But its the next best thing because it actively makes you use your interface with a mindset that’s as close as you can get to being a user.
A cognitive walkthrough is where you (the UX professional) attempts key tasks with the interface imaging that he or she is a user. To aid this process, you make reference to your personas and scenarios. Its best to do it somewhere quiet where you won’t be interrupted.
If you don’t use this then there’s a chance that you’ll have a lovely set of personas and scenarios that remain largely disconnected from your design process. Cognitive walkthroughs provide a mechanism to expose the design to the personas and scenarios.
This forces you to approach the website from different points of view and different needs. Its not a substitute for involving real users though. Instead it should be part of your early design process.
Guerilla testing
This is where you approach someone in a cafe with a laptop, offer to buy them a coffee and get them to try a quick task with your interface. This is typically 5-10 minutes. Record the session using Silverback. Its probably a good idea to get permission from the cafe owner beforehand 
Good points
Its very very cheap and brilliant for evaluating a specific part of a design due to time constraints. Its suitable for an interface that is a finished product or at least a working prototype on a laptop.
Drawbacks
You can’t really evaluate a whole website or software product like this. You don’t get the client’s buy in so well because they can’t see it live in an observation room. Watching recordings after the fact is seldom as compelling and the environment isn’t as “scientific” so for some this adds less credibility to the findings. This is important when you have to resolve differences of opinion or reluctance to acknowledge the problem. If you’re lucky enough not to have to contend with this, then good for you.
This approach also works better when you actually have a working prototype. I haven’t tried it with paper prototypes but I imagine it would just be too messy because of all the physical props and bits of paper needed.
Informal testing with paper prototypes
This is where you get users to attempt real tasks using paper mockups of your website. These are usually derived from your early wireframes. You want just enough detail for the design to make sense to them. Its almost half way between an interview and a usability test because there are often gaps in the design and pauses in the flow to find and produce different screen states.
I use a meeting room and schedule participants much the same way as for formal usability testing but its more informal. Tests can be run with one or two consultants depending on budget. Having a note taker as well simply improves the quality because you’re not trying to take notes and facilitate at the same time.
Good points
Its pretty cheap compared to getting the design wrong and only finding out later or formal usability testing (see below). This gets you some valuable insights, early design validation and feedback from users. Its more about involving users in developing your design than producing formal reports on design problems. Its a great input to have early so you don’t go on to waste time refining a design that has major problems. I see it as an essential part of the early design process for anything complex. “Get them in early and often” as someone once said.
Drawbacks
It is informal and doesn’t lend itself to easy capturing and transmitting of the interactions without a lot of setup work. Pieces of paper put in front of participants can’t be recorded so easily as screen capture. So only you and your notetaker can really see what’s going on. This isn’t so good if your goal is to share the evidence with a wider group of people that a design needs fixing or that certain elements of the design don’t work. It can also be hard with complex software interactions as your manual intervention to replace parts of the interface can really slow things down and disrupt the user’s flow. You can’t fully model the manner of the interactions, just the rudimentary states.
For more about this, see my article: “Why you should test early with paper prototypes” (over on the LeftClick Blog).
Formal Usability Testing
This is the traditional approach where you have the full usability lab setup. Its what people usually think of when you mention “usability testing”: a testing room, video camera, observation room, formal testing scripts, appoints for participants, a facilitator, a note taker – the works. There’s plenty of articles and books out there about this approach. This would be with with the actual software of with a mockup of the new design, or a blend of both.
Good points
Getting a really indepth idea of where a design is falling down and sharing this with whichever stakeholders matters via direct observation. You can measure all kinds of in depth statistics. You can get quantitative with stats like time to complete each task and you can track eye movements and make it very very scientific if needs be. I have rarely needed to go this far though to gain enough insights to arrive at a solid design decision.
Drawbacks
Its really expensive and time consuming due to the indepth approach, cost of running or hiring the lab, time to setup, scheduling participants, formal note taking, formal reporting, organising observers etc. You really don’t want to get into all this if 90% of it is going to be testing the bleeding obvious and you don’t have to prove anything major to stakeholders. If you just want to inform your design process, test the 10% you’re not sure about and save a whole lot of time and cost.
Remote usability testing
Sometimes your users are far away. If you have at least a working prototype then you can consider remote testing. The best approach I’ve found is to use Skype screen share in conjunction with Silverback – an approach which was first suggested to me by Nick Bowmast in his blog post “Skype Takes the hassle out of Remote Usability” .
Good points
Its certainly a lot cheaper than flying to where your users are. It saves using or needing a lab, allows users to be in their natural environment.
Drawbacks
Given a choice, face to face is always better for me. You can build a rapport with the person, see non-verbal cues better and make them feel more at ease. Users need to be literate with Skype and have accounts. And Skype has a different screen share button in every place for different versions across different platforms so you can sometimes burn test time just trying to get the damn thing going. You also need a fully working prototype because as far as I can tell, you can’t test paper prototypes remotely
The factors to think about
So I’ve found the right testing approach depends mostly on:
- the design problem to be tested
- the politics involved
- the budget
- the part of the design cycle you’re in (eg. if all you have is paper prototypes then the testing approaches are pretty much dictated by that).
More approaches for the toolbox
I was very inspired by Christine Perfetti’s talk on “Adventurous Usability Techniques” at this year’s Webstock. Christine illustrated a very focused approach to testing and suggested a range of new techniques. Time spent was specifically to solve certain design problems and I’ve begun exploring these more in my consultancy. Focused incremental testing as part of the design process using a broader toolkit is the future I think.
