CHI 2009 – Day 3

I’m really enjoying the fact that CHI is in Boston this year.  I can sleep in my own bed and go into work for half the day if need be.  So, it was a light session day, but the one full session I saw was fantastic: Usability Methods

Comparison of Three One-Question, Post-Task Usability Questionnaires – Oracle

This paper was presented by Jeff Sauro and Joe Dumas.  Joe (Dr. Usability) is a pioneer in the field of usability testing and was a Bentley professor until he recently retired – I justmissed having him as a professor.  Their study sought to determine which post-task questionnaire type is the best to use: a Likert scale, the Subjective Mental Effort Questionnaire (SMEQ), or Usability Magnitude Estimation (UME).  There were four criteria use to determine which scale was the “best:”

• Validity – Does it do what it’s supposed to do?
• Ability to discriminate – Does it find what’s important?
• Sensitivity – Does is discriminate with different sample sizes?
• Easy to use – from both the participants’ and testers’ view

To make a long story short, SMEQ performed the best out of the three questionnaire types.  The SMEQ is a simple vertical line with 0 at the bottom and 150 at the top.  0 indicates that the task was not at all hard to do and ~115 indicates the task was tremendously hard to do.  The participant simply moves a horizontal line (in a Flash interface) to the appropriate point.  This certainly keeps it simple – I think it is worth investigating some more and integrating into usability studies.

Correlations among Prototypical Usability Metrics: Evidence for the Construct of Usability – Oracle and IBM

Jeff Sauro was a busy man during this session, as he presented this one as well – and it was another gem.  The authors suggested that we may be able to determine a usability construct.  Something along the lines of

u = effectiveness + efficiency + satisfaction

They recounted various correlations between errors, completion time, and user satisfaction.    These measures were considered at task level and at the test level.  The two highest of the correlations were between error & time and task satisfaction & test satisfaction (unsurprisingly).  Overall, I think this paper gives some nice groundwork for the creation of a construct that will help us usability practitioners translate qualitative data into quantitative data.

Let Your Users Do the Testing: A Comparison of Three Remote Asynchronous Usability Testing Methods- Aalborg U. (Denmark) & others

The authors compared three asynchronous remote usability testing methods: user reported critical incident (UCI), a user forum, & a user diary.  They used a traditional usability study in a lab as a benchmark.  The lab study found the most usability problems of the four methods used, and there was no significant difference in the number of problems found between the remote methods.  UCI and the diary found a significant number of problems and the lab and the diary had the most agreed upon problems.  Overall, this presentation was interesting but not terribly useful.  If you want to find the most usability problems, run the study in the lab.  If you can’t, don’t use a user forum as your remote method.