Qualitative or Quantitative?

by Feb 15, 2001, 5:50 AM

• Comment
• Recommend (2)
• Tweet
• 

Qualitative or Quantitative?

Jakob Nielsen’s Alertbox, reported in useit.com recently, suggested some specific criteria to decide when and how to use quantifiable data in evaluating usability alternatives. He suggests that usability is rarely measured because metrics are expensive and are a poor use of typically scarce resources. Generally, to improve a design, insight is better than numbers.

However, collecting actual measurements is a natural next step and does provide benefits. In general, usability metrics let you:

- Track progress
- Assess your competitive position
- Make a Stop/Go decision before launch
- Create bonus plans

Typically, usability is measured relative to users' performance. The most basic measures are: - The time a task requires
- The error rate
- Users' subjective satisfaction

When collecting usability metrics, in order to get a reasonably tight confidence interval on the results, the author recommends testing 20 users for each design.

To illustrate quantitative results, look at those recently posted by Macromedia from its usability study of a Flash site. Macromedia took a design, redesigned it according to a set of usability guidelines, and tested both versions with a group of users. Here are the results:

+---------------------+-----------+----------+
|                     |  Original | Redesign |
|                     |    Design |          | 
+---------------------+-----------+----------+ 
| Task 1              |   12 sec. |   6 sec. | 
| Task 2              |   75 sec. |  15 sec. | 
| Task 3              |    9 sec. |   8 sec. | 
| Task 4              |  140 sec. |  40 sec. | 
| Satisfaction        |     44.75 |    74.50 | 
| (Measured on a      |           |          | 
| scale ranging from  |           |          | 
| 12 (unsatisfactory) |           |          | 
| to 84 (excellent)   |           |          | 
+---------------------+-----------+----------+ 

Given that the redesign scored better than the original design on all measures, there is no doubt that the new design is better than the old one. However, in many cases, results will not be so clear cut. In those cases, it's important to look in more detail at how much the design has improved.

The remainder of the article is devoted to the mathematical extrapolations necessary to separate the wheat from the chaff.

For those researchers handling the support of design strategies, plans, campaigns, placements, etc. the complete report is located here.

• Comment
• Recommend (2)
• Tweet
• 

Be the first to comment on "Qualitative or Quantitative?"

Leave a Comment