Intranet Journal
The online resource for intranet professionals
This study was different from most other studies we've done in the past in that we included two Cisco TAC engineers (of six total participants) because they are heavy users of the support pages. We gave our list of criteria to Breakthrough Usability, which outsources the user recruiting. They recruit participants from a TAC Web database of thousands of customers and partners who have signed up online to be beta testers. Built into the cost of our study is compensation for the Cisco customers and partners who participate.
How are users compensated?
We offer recruits between $150-250 for 1.5-2 hours of their time. We do this to reduce the
number of no-shows. Network administrators typically find it tough to get away, so we have
to compensate them for taking the time to come to our office. The participants who show
up for the study receive either a check or a gift certificate upon completion of the study.
How did you proceed after recruiting subjects?
Once the recruiting effort was underway, we defined the issues for which we needed
customer data and Breakthrough Usability used our outline to build the test format.
The format for this particular test consisted of three parts.
What were the three parts?
The first portion of the test presented two common task scenarios derived from prior customer
feedback. The test participants were asked to complete the tasks using two new support page
designs. The second part deals with gathering feedback on how our users would like to see the pages
formatted. The final part deals with presenting two alternatives for product-level categories on the pages.
How do you measure usability in task study?
There are two ways to measure usability in a task study: quantitative and qualitative. The
quantitative study measures how successful the user was in getting to a correct solution. An
interface that enables its users to be successful a certain percentage of the time is considered
successful. The specific percentage used to determine success varies depending on the
requirements for the project. It should be defined before the results come back. In this
section of the study, we used quantitative metrics in conjunction with a more qualitative
approach.
The qualitative approach compiles the user's response to various aspects of the Web pages and evaluates preference. In a comparison study like we did, it is not uncommon that a user will prefer the interface with which she has not been as successful, though we hope for agreement between success and preference. Users sometimes don't know what interface will make them most successful, especially having only used a prototype for a few minutes in an artificial situation. Some human-computer interaction experts will rely only on quantitative data, but I find that for iterative design, it is quite enlightening to ask users what they like and dislike about various aspects of the interfaces. Conversely, a pure quantitative study is great for deciding whether a Web application is ready to be launched or if it needs more work.
What do you present to users during the studies?
In this study, we reused some old code and were able to use fully functional prototypes,
though only the two products relevant to the tasks were complete. This is much more than
we typically build for a usability study. Typically, we use either paper prototypes or online
mockups to reduce the investment at this stage in the process. We may find that we need to
dramatically change the application, in which case it's only a matter of modifying some
HTML mockups, rather than rewriting a Perl or Java application. Having lightweight
mockups also allows us to make changes to the mockups during the day of the usability
study if we find a major problem early on.
Will you address the part of the study dealing with page formatting?
We gathered feedback on how our users would like to see the pages formatted. We showed
them mockups of various methods for displaying pagination, media type, content type, and
opening links in a new window.
The final section of the study presented two alternatives for technology-level categories.
For example, the current categorization for technologies on the TAC Web site is (as found
on any of the technologies listed at www.cisco.com/tac/technologies:
At a prior stage, we may have done a sorting task to determine users' mental models when thinking about Cisco product support. At this stage, however, we wanted to compare the model above to an alternate approach that had been suggested. In order to compare the two models, we had the users think of tools or resources they need when they go to Cisco's TAC Web site. We then asked them to assign these resources to a category in each of the two models. We were able to gather quantitative data on how accurate users were in assigning resources to categories. We also were able to ask a few questions about preference to get qualitative data.
What results have you gathered so far?
The most important finding was that we were letting "feature-creep" get in the way of
usability. We added in lots of bells and whistles, like sorting and document rating, but these
were either not noticed or criticized for getting in the way of the task at hand. Our next
iteration will have a simpler interface. Because we had not invested any time developing
the systems to support these bells and whistles, we could easily discard those ideas without
wasting precious development time.
In terms of the organizational categories we were testing at the technology level, we learned
that there were successful aspects in both of our models, so our next test will include a new
model that aggregates the best points of both.
Excerpt From Chapter 10 of :
eSupport : How Cisco Systems' Saves Millions While Improving Customer Support
ISBN : 158720-052-X
www.ciscopress.com
For a detailed description of this title, click here