What it is
A quantitative usability study that captures standardized metrics (success rate, time on task, satisfaction) with enough participants to compare across releases, versions, or competitors.
When to use it
During Implement, to establish a baseline and track change, or to compare against competitors.
Why (value)
Turns usability into trackable numbers, so you can prove improvement or regression over time with comparable data.
How to run it
- Choose standardized tasks and metrics.
- Recruit enough participants for statistical comparison.
- Run the study identically each round.
- Compute metrics with confidence intervals.
- Compare to baseline, prior rounds, or competitors.
Inputs and outputs
- Inputs: Fixed tasks and metrics, a sizable sample, a stable protocol.
- Outputs: Comparable usability metrics over time.
Time and effort
High. Larger samples and strict consistency are required.
Evidence and limitations
Produces quantitative direct-user evidence that is comparable over time, but only if the protocol stays identical; it measures the tested tasks, and numbers alone do not explain causes. See Evidence and limitations.
Tips and pitfalls
- Keep tasks and protocol identical across rounds.
- Report confidence intervals, not just means.
- Pair with qualitative sessions to explain changes.
Variations
Internal trend benchmarking and competitive benchmarking.
Related methods
Need help planning research or facilitating this method?
Gumshoe helps teams connect research, collaborative workshops and service design to the decisions they need to make.
Sources and further reading
- NN/g, “Benchmarking UX: Tracking Metrics.”
Explore related methods
- Usability testing — Watch representative users attempt real tasks to find where the design helps or gets in their way.
- UX metrics frameworks — Choose a coherent set of UX metrics tied to goals so you measure what actually matters.
- Analytics review — Examine real usage data to see what large numbers of users actually do.
