Gumshoe · Method reference

A/B testing

Compare two live variants with real traffic to see which performs better on a chosen metric.

Conceptual illustration of a/b testing.
Effort
High
Participants
Needed
Remote friendly
Yes
Evidence / output
Direct user

What it is

A controlled online experiment that randomly serves two or more variants to live users and measures which produces a better outcome on a defined metric.

When to use it

During Implement, when you have enough traffic and a clear metric, to decide between concrete alternatives.

Why (value)

Provides causal evidence about which variant performs better under real conditions, settling debates with data.

How to run it

  1. Form a specific hypothesis and success metric.
  2. Build the variants and split traffic randomly.
  3. Run until you reach adequate statistical power.
  4. Analyze for a significant difference.
  5. Ship the winner or iterate.

Inputs and outputs

  • Inputs: Variants, sufficient traffic, a metric, an experimentation platform.
  • Outputs: A statistically grounded winner and effect size.

Time and effort

High. Needs instrumentation, traffic, and run time.

Evidence and limitations

Strong causal direct-user behavioral evidence for the tested metric, but it needs volume, only compares what you built, and can optimize local metrics while missing the why. See Evidence and limitations.

Tips and pitfalls

  • Define the metric and duration before you start.
  • Do not peek and stop early at a false win.
  • Watch for metrics that improve locally but harm the whole.

Variations

A/B, A/B/n, split-URL, and server versus client-side tests.

Need help planning research or facilitating this method?

Gumshoe helps teams connect research, collaborative workshops and service design to the decisions they need to make.

Discuss your project with Gumshoe Explore Gumshoe services

Sources and further reading

  • NN/g, “A/B Testing.”

Explore related methods

  • Multivariate testing — Test several element changes at once to learn which combinations drive the outcome.
  • Analytics review — Examine real usage data to see what large numbers of users actually do.
  • UX metrics frameworks — Choose a coherent set of UX metrics tied to goals so you measure what actually matters.

A general practice reference. This page does not establish that a method was used on a particular client project or that an outcome was measured.