What it is
A controlled experiment that varies multiple page elements simultaneously and measures how combinations affect an outcome, revealing both individual and interaction effects.
When to use it
During Implement, with high traffic, when you want to understand how several elements interact rather than test one change.
Why (value)
Reveals interaction effects between elements that sequential A/B tests would miss, optimizing combinations.
How to run it
- Choose the elements and their variations.
- Generate the combinations to test.
- Split traffic across combinations.
- Run until each combination has enough data.
- Analyze main and interaction effects.
Inputs and outputs
- Inputs: Multiple element variations, high traffic, a metric, a platform.
- Outputs: Winning combination and element-level effects.
Time and effort
High. Needs much more traffic than A/B testing.
Evidence and limitations
Gives causal direct-user behavioral evidence including interactions, but combinatorial explosion demands large samples; underpowered MVT yields noise. See Evidence and limitations.
Tips and pitfalls
- Confirm you have the traffic before starting.
- Limit elements to keep combinations tractable.
- Prefer A/B tests when you only need one answer.
Variations
Full-factorial and fractional-factorial designs.
Related methods
Need help planning research or facilitating this method?
Gumshoe helps teams connect research, collaborative workshops and service design to the decisions they need to make.
Sources and further reading
- 18F Methods (US government, public domain).
Explore related methods
- A/B testing — Compare two live variants with real traffic to see which performs better on a chosen metric.
- Analytics review — Examine real usage data to see what large numbers of users actually do.
- UX metrics frameworks — Choose a coherent set of UX metrics tied to goals so you measure what actually matters.
