A/B Testing

Purpose

Compare two (or more) variants of a prototype or feature by randomly assigning users to each variant and measuring which produces better outcomes on pre-defined metrics. A/B testing replaces opinion (“I think version A is better”) with evidence (“Version A produces 23% higher task completion, p < 0.05”), enabling data-driven design decisions at A3.2 and controlled feature evaluation at A3.3.

In the ILF context, A/B testing serves two distinct purposes:

• A3.2: Compare prototype variants (e.g. two checkout flows) to determine which performs better in user testing
• A3.3: Compare feature variants within a live pilot (e.g. recommendation algorithm A vs. B) to optimise before A3.4 evaluation

When to Use

Use A/B testing when:

• Two or more prototype variants exist and the team needs evidence to choose between them
• The decision criterion is measurable (completion rate, time-on-task, conversion, retention)
• Sample size is sufficient for statistical significance (≥20 per variant for usability metrics; ≥100 per variant for conversion metrics)
• Random assignment is possible (users can be directed to variants without knowing)
• Feature flags (the referenced method) or testing platform supports variant assignment

Do NOT use when:

• Only one variant exists—A/B testing requires a comparison; use task-based usability testing (the referenced method) instead
• Sample size is too small (<20 per variant for usability; <100 per variant for conversion)—results will be inconclusive
• Variants differ on too many dimensions simultaneously (confounded variables—you won't know which change caused the difference)
• The decision is qualitative (brand perception, emotional response)—A/B testing measures behaviour, not sentiment
• Ethical concerns exist about differential treatment (e.g. healthcare, safety-critical systems)

Sample Size and Duration

A3.2 context: 20–25 per variant (40–50 total); 1–2 weeks

A3.3 context: 50–250 per variant; 2–4 weeks

Rule of thumb: If you cannot recruit enough users for statistical power, do not run an A/B test—use qualitative comparison (task-based testing + think-aloud) instead.

Prerequisites

• Two variants: Clearly defined A (control) and B (treatment) that differ on a specific, identifiable dimension
• Primary metric: One measurable outcome that determines the “winner” (e.g. task completion rate, conversion rate, retention)
• Sample size calculation: Minimum users per variant for desired statistical power (typically 80%) and significance level (typically = 0.05)
• Randomisation mechanism: Feature flags (the referenced method), testing platform (Optimizely, LaunchDarkly), or manual random assignment for moderated testing
• Duration estimate: How long to run the test to reach required sample size
• Analysis plan: Pre-registered: primary metric, statistical test, stopping rules

Complete Procedure

Step~1: Formulate Hypothesis (1–2 hours)

State the hypothesis in testable form:

“Version B (simplified 2-step checkout) will produce a higher task completion rate than Version A (current 3-step checkout), with minimum detectable effect of 15 percentage points.”

Define: primary metric (completion rate), secondary metrics (time-on-task, satisfaction), and minimum detectable effect (the smallest improvement worth detecting).

Step~2: Calculate Sample Size (30 minutes)

Use power analysis to determine users per variant. For usability metrics (task completion rate):

Step~3: Implement Randomisation (2–4 hours)

A3.2 (moderated testing): Randomly assign participants to Variant A or B before sessions begin. Use a random number generator; do not let the facilitator choose.

A3.3 (live pilot): Use feature flags (the referenced method) for server-side random assignment. Ensure: consistent assignment (same user always sees same variant), even split (50/50 unless otherwise designed), no selection bias (assignment independent of user characteristics).

Step~4: Run the Test (days to weeks)

A3.2: Test runs for the duration of Wave~1 or Wave~2 user testing sessions.

A3.3: Test runs for 2–4 weeks within the pilot period. Do not peek at results and stop early when they look good—this inflates false-positive rates. Use pre-committed stopping rules or sequential analysis methods.

Step~5: Analyse Results (2–4 hours)

• For completion rates: Chi-squared test or Fisher's exact test (small samples)
• For continuous metrics (time, score): Independent samples t-test or Mann-Whitney U (non-parametric)
• Report: Effect size, confidence interval, p-value. A result is “significant” when p < 0.05 and the effect size is practically meaningful

Step~6: Decide and Document (1 hour)

Three possible outcomes:

1. B wins (significant): Adopt Variant B. Document evidence.
2. No significant difference: Choose based on secondary metrics, implementation cost, or simplicity. Document that both are equivalent.
3. A wins (B is worse): Keep Variant A. Document why the hypothesis was wrong—this is valuable learning.

Quality Criteria

1. Pre-registered hypothesis: Primary metric and minimum detectable effect defined before test starts
2. Adequate sample: Power analysis performed; target sample reached before analysis
3. Proper randomisation: Users randomly assigned; no selection bias
4. Single variable: Variants differ on one identifiable dimension (no confounds)
5. Statistical rigour: Appropriate test applied; confidence interval and effect size reported (not just p-value)
6. Practical significance: Statistically significant result is also practically meaningful (a 0.1% improvement may be real but not worth the complexity)

Theoretical Foundation

Seminal references:

• : The definitive reference on online controlled experiments. Based on 20+ years at Microsoft, Amazon, and Google, Kohavi demonstrates that A/B testing is the only reliable method for establishing causation (not just correlation) between design changes and outcome improvements. Key insight: most ideas fail (60–90% of A/B tests show no improvement or negative results)—which is why testing matters.
• : Established the statistical foundations of controlled experiments: randomisation, control groups, significance testing. A/B testing is Fisher's experimental design applied to digital products.

Contemporary references:

• : Positioned A/B testing as the core mechanism of the Build–Measure–Learn loop: build two variants, measure which performs better, learn what users actually prefer (vs. what they say they prefer).

A3.2 vs. A3.3 Application

Challenges and Solutions

Challenge 1: Peeking and Early Stopping

Symptoms: Team checks results daily; stops test when p < 0.05 after 3 days (with only 30% of target sample).

Solutions: Pre-commit to sample size and duration. Do not analyse until target reached. If sequential monitoring is needed, use methods that control for multiple looks (e.g. Bayesian sequential testing, alpha spending functions).

Challenge 2: Too Many Variants (Multivariate Sprawl)

Symptoms: Test 5 variants simultaneously; insufficient sample per variant; no clear winner.

Solutions: Limit to 2–3 variants maximum. Each additional variant requires proportionally more users. In A3.2 (limited sample), stick to A/B (two variants only).

Challenge 3: Novelty Effect

Symptoms: Variant B outperforms in Week~1 (users explore the new design) but performance equalises by Week~3.

Solutions: Run A3.3 tests for ≥2 weeks. Analyse by cohort (early adopters vs. later users). If novelty effect is suspected, extend test duration.

Relationship to Other Methods

A/B Testing receives input from:

• Feature Flags (the referenced method): Enable variant assignment in live pilot environments
• Task-Based Usability Testing (the referenced method): Provides the measurement framework (tasks, metrics) for A3.2 A/B comparisons

A/B Testing provides input to:

• A3.4 Evaluation: Variant comparison evidence for desirability lens
• Cohort Analysis (the referenced method): A/B results can be segmented by cohort to understand differential effects
• A3.5 Decision: Clear evidence that one approach outperforms another supports go/no-go

A/B Testing is complemented by:

• Think-Aloud Protocol (the referenced method): A/B testing shows which variant wins; think-aloud reveals why
• Controlled Rollout (the referenced method): After A/B test determines winner, controlled rollout gradually deploys it

Tools and Templates

• A3.2 (moderated): Random assignment spreadsheet; manual analysis in Excel/R/Python
• A3.3 (live pilot): Optimizely, LaunchDarkly, Google Optimize, Statsig, Split.io (experiment platforms with built-in analysis)
• Sample size calculators: Evan Miller's calculator, G*Power, Optimizely sample size tool
• Analysis: R (power.prop.test, chisq.test), Python (scipy.stats), Excel (basic)

• E. Ries (2011). The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business.
• J. Sauro & J. R. Lewis (2016). Quantifying the User Experience: Practical Statistics for User Research. 2 ed. Morgan Kaufmann.
• R. Kohavi, D. Tang & Y. Xu (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
• Fisher (1935). fisher1935design.