A/B Testing for ML
Why offline metrics are not enough
When you train a new model you measure it on a held-out test set: accuracy, AUC, F1, RMSE, NDCG. These offline metrics tell you the model fits historical data better. They do not tell you that deploying it will actually improve the things a business cares about — clicks, conversions, revenue, retention, watch time. A model can score higher offline and still make the product worse online.
The gap exists for many reasons: the offline metric is a proxy for the real objective; the training distribution differs from live traffic; the model changes user behavior in ways the static test set never captured (feedback loops); or a higher AUC simply does not move the downstream KPI. The only way to know the causal effect of shipping a model is to ship it to some users and not others, and compare what actually happens. That experiment is an A/B test (a randomized controlled trial).
Offline = correlation with past data. Online A/B = causal effect on real outcomes. Offline evaluation is a fast, cheap filter to decide which candidates are worth testing online; the A/B test is the verdict.
The setup
The core idea is simple. Randomly assign each incoming unit (usually a user, sometimes a session or request) to one of two groups:
Control (A)
Keeps the current production model (or no model). This is the baseline you compare against.
Treatment (B)
Gets the new candidate model. Everything else is held identical — only the model differs.
Because assignment is random, the two groups are statistically equivalent on average in every respect except the model. So any difference in the chosen target metric (e.g. conversion rate) can be attributed to the model rather than to confounding differences between the users. You pick one primary metric, decide the sample size in advance, run the experiment, and then compare.
Statistical foundations
Observed rates wobble because of sampling noise. Even two identical models will show slightly different conversion rates over a finite sample. Hypothesis testing is the framework for asking: is the gap we see bigger than noise would plausibly produce?
Null vs. alternative hypothesis
The null hypothesis H₀ says there is no real difference (rate of B = rate of A). The alternative H₁ says there is a difference. We assume H₀ and ask how surprising the data would be under it.
p-value
The p-value is the probability of observing a difference at least as extreme as the one we got, assuming H₀ is true. A small p-value means the data would be unlikely under “no effect,” which is evidence against H₀.
Significance level α
We pick a threshold α (commonly 0.05) before running. If p < α we reject H₀ and call the result “statistically significant.” α is exactly the probability of a false positive (Type I error) — declaring a winner when there is none.
Power (1 − β)
Power is the probability of correctly detecting a real effect of a given size. A Type II error (probability β) is missing a real effect. Power grows with sample size and with effect size. A common target is 80% power. An underpowered test that “finds nothing” is uninformative — it may simply have been too small.
Sample size & minimum detectable effect (MDE)
Before launching you choose the smallest lift worth caring about (the MDE), your α, and your target power, and from those compute the required sample size per arm. Smaller MDE ⇒ much larger n. For a two-proportion test the required per-arm size scales roughly like n ≈ (zα/2 + zβ)² · 2p̄(1−p̄) / δ², where δ is the MDE — note the 1/δ²: halving the effect you want to catch roughly quadruples the sample you need.
Confidence intervals
A 95% CI is a range constructed so that, over many repeated experiments, 95% of such intervals contain the true value. Reporting the CI on the lift (e.g. “+0.8 pp, 95% CI [−0.1, +1.7]”) is more informative than a bare “significant / not.” If the CI for the difference excludes 0, the result is significant at that level.
Crucial subtlety about p-values: the p-value is not the probability that H₀ is true, and it is not the probability your result is a fluke. It is computed assuming H₀ holds. “p = 0.03” means “if there were truly no effect, data this extreme would occur 3% of the time,” not “there is a 97% chance B is better.” A large p-value also does not prove H₀; absence of evidence is not evidence of absence.
Interactive: A/B test simulator
Set the true conversion rates for A and B (in reality you never know these — here we do, so we can see what the test gets right and wrong) and the per-variant sample size. We draw real Bernoulli outcomes, plot the observed rates with 95% confidence intervals as error bars, and run a proper two-proportion z-test. Watch how sample size controls whether a real difference is detectable.
Try this
Set A = 10% and B = 11% (a true 1 pp lift). At n = 2,000 the confidence intervals overlap and the test is usually not significant — you would wrongly conclude “no difference” even though B really is better. This is a Type II error: the test is underpowered. Now drag n up toward 30,000–50,000 and watch the intervals shrink until the true difference becomes reliably detectable. The smaller the effect you want to catch, the larger the n you need.
The two-proportion z-test
When the metric is a rate (converted / did-not-convert), the standard test for A vs. B is the two-proportion z-test. With successes x and sample sizes n in each arm:
Under H₀ the difference in observed rates is approximately normal with mean 0 and standard deviation SE, so z measures how many standard errors apart the two arms are. The simulator above computes exactly these formulas — nothing is faked.
Why peeking is dangerous
The α = 0.05 guarantee — at most a 5% false-positive rate — holds for one test at a pre-committed sample size. If instead you watch the dashboard and check significance over and over as data trickles in, stopping the moment you see p < 0.05, you are running many tests. The p-value fluctuates randomly; given enough peeks it will eventually dip below 0.05 by chance even when nothing is happening. Early stopping on the first “win” therefore badly inflates the false-positive rate. The demo below shows this on an A/A test where the two arms are provably identical.
Here both arms are identical (true rate = 10% for A and B, so H₀ is true). The line is the running two-sided p-value, re-computed every 100 samples per arm as data streams in. The red dashed line is α = 0.05. Any dip below it is a false positive — there is no real effect to find.
Common pitfalls
Peeking / early stopping
Repeatedly testing as data accumulates and stopping at the first significant result inflates Type I error far above α. Fix n up front, or use sequential / always-valid methods (e.g. group sequential tests, Bayesian, or e-values) designed for it.
Multiple comparisons
Testing many metrics or many variants multiplies the chance that something looks significant by luck. With 20 independent metrics at α = 0.05 you expect about one false positive. Pre-register a single primary metric, or correct (Bonferroni, FDR).
Novelty & primacy effects
Users may react to a change simply because it is new (a temporary bump), or be slow to adopt it. Short tests can over- or under-estimate the steady-state effect. Run long enough to let novelty wear off.
Simpson's paradox
A trend in aggregate can reverse within every subgroup if the groups are mixed in different proportions across arms (e.g. a skewed split by platform or country). Check that randomization is balanced and analyze key segments.
Network / interference effects
A/B tests assume one unit's treatment does not affect another's outcome (SUTVA). That breaks for social networks, marketplaces, or shared inventory, where treated users spill over onto control. Use cluster-level randomization or geo-experiments.
Sample-ratio mismatch
If a 50/50 split arrives as, say, 52/48 at scale, something is broken in assignment or logging — and the comparison is suspect. A chi-square check on the split sizes is a cheap, essential sanity test before trusting any result.
Beyond a simple A/B: shadow, interleaving, guardrails
Shadow deployment
The new model B receives real production traffic and produces predictions, but its outputs are logged and never served to users — the user always sees A. This validates latency, throughput, error rates, and prediction distributions under live load with zero user risk. It cannot measure causal impact on user behavior (users never experience B), so it is a pre-launch safety and parity check, not a replacement for an A/B test.
Interleaving (for ranking / search)
For ranked lists, instead of showing some users ranker A and others ranker B, interleaving blends both rankings into a single list shown to every user, then attributes clicks to whichever ranker contributed each item. Because each user compares both rankers within the same session, it controls for between-user variance and is dramatically more sensitive — it can detect ranking differences with far less traffic than a standard A/B test. (It measures relative preference between rankers, so it complements rather than replaces an A/B test on end metrics.)
Guardrail metrics
Besides the primary metric you want to improve, you monitor guardrail metrics that must not get worse: latency, crash rate, error rate, revenue, unsubscribe / churn, complaints. A model that lifts clicks but tanks latency or revenue should not ship. Guardrails catch the cost side of a tradeoff that a single success metric would hide.
Offline vs. online metric mismatch
A recurring lesson in production ML: the offline metric and the online outcome often do not agree, and reconciling them is part of the job.
| Offline metric | Online metric | Why they can diverge |
|---|---|---|
| AUC / log-loss of a click model | Click-through rate, dwell time | Better calibrated probabilities need not change what gets shown enough to move behavior; ranking quality ≠ ranking probability quality. |
| Recommendation NDCG on logged data | Engagement, retention, revenue | Logged data is biased by the old model (you only see what it showed); the new model surfaces items with no historical feedback. |
| Higher accuracy on a fraud model | Dollars saved, false-decline rate | Accuracy ignores the asymmetric cost of false positives vs. false negatives that the business actually feels. |
The practical workflow: use offline metrics to shortlist candidates cheaply, then let the A/B test on the real KPI make the final call. When offline and online disagree repeatedly, treat it as a signal that your offline metric is the wrong proxy.
Key Takeaways
- Offline metrics (accuracy, AUC, NDCG) measure fit to past data; only an online A/B test measures the causal effect of a model on real user and business outcomes.
- Randomly split traffic into control (A) and treatment (B), pick one primary metric, fix the sample size in advance, then compare.
- A p-value is the probability of data this extreme assuming H₀ is true — it is not the probability the null is true, nor the chance your result is a fluke.
- Power and sample size matter as much as α: an underpowered test that “finds nothing” may simply have been too small to detect a real effect. Required n scales like 1/δ² in the effect size.
- Peeking / early stopping inflates false positives far above α; so do multiple comparisons. Fix n ahead of time, pre-register one primary metric, or use tests built for continuous monitoring.
- Watch for novelty effects, Simpson's paradox, network interference, and sample-ratio mismatch — each can invalidate a naive comparison.
- Shadow deployment de-risks launches (run B without serving it); interleaving compares rankers far more sensitively; guardrail metrics ensure a win on one metric is not a loss everywhere else.