Audit of bandit campaign-mode allocation

Simple terms

"Send more traffic to the winning variant while the promo runs" sounds like free money. We simulated over a million short merchant campaigns to measure how much money it actually is, and what it does to the numbers you see afterwards. The money is small. The distortion of the reported result is large, unless the bandit is built with two specific safeguards.

The setup

The audit simulated two-variant campaigns under realistic e-commerce conditions:

• Campaign lengths of 3, 7, and 14 days; 2,000 to 50,000 visitors; conversion rates of 1% to 3%; true lifts of 0%, +5%, +10%, and +20% relative; heavy-tailed order values for revenue scenarios.
• Realistic frictions: delayed purchases (most purchases do not land in the same session the visitor was assigned in), hour-of-day and weekday seasonality, sale-shaped demand drift, and a scenario where the winner alternates — better on weekends, worse on weekdays.
• Allocation policies: a fixed 50/50 split, epsilon-greedy, Thompson sampling, and Thompson sampling with a hard 10% minimum allocation per arm.

The statistics reuse Apex's production decision code — the same sequential method, seed derivation, and heavy-tailed order-value model used in the Apex certification. The run covered 233 policy-scenario cells and 1,156,800 replications with fixed seeds; re-running reproduces every number byte-for-byte.

Result 1 — the money is small

The entire prize is bounded by hindsight: routing 100% of traffic to a true +10% winner for the whole campaign is only worth +4.76% over a 50/50 split. A bandit has to spend the early campaign learning, so it captures a fraction of that.

Scenario	Maximum possible gain over 50/50	Thompson sampling captures
7 days, +5% true lift, 10k visitors	2.44%	12%
7 days, +10% true lift, 10k visitors	4.76%	22%
7 days, +20% true lift, 10k visitors	9.09%	40%
7 days, +10% true lift, 50k visitors	4.76%	43%

In money: a typical 7-day, 10,000-visitor campaign with a genuine +10% winner earns +131 EUR on roughly 22,000 EUR of campaign revenue. The best case tested (50,000 visitors) earns +1,371 EUR on roughly 110,000 EUR.

When there is no real winner, the gain is exactly 0.00% — always. If the arms are identical, no allocation can beat any other. The bandit's entire value is conditional on a real, stable difference existing.

Result 2 — the bandit picks a side even when there is nothing there

In an A/A campaign — two identical variants — the allocation still drifts:

Campaign length	Campaigns ending visibly concentrated on one arm (Thompson)	(epsilon-greedy)
3 days	75%	87%–90%
7 days	80%	94%–96%
14 days	82%	96%–98%

The merchant watches the dashboard tilt toward "the winner" all week, and there is nothing there. With a real but small winner (+5%), Thompson ends the campaign pointed at the losing arm 28.5% of the time; epsilon-greedy 38.8%.

That matters beyond the campaign. A merchant who ends the week with 80% of traffic on variant B concludes B won and ships it permanently. A permanent decision worth tens of thousands of euros per year gets made from an allocation signal that is wrong roughly one time in three at small effect sizes.

Result 3 — the reported lift more than doubles

Adaptive allocation biases the measured lift upward, because traffic concentrates on whichever arm is currently lucky, and that arm's measured rate inherits the selection.

Policy, 7 days, true lift +10%	Reported lift (typical)
Fixed 50/50	about 11%
Thompson sampling, uncorrected	about 24%
Thompson + 10% floor + corrected estimate	about 13%

Two findings worth stating plainly:

• The obvious fix makes it worse. Stratifying the estimate by time (hour or day) increases the bias under an unfloored bandit, because it up-weights exactly the periods where the starved arm has almost no data.
• The correct estimator only works with a floor. Inverse-probability weighting — re-weighting outcomes by the allocation probabilities the platform itself set — is unbiased in principle, but with unfloored allocation its variance explodes and the estimate is useless. With a hard 10% minimum per arm, the corrected estimate lands within about 2 points of the honest 50/50 baseline, at roughly 1.5 times the noise.

The minimum-allocation floor is therefore not a safety nicety. It is the precondition that makes an honest lift number computable at all.

The alternating winner is pure cost

When the true winner alternates — variant B better on weekends, slightly worse on weekdays — every adaptive policy earned zero or negative money against 50/50, while still ending on the wrong arm 27%–43% of the time and inflating the reported lift by up to 10 points. A bandit chasing a moving winner is pure cost.

Promotional weekends are precisely where weekday/weekend interactions are most plausible — which is the same window where campaign-mode features are marketed hardest.

When adaptive allocation earns anything

All of these must hold at once:

Condition	From the data
A real, stable winner exists	0% lift pays exactly 0.00%
The effect is large	+20% pays about +3.6%; +5% pays about +0.3%
Traffic is high	50k visitors captures 43% of headroom; 2k captures about 10%
The winner does not alternate	An alternating winner pays zero or negative
The campaign is long enough to learn	3-day campaigns leave most of the prize uncollected

Questions to ask any platform selling auto-optimization

1. What does the allocation do on an A/A test — does it drift toward a "winner" that is not there, and is that shown to the user as signal?
2. Is there a hard minimum allocation per arm, and what is it?
3. Is the reported lift corrected for adaptive allocation, or is it the raw dashboard number? By how much do the two differ on a simulated known-truth campaign?
4. Can harm on the losing arm still be detected once that arm has been starved of traffic?
5. Can the vendor show a deterministic audit with stated seeds and scenario grid?

Where Apex stands

Apex does not ship adaptive allocation. This audit is the reason, with numbers: for typical campaign sizes the honest expected gain is around 1% of campaign revenue, while an unsafeguarded bandit distorts the ship/no-ship decision that is worth far more than the campaign itself. If a campaign mode ever ships, it ships in the only configuration the data supports — Thompson sampling with a hard allocation floor, slow reallocation, and a corrected lift estimate — and the uncorrected number is never shown.

Interpretation limits

The study covers two-arm campaigns; bandits gain more with many arms, and that shift was not measured. Returning-visitor experience under changing weights was not priced. Hourly traffic follows a fixed seasonality curve rather than being random itself. Results apply to the tested grid of traffic, lift, delay, and seasonality scenarios; the full per-cell output and the simulation code are archived with the audit.