The test: we added urgency messaging to the ancillary selection screen. 'Only 3 meals left at this price'. It was real scarcity — the meals were genuinely limited.
2-week A/B test: +8% meal attachment. Statistical significance: 97%. We shipped.
6 months later, support tickets about 'misleading' offers had increased 40%. A segment of frequent flyers — our most valuable users — had started selecting 'no meal' specifically to avoid what they perceived as pressure tactics.
The conversion uplift was real. The trust damage was also real. A/B tests measure what users do in the short term — not what they feel about the product long term.
What changed: we kept the scarcity signal but made it genuinely informational rather than urgency-triggering. 'Meals are limited on this route' rather than 'Only 3 left!'. Lower conversion. Higher satisfaction. Better retention.
The Takeaway
Test for behaviour and sentiment. A metric that goes up while another goes down isn't a win — it's a tradeoff you need to understand before you ship.