As AI automates buying and creative, the question "what actually worked?" gets harder — and more important. Platform dashboards and last-click attribution tell you what happened next to an ad, not what the ad caused.
The classic warning
In a large field experiment published as an NBER working paper, economists Tom Blake, Chris Nosko and Steven Tadelis turned off eBay's brand-keyword search ads in some regions. They found brand-keyword ads had no measurable short-term benefit — the people clicking would have found eBay anyway — and that frequent users, who drove most ad costs, were not influenced by the ads. Attribution had credited ads for sales they didn't cause.
Three tools that measure causation
- Experiments and holdouts: switch a channel or campaign off (or on) for a random group of people or regions and compare. The gold standard for a specific question.
- Marketing mix models (MMM): statistical models of how spend across channels relates to sales over time, accounting for seasonality, price and promotions. Good for budget allocation across channels. Google made Meridian, its open-source MMM, available to everyone in January 2025; other open-source and commercial options exist.
- Calibration: use experiment results to tune the MMM, so the model matches what tests show.
Where AI helps measurement
- Faster data preparation and model runs.
- Plain-language explanations of results for non-analysts.
- Scenario planning: "what if we move 10% from search to video?"
Where AI makes measurement harder
- Automated platforms change many things at once.
- Thousands of creative variants dilute sample sizes.
- Platforms reporting their own success.
A practical program
Run at least one incrementality test per quarter on a big channel; maintain an MMM updated quarterly; reconcile both with finance.