Marketing AI comes with a flood of numbers: "10x content", "50% lower cost", "3x ROAS". Some are real; many are not comparable to your situation. A CMO needs a quick way to sort them.
Four kinds of evidence, weakest to strongest
- Vendor demos and case studies. Selected successes, often with unclear baselines. Useful for ideas, not for business cases.
- Surveys. Tell you what peers say they do or plan. Good for benchmarks of spend and adoption; weak on results, which are self-reported.
- Company disclosures. Numbers a brand reports about itself (for example in earnings calls or press releases). Better, but still chosen by the company.
- Experiments. A test with a control group: some customers or regions get the AI-made campaign, others don't. This is the only way to know what AI caused.
Questions to ask of any number
- Compared with what? A baseline, a control group, or last year?
- Who measured it? The vendor, the customer, or an independent party?
- Over how long? Novelty effects fade.
- Does it fit us? A global consumer brand's result may not transfer to a B2B niche.
Regulators are watching AI claims too
In September 2024 the US Federal Trade Commission announced "Operation AI Comply", a set of actions against companies that used AI hype or AI tools to deceive people. FTC Chair Lina Khan said: "There is no AI exemption from the laws on the books." This matters twice for marketers: when you buy AI (check the vendor's claims), and when you market AI features in your own products (your claims need proof like any other).
A practical rule
Treat vendor numbers as a hypothesis. Before scaling, run your own pilot with a baseline and, where possible, a holdout group. Write down the metric and target before you start, so the result can't be reinterpreted afterwards.