How to measure AI engineering ROI — CTO AI Forge

When it comes up

When the CFO asks what the AI tooling budget returns, at contract renewal, or before scaling a pilot.

Your options

Adoption and time-saved surveys

Fits when: You need a quick first signal and have no delivery metrics yet.

Watch out: Self-reported time saved can be wrong in either direction; says nothing about delivery or quality.

Delivery metrics before and after (DORA-style)

Fits when: You can pull lead time, deploys, change-failure rate and review time per team.

Watch out: Other changes (reorgs, seasonality) confound results; needs a baseline.

Controlled pilots with comparison teams

Fits when: You can run similar teams with and without a tool, or stagger rollout.

Watch out: Takes time; comparison teams must be genuinely similar.

Balanced scorecard with cost per unit

Fits when: You want an ongoing view combining adoption, delivery, quality, experience and cost.

Watch out: Must be kept at team level to avoid gaming; needs consistent definitions.

How to decide

Baseline delivery metrics for 2–4 periods before scaling.
Measure layers separately: adoption, engagement, delivery impact, quality, cost.
Use comparisons where you can (staggered rollout, similar teams).
Convert carefully to money: value = capacity freed that went to roadmap work, or delivery improvements tied to business outcomes — not surveyed hours × salary.
Report speed and quality together, with cost per unit (for example AI cost per merged PR).

The productivity scorecard gives you the per-team, per-period structure.

Trade-offs

Surveys are cheap and fast but unreliable; delivery metrics are objective but confounded; controlled pilots are rigorous but slow; a balanced scorecard is the most complete but needs discipline to keep honest.

Real examples

METR's randomized study (July 2025) found 16 experienced open-source developers took 19% longer with early-2025 AI tools, while believing afterwards that AI had sped them up by 20% — a warning against relying on self-reported time savings. Source: METR — Measuring the impact of early-2025 AI on experienced open-source developer productivity (Jul 10, 2025) ↗
DORA's 2024 report estimated that a 25% increase in AI adoption was associated with a 1.5% reduction in delivery throughput and a 7.2% reduction in delivery stability, even as documentation quality, code quality and review speed improved. Source: DORA — 2024 Accelerate State of DevOps report (PDF) ↗
DX's Q4 2025 report found developers self-report saving 3.6 hours a week on average, and daily AI users ship 60% more PRs than non-users; DX cautions that impact varies widely between organizations. Source: DX — AI-assisted engineering: Q4 impact report (Nov 4, 2025) ↗

What to check first

Do you have a pre-AI baseline per team?
Which outcome will you translate into money, and does finance agree?
Are quality and stability reported next to speed?
Is the data used only at team level?

Learn and use

Sources

Last reviewed 2026-10-01.