A good scorecard is small, balanced and stable over time. Research frameworks agree on one thing: no single metric captures developer productivity.
Frameworks to borrow from
- SPACE (Forsgren and colleagues, ACM Queue, 2021) argues that developer productivity "cannot be measured by a single metric or dimension" and spans satisfaction, performance, activity, communication and efficiency.
- DORA's four delivery metrics cover throughput and stability.
- DX Core 4 (Noda, Tacho, Storey, Greiler) groups measures into four dimensions — speed, effectiveness, quality and business impact — and warns that improving one can hurt another.
A practical AI scorecard, per team per period
| Area | Metric |
|---|---|
| Adoption | % of engineers actively using AI tools |
| AI share | % of merged code AI-authored (where your tools can estimate it) |
| Throughput | PRs merged per engineer; deploys |
| Speed | Cycle time (commit to production) |
| Review | Median review time; PR size |
| Quality | Change-failure rate; rework |
| Experience | A short quarterly developer survey |
| Cost | AI spend per engineer or per merged PR |
Rules that keep it honest
- Baseline first. Capture 2–4 periods before a rollout.
- Teams compare with themselves. Different codebases and products make cross-team rankings misleading.
- Never use it for individual performance. It will be gamed, and it will damage trust.
- Balance speed with quality. Report them side by side, always.
- Pair numbers with conversation. Use the scorecard to ask better questions, not to issue verdicts.
Where the data comes from
Version control and CI/CD systems give PRs, review times, deploys and lead time; incident tools give failures; AI tools' admin consoles give usage; surveys give experience. Start with what you can pull today and improve definitions over time.