Finance automation can harm people and institutions when it makes decisions about money at scale without enough human judgment, legal grounding or a way to challenge it. Two public-sector cases are studied worldwide because their lessons apply to any organization that automates money decisions.
Robodebt (Australia)
Australia's Robodebt scheme raised welfare overpayment debts relying heavily on "income averaging" of tax-office income data. It persisted despite heavy criticism until November 2019, when debts stopped being raised solely on averaged income; in 2020 a class action was settled and debts raised in whole or part through averaging were reduced to zero. A Royal Commission later recommended, among other things, more review and oversight of automated decision making.
The Dutch childcare benefits scandal
The Dutch tax authorities used an algorithmic system to risk-score childcare benefit applications. Amnesty International reported that it used Dutch nationality as a risk factor and adapted over time with no meaningful human oversight; tens of thousands of parents were wrongly accused of fraud, and the scandal brought down the Dutch government in January 2021.
Lessons for finance leaders
- Legality first: check that the rule the system applies is lawful and matches policy.
- Human review where it matters: especially for adverse decisions about people.
- Bias testing: check for proxies for protected characteristics in risk scores.
- Challenge and redress: people affected must be able to contest decisions.
- Stop rules: act on early warning signs instead of defending the system.
In a company
The same patterns appear in credit holds, collections, expense-claim rejections, supplier risk scores and payroll corrections. Apply the same five lessons.