Hiring AI rarely fails because someone wanted to discriminate. It fails because it learns from the past, uses proxies, or is set up with a careless rule. Knowing the patterns helps you spot them.
Pattern 1: it learns past bias
A model trained on who was hired before learns to prefer people like them. If past hiring favored one group, the model does too — even if you remove the protected characteristic, because other data (hobbies, schools, words) act as proxies.
Pattern 2: a hard-coded rule
Sometimes a tool is simply configured to exclude people — by age, by gaps in employment, by location — in ways that break the law.
Pattern 3: opaque features
Some tools score video, voice or games with features no one can explain. When the link to job performance is weak, the risk of unfairness is high. Vendors have retired features after scrutiny: HireVue said in January 2021 that it had stopped using facial analysis in its assessments, after internal research found it no longer added significant value.
Pattern 4: no one checks
The tool runs for months; nobody compares selection rates between groups, and the first sign of a problem is a complaint or a lawsuit.
Real cases
- Amazon (reported 2018): an experimental résumé-scoring tool trained on ten years of mostly male applications penalized résumés containing the word "women's"; Amazon dropped the project.
- iTutorGroup (2023): the U.S. EEOC said the company's application software automatically rejected female applicants aged 55 or older and male applicants aged 60 or older; it settled for $365,000.
- Mobley v. Workday (2025): a U.S. federal court preliminarily certified a nationwide age-discrimination collective action against an HR-software vendor over its AI-based applicant recommendations — a sign that vendors, not only employers, face claims.
The lesson
Bias testing, clear configuration, explainable features and regular review are not optional extras. They are the minimum.