AI systems fail quietly. A model drifts as the world changes, a vendor ships an update, a prompt-injection attack leaks data, a chatbot starts promising refunds. Monitoring is how you notice; incident response is what you do next. Both are now legal duties, not just good practice.
What the laws say
- EU AI Act, Article 26: deployers of high-risk systems must monitor operation according to the instructions for use and inform the provider (and the market surveillance authority) when they identify a risk or a serious incident. Article 73 requires providers to report serious incidents to authorities; the default deadline is 15 days after awareness, shorter (two to ten days) for deaths, widespread infringements or critical-infrastructure disruption.
- New York's RAISE Act (effective 1 January 2027 after its 2026 chapter amendment) requires large frontier-model developers to report safety incidents within 72 hours. Even if you are not covered, 72 hours is becoming the reference clock.
- Privacy law already requires breach notification (GDPR: 72 hours to the regulator; PIPEDA: as soon as feasible; Quebec: promptly). An AI incident involving personal data is a privacy breach too.
- Colorado SB 26-189 and Quebec's Law 25 give people a right to human review of automated decisions; every such request is an incident signal worth logging.
Define "incident" before you need to
Write a list. For example:
- The system produced an outcome that harmed or could harm a person (wrong denial, wrong accusation, unsafe advice).
- A user or reviewer reports discrimination or a pattern of errors.
- The model changed behaviour after a vendor update.
- Personal or confidential data was exposed via the system.
- The system was manipulated (prompt injection, jailbreak, data poisoning).
- The system was used outside its approved purpose.
Monitoring that a small team can run
- Performance: accuracy or agreement rate against a human sample, monthly.
- Fairness: impact ratios by group, quarterly (see the fairness lesson).
- Volume and drift: sudden changes in the share of approvals, escalations or refusals.
- Complaints and review requests: count them, read them.
- Vendor changes: subscribe to release notes; test after updates.
- Transcripts: for chatbots, a weekly sample read by a person. Rite Aid's failure was not one bad match; it was years of matches nobody measured.
The response plan (one page)
- Report: any staff member can report to a named inbox or person; no blame for reporting.
- Triage within one business day: severity, people affected, data involved, legal clocks started (72 hours for GDPR breaches, 15 days for EU AI Act serious incidents, contractual notice to vendors).
- Contain: pause the system or the feature; the off switch you documented in the oversight lesson.
- Notify: affected people, regulators where required, the vendor, insurers.
- Fix and verify: root cause, correction, re-test.
- Record: keep the incident in the system's record file, with dates and decisions.
- Learn: update the risk assessment, the training, and if needed the policy.
Run a tabletop exercise once a year: pick a scenario from your incident list and walk the plan in an hour.