AI systems run on data, and most useful data is about people. Privacy law existed long before the AI Act and it is where most AI enforcement still happens. The good news: the same handful of questions satisfies the GDPR, PIPEDA, Quebec's Law 25 and California's rules.
Question 1: What personal data goes in?
Map the flows for each inventory entry: training data, fine-tuning data, prompts and uploads by staff, the outputs, and logs. Staff pasting a customer complaint into a free chatbot is a data transfer to a third party. So is a vendor that retains your prompts to improve its model.
Question 2: On what legal basis, and for what purpose?
- In the EU/UK, you need a GDPR lawful basis for every processing step. The European Data Protection Board's Opinion 28/2024 (December 2024) explains when "legitimate interests" can cover model training and when a model built on unlawfully processed data taints later use.
- In Canada, PIPEDA requires meaningful consent or an exception, and purposes must be ones "a reasonable person would consider appropriate". The Privacy Commissioner's generative-AI principles (December 2023) spell this out: document legal authority, limit collection, be open, and stay accountable for outputs. Bill C-36, tabled in June 2026, would replace PIPEDA's rules with stronger enforcement; watch it.
- In Quebec, Law 25 requires a privacy impact assessment before acquiring or building systems that process personal information, and notice when a decision is based exclusively on automated processing (in force since 22 September 2023).
- In California, AB 2013 requires developers of generative AI released to Californians to publish a summary of training data, including whether it contained personal information (from 1 January 2026).
Question 3: Have you minimised and protected it?
- Turn off vendor training on your data where the setting exists, and write it into the contract.
- Redact or pseudonymise before sending data to a model where you can.
- Set retention limits on prompts and logs.
- Restrict who can upload what: a written "what you may paste into AI tools" rule is the single highest-value control for most small organisations.
Question 4: Can people exercise their rights?
Access, correction and deletion requests do not stop because the data went into a model. Know where you could find a person's data in your AI pipeline, and what you would tell them. Quebec, Colorado (from 2027) and the GDPR also give people a right to know when a decision about them was automated and to ask for review.
Special categories
Biometrics, health, children's data and criminal records carry extra rules everywhere. Face and voice data in particular: the Clearview fines, the Rite Aid ban and Texas's TRAIGA biometric provisions all turn on it. If your tool touches them, escalate to the risk tier above whatever you first assumed.
A cautionary tale with a twist
Italy's regulator fined OpenAI EUR 15 million in December 2024 for, among other things, training without an adequate legal basis, not telling users, and weak age checks. In March 2026 the Court of Rome annulled the fine on procedural grounds without ruling on the substance. The questions the regulator asked (basis, notice, minors, accuracy) are still the ones every deployer in Europe will be asked.