LLMs generate the most likely next words. Most of the time that produces useful text. Sometimes it produces hallucinations: invented facts, citations, numbers or quotes delivered with the same confidence as true ones. The model does not "know" it is wrong.
Why it happens
- The model predicts plausible text; it has no built-in fact check.
- It fills gaps when it lacks your data (a contract term, a policy, last month's numbers).
- Long, complex prompts and long documents raise the error rate.
Where it bites in finance
- Citations and references in memos, technical accounting positions and tax research.
- Numbers typed out by the model instead of pulled from a system.
- Policy and contract terms the model guesses rather than reads.
- Summaries that drop the one exception that mattered.
Practical checks
- Ground it: give the model the source documents and ask it to quote the passage it relied on.
- Verify every citation and figure against the source before anything leaves finance.
- Keep numbers in systems: totals from the ledger or a spreadsheet, never from the chat.
- Disclose AI use where your policy or contract requires it, and keep the prompt and output for anything material.
- Name a reviewer: the person who signs off owns the content, whatever drafted it.
The professional standard has not changed
Using AI does not lower the bar for accuracy. Clients, auditors, regulators and boards hold the human and the firm responsible for what is published.