When an AI feature gives a wrong answer or an agent does something odd, you need to reconstruct what happened: what went in, what the model returned, which tools it called, how long it took and what it cost. That is AI observability.
What to capture
- Traces for each request or agent run, with spans for model calls, retrieval and tool calls.
- Model metadata: provider, model and version, parameters.
- Usage: input and output tokens, cost, latency, errors and retries.
- Content (per your privacy and retention rules): prompts, retrieved documents, outputs — or hashes and samples where full content is too sensitive.
- Quality signals: eval scores on sampled traffic, user feedback, guardrail triggers.
Use open standards
The OpenTelemetry project publishes semantic conventions for generative AI covering spans, metrics and events, including agent spans, provider-specific conventions (for example Anthropic, AWS Bedrock, Azure AI Inference and OpenAI) and the Model Context Protocol. The conventions are still evolving, but building on them avoids locking your telemetry into one vendor's format and lets AI traces sit next to your existing service traces.
Dashboards that matter
- Quality: eval score and feedback per use case over time.
- Cost: spend per use case, per team and per unit of value.
- Latency and errors: p50/p95 latency, error and retry rates, fallbacks triggered.
- Safety: blocked prompts, redactions, injection attempts, tool calls denied.
- Agents: steps per run, tools used, runs that hit limits or were stopped.
Privacy by design
Prompts and outputs can contain personal data, secrets or customer code. Decide retention, access and redaction before you turn on full logging — and make the gateway the place where that policy is enforced.