Once more than a couple of teams build with LLMs, direct integrations with each vendor become a liability: duplicated code, scattered API keys, no cost visibility, inconsistent data handling, and painful model switches. A model gateway (also called an LLM or AI gateway) is a shared service that sits between your applications and every model provider.
What a gateway does
- One interface to many providers and self-hosted models, often mirroring a popular API format so existing libraries work.
- Authentication and authorization: which team or service may call which model, with what limits.
- Data controls: redact personal or secret data before it leaves, enforce region and vendor rules.
- Cost attribution: tokens and spend per team, service and use case; budgets and rate limits.
- Logging and audit: requests, responses (per your retention policy), latency and errors.
- Reliability: retries, fallbacks to another model, timeouts, caching.
- Model management: switch or upgrade the model behind a use case through configuration.
A real example: Uber
Uber's engineering blog (July 2024) describes its GenAI Gateway, built because more than 60 LLM use cases across the company were integrating in different ways. The gateway gives uniform access to models from OpenAI and Google's Vertex AI as well as Uber-hosted models, mirrors OpenAI's HTTP/JSON interface, includes a PII redactor that anonymizes sensitive information before requests reach third-party vendors, provides cost attribution and audit logs, and requires a security review of each use case against Uber's data handling standard before access is granted. At the time it served close to 30 customer teams and 16 million queries a month.
Build or adopt?
Open-source and commercial gateways exist, and cloud providers offer managed options. The architecture matters more than the product: make the gateway the default path for every AI call, including coding tools where possible.
Where it fits
apps & agents → model gateway → { vendor APIs | self-hosted models }
↓
logs · costs · evals · policies