How to choose models and AI vendors — CTO AI Forge

When it comes up

When a new model is released, when a vendor contract renews, when costs spike, or when a provider changes terms or capacity.

Your options

Single strategic provider

Fits when: You value simplicity, enterprise terms and support, and your use cases are well served by one provider's models.

Watch out: Concentration risk: pricing, capacity or policy changes hit everything at once.

Multi-model via a gateway

Fits when: You run several use cases with different cost and quality needs and want to switch models through configuration.

Watch out: Needs a platform team, evals per use case and more vendor management.

Open-weights first, APIs where needed

Fits when: Data control or cost at high volume dominates, and you have the skills to host and update models.

Watch out: Operations burden; may lag frontier quality on hard tasks.

How to decide

Evaluate on your tasks, not leaderboards. Build evals per use case: quality, latency, cost per request and safety.
Check the non-functional terms: data use and retention, residency, security certifications, rate limits and capacity commitments, deprecation notice periods.
Keep switching cheap: route through a gateway, version prompts, test at least one alternative model for every critical use case.
Match model to task: smaller, cheaper models for simple tasks; frontier models where quality pays.
Re-evaluate on a schedule and when a provider changes models or terms.

Trade-offs

One provider is simpler and may win better terms but concentrates risk; multi-model buys resilience and cost optimization at the price of platform work; open-weights maximizes control but adds operations and may trail on quality.

Real examples

TechCrunch reported on June 3, 2025 that Anthropic cut off nearly all of Windsurf's direct capacity to Claude 3.x models with less than five days' notice, amid reports that OpenAI was acquiring Windsurf; Windsurf users faced short-term availability issues and some brought their own API keys. Source: TechCrunch — Windsurf says Anthropic is limiting its direct access to Claude AI models (Jun 3, 2025) ↗
GitHub described in January 2025 how it evaluates models before offering them in Copilot: more than 4,000 offline tests across about 100 containerized repositories, measuring tests passed, answer accuracy, token usage and safety, then live internal canary-style evaluations — and weighing trade-offs such as acceptance rate against latency. Source: GitHub Blog — How we evaluate AI models and LLMs for GitHub Copilot (Jan 17, 2025) ↗
Uber's GenAI Gateway (July 2024) gives teams one interface to OpenAI, Vertex AI and Uber-hosted models, mirroring OpenAI's API so applications can use different models without rewriting integrations. Source: Uber Engineering — Navigating the LLM landscape: Uber's innovation with GenAI Gateway (Jul 11, 2024) ↗

What to check first

Do you have evals for each critical use case?
Can you switch the model behind a use case without code changes?
What notice period and capacity commitments are in your contracts?
What is the cost per request at expected volume for each candidate?

Learn and use

Sources

Last reviewed 2026-10-01.