What Inference Provider Lock-In Actually Looks Like in Production

“OpenAI-compatible” mostly describes the shape of a request and response, plus a rough sketch of shared semantics like auth headers and streaming. Different providers implement different parts of that surface faithfully, and none of it says much about what happens once the request reaches the model, or what your code has to do with the response once it arrives. That gap is where lock-in accumulates, and it splits across six surfaces.

Prompt calibration

A prompt is tuned, whether deliberately or through trial and error, to a specific model’s tendencies: how literally it follows formatting instructions, how it handles ambiguous requests, where its refusal boundaries sit. Point the identical prompt at a different model and the output distribution shifts. It can shift even on a nominally “same” open-weights model if the new provider serves it with a different quantization, a different serving engine, or different inference optimization such as speculative decoding. All of these can change the output on their own, independent of settings the client controls, like temperature or top_p. Migrating providers means re-running your evaluation suite against every production prompt and re-tuning the ones that regress. For a team running 40 production prompts, that is not an afternoon’s work. As a rough rule of thumb rather than a benchmarked figure, building or re-running automated evals for each prompt, reviewing outputs, and adjusting wording tends to take days to weeks. The real time depends heavily on how rigorous the evals already are and how many prompts fail on the first pass.

Output format assumptions

Every provider claims structured output support, but the guarantees and failure modes differ in ways that matter to a parser. A side-by-side comparison of drop-in OpenAI-compatible APIs makes the same point from the testing side: basic chat completions worked across every provider tested, and tool calling and streaming edge cases were where the compatibility broke down. Compatibility is a spectrum, not a single yes-or-no property.

OpenAI’s response_format with json_schema and strict: true is documented to always produce schema-conformant JSON, though the schema itself is capped (up to 5,000 object properties and a 120,000-character combined limit, raised from a much smaller cap in 2025). The older json_object mode only guarantees valid JSON, not schema conformance, and OpenAI now recommends against it.

Anthropic’s Claude API has a comparable structured output feature (output_config.format) built on constrained decoding rather than prompting the model to “please return JSON.” But the guarantee has documented exceptions: truncation on stop_reason: "max_tokens" can still produce invalid JSON, enum casing isn’t guaranteed, and the feature is incompatible with citations and message prefilling, which a migrating team may already depend on.

Together.ai and Fireworks AI both support schema-constrained JSON mode, but both vendors’ own docs say passing the schema through response_format alone isn’t enough. You also have to restate the schema in the prompt text. Fireworks and Together are explicit that “the model doesn’t automatically ‘see’ the schema.” That’s a difference in how the two vendors document their implementations rather than a proven difference in reliability. Still, a parser built assuming OpenAI-style enforcement may need extra validation here that would be redundant on OpenAI.

Tool calling diverges more structurally. OpenAI, Together, and Fireworks return tool call arguments as a JSON-encoded string the client must parse. Anthropic returns an already-parsed JSON object instead, and enforces a strict message-ordering rule the others don’t: the tool result must immediately follow the tool call, with no text before it. Code written against OpenAI’s more permissive ordering needs real changes, not a field rename, to run against Claude.

finish_reason (OpenAI’s name) and stop_reason (Anthropic’s) diverge too. OpenAI, Together, Fireworks, and DigitalOcean’s OpenAI-compatible endpoint all collapse “hit a natural stop” and “hit a stop sequence” into one value, stop. Anthropic splits them into end_turn and stop_sequence, which carry different information, not just different names. And content_filter appears in OpenAI’s and DigitalOcean’s enum but is documented as absent from Together’s and Fireworks’s, so branch logic written for one will silently never fire on the other.

Error handling and retry logic

Retry logic tuned against one provider’s failure modes can misbehave against another’s, because the error taxonomies aren’t aligned, and not every error under the same status code should be retried the same way. OpenAI documents two textually distinct 429 responses under the same status code. One signals request-rate pacing, worth retrying with backoff. The other signals an exceeded billing quota, which won’t resolve itself no matter how many times you retry the request. Anthropic uses HTTP 529 for “overloaded across all users,” a status code distinct from 429. Its own docs note that, in rare cases, capacity pressure can still surface as 429 through a separate acceleration-limit mechanism, so Anthropic effectively runs a two-tier throttling model. Together.ai documents a clean split between 429 (caller exceeded their own limit) and 503 (a platform-side capacity fault, not the caller’s problem). Fireworks documents an even more granular set of codes, including 502 for “invalid response from an upstream server.” Fireworks also notes that staying within your rate limit doesn’t guarantee a request succeeds, since 503 “Service Overloaded” can still occur. A retry policy that treats every 429 as retryable, or every 5xx as identical, will handle at least one of these cases wrong.

Rate limit response headers aren’t standardized either. OpenAI uses x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens. Anthropic uses anthropic-ratelimit-requests-remaining and includes a retry-after header. DigitalOcean’s headers drop the x- prefix entirely (ratelimit-remaining, retry-after). That retry-after header only shows up in burst-limit error responses, not on every response. Many teams never touch these headers directly, since an SDK or API gateway already normalizes them, and in that case this divergence costs nothing. It only becomes real migration work for whoever wrote or maintains that normalization layer, or for teams parsing headers directly without one.

Caching architecture

Prompt caching economics differ enough from one provider to the next that a prompt structure optimized for one provider’s cache is often noticeably less effective, though not actively harmful, on another’s.

Anthropic’s caching is explicit: up to four cache_control breakpoints per request, a default five-minute TTL refreshed on every hit (or a paid one-hour tier), a write premium of 25% (or 100% for the one-hour tier), and a read discount of 90%. The minimum cacheable prefix runs from 512 tokens on the newest models up to 4,096 on some others, and doesn’t track release order cleanly, so it’s worth checking your specific model rather than assuming.

OpenAI’s caching is automatic and, for model generations before GPT-5.6, free to write to, with a 1,024-token minimum prefix. Retention comes in two shapes: an older in-memory cache lasting 5 to 10 minutes, and a 24-hour tier that’s now the only option on newer models. The discount itself ranges from around 50% on older model families to around 90% on the newest, so the same caching behavior produces different economics depending on which OpenAI model you’re calling. Microsoft’s Azure OpenAI documentation also states that GPT-5.6 and later now bill for cache writes, cutting against OpenAI’s “no write cost” reputation, though that’s confirmed for Azure specifically and not independently checked against OpenAI’s own docs.

Google’s Gemini caching is a third shape entirely: an automatic “implicit” cache with no cost guarantee, plus a manual “explicit” cache that guarantees around 90% off input price, and, unlike any other provider here, a separate storage fee billed per million tokens per hour the cache stays open.

Together.ai and Fireworks both cache automatically with no toggle, but neither publishes a fixed TTL or minimum prefix the way the three above do. Together’s own docs call its serverless cache “best-effort and short-lived,” a real transparency gap by comparison.

None of this tells you whether caching is worth it for your workload, which comes down to a break-even question: does your prefix stay stable, and do requests arrive close enough together, to earn back what you paid on the write? Our article on prompt caching works through that math, including a firsthand test showing how a timestamp at the top of a prompt can collapse a cache hit rate from 98% to under 1%. A platform that passes requests straight through to Anthropic’s or OpenAI’s hosted models inherits that model’s caching economics rather than inventing its own. A routing gateway like OpenRouter, or a self-hosted LiteLLM proxy, adds its own retry, fallback, and caching logic on top, which is a dependency a migration also has to account for.

Cost optimization calibration

Two models that look identically priced per token can cost meaningfully different amounts for the same workload. Tokenizer differences and invisible reasoning tokens are the two most citable, best-documented reasons, but they’re not the only ones. Output length, how much reasoning effort a request uses, the size of the system prompt, and tool-call overhead that doesn’t show up in a naive per-message token estimate all move the real bill independent of the sticker per-token rate.

On tokenizers specifically, the difference isn’t trivial. OpenAI uses different named encodings across its own model lineup (cl100k_base, o200k_base, and older ones), so identical text produces different token counts even within one vendor’s model family. Anthropic’s own documentation states that its newer tokenizer (used from Claude Opus 4.7 onward) produces approximately 30% more tokens for the same input text than its earlier tokenizer. That matters if you’re estimating migration cost by token count rather than by content volume. Anthropic’s own engineering guidance warns against using OpenAI’s tiktoken library to estimate Claude token counts, noting it undercounts by 15 to 20% on ordinary text and more on code.

Reasoning models also bill for tokens the caller never sees. OpenAI’s documentation states plainly that reasoning tokens, while invisible in the API response, “occupy space in the model’s context window and are billed as output tokens.” It also warns that a response can be truncated before producing any visible output at all, so a caller can be charged for input and reasoning tokens with nothing to show for it. Together.ai’s documentation makes the same point about reasoning-model billing on its own platform, though the exact wording and which model families it applies to differ, so check the specific model page before assuming parity. This isn’t a hidden fee in the sense of being undisclosed, but it’s easy to miss when budgeting a migration. A workload that looked affordable on token count alone can end up spending a large and unpredictable share of its budget on invisible reasoning tokens.

Output tokens deserve their own callout here, since they’re easy to underweight when you’re comparing providers on input price alone. A closer look at output token pricing for Llama 3.3 70B found output tokens carrying a 2.2x to 5.4x premium over input tokens on most models, and unlike input tokens, they can’t be cached away, since each one is generated serially. At the level of an individual request rather than a session average, that premium can flip which model comes out cheaper, which is exactly what a per-token sticker-price comparison misses.

Operational tooling

The least glamorous surface is also one of the stickiest in practice: dashboards, observability integrations, usage attribution, and the finance and on-call workflows built around a specific provider’s console and billing export. A fair share of this tooling lock-in lives outside the provider’s own dashboard entirely, in a third-party observability layer such as LangSmith, Helicone, Arize, or Weights & Biases that the team has wired up to one provider’s usage format. None of this shows up in application code, but replacing it is real work that shows up on someone’s roadmap, not in a pull request.

These six are the surfaces this catalog focuses on, but they’re not the only ones a real migration can hit. Fine-tuned models and adapters (LoRA weights trained against one provider’s base model) are frequently not portable at all. Switching embedding models means re-embedding your corpus and rebuilding vector indexes, a cost that has nothing to do with chat completions. Moderation and refusal behavior can shift noticeably between providers and between versions of the same model. Streaming event schemas, covered briefly above for text, get much more divergent once tool-call streaming and reasoning streaming are involved. Each of these deserves its own treatment; they’re flagged here so the six-surface catalog isn’t mistaken for a complete list.

Contract terms are the layer people usually mean by “avoiding lock-in.” The table below covers four dimensions: spend or term commitment, model-deprecation notice, price-change notice, and data egress. Figures come from each provider’s own pricing pages and terms of service, accessed July 2026; treat anything not independently re-verified as a starting point rather than a final answer.

A few things stand out. Only DigitalOcean, Together.ai, and Baseten publish a specific, numeric deprecation notice period at all; Fireworks has none documented, likely a documentation gap rather than an absence of internal practice. None of the seven non-hyperscaler providers publish a numeric price-change notice period, Nebius being a partial exception for paid agreements. OpenAI and Anthropic, by contrast, publish more detailed commitments here. That’s a reminder that “no contract required” and “well-documented” are different qualities. On this narrow dimension, the providers most associated with vendor concentration risk are more transparent than some of the alternatives marketed as lock-in-free.

None of this touches Section 1, though. A provider with a generous deprecation notice and no minimum spend can still be a hard migration if your prompts, parsers, and retry logic are calibrated to its behavior. Contractual freedom and practical portability are different things, and treating them as the same is how teams get surprised mid-migration.

There’s a less obvious cost hiding underneath the contractual one. You can only credibly threaten to switch providers, and keep pricing honest through competition, if you can accurately price your own workload on each one. Opaque cost structures make that comparison shopping harder. That raises the practical cost of switching even when nothing in the contract is stopping you, because a provider whose true cost is hard to calculate is also a provider whose price increases are hard to notice.

The clearest example is reasoning-token billing, already described above: tokens the caller never sees, billed as if they were ordinary output, on a model whose reasoning length isn’t fully predictable in advance. A second is tokenizer variation, also described above. The same content can cost meaningfully more on one model than another even at an identical advertised per-token rate, something Anthropic’s own documentation quantifies directly for its own tokenizer transition. A third is the patchwork of infrastructure and per-request fees layered on top of token or compute pricing. Anthropic, for instance, discloses several add-on charges beyond its base token price: a per-search fee for its web search tool, container-time billing for code execution that can apply “even if the tool is not called” when files are preloaded, and a documented multiplier for certain data-residency and priority-processing options. None of this is secret, since it’s published, but it’s easy to miss when estimating a migration’s cost purely from the headline per-token rate.

Compute-billed platforms add a different kind of complexity, and the two commonly grouped together here aren’t quite the same shape. Modal bills purely by compute time, per second of GPU use, with no token-priced option at all. Baseten offers both: a per-token “Model APIs” tier for popular models that behaves like a normal token-priced provider, and separate per-minute dedicated deployments for custom or high-volume workloads. Comparing either provider’s compute-billed tier to a token-priced provider requires estimating your own throughput, not just reading a price list, and that estimation step is itself a place where cost comparisons quietly go wrong.

A useful discipline is to run a short predictability audit against any provider’s pricing page before committing to it. Ask whether output token pricing is listed separately and visibly from input pricing, since some providers bury it. Ask whether reasoning tokens, if the model produces them, are explicitly disclosed as billed and reported in the usage object, not just alluded to. Ask whether the provider gives you a calculator or per-request cost breakdown you can reproduce independently, rather than requiring you to trust a monthly invoice. And ask whether the API response itself exposes a usage object with enough detail, prompt tokens, completion tokens, cached tokens, reasoning tokens, to let you reconcile your own estimate against what you were charged. A provider that fails several of these is one where cost creep is hard to detect until the bill arrives.

Once the surfaces are named, the practical question is how much abstraction effort is worth spending to contain them, and the honest answer is less than most engineering instincts suggest.

The gateway pattern. A thin internal layer sitting between your application and each provider’s API is worth building, but only for a narrow set of jobs: authentication, transport, normalized error handling and retries, and consistent usage logging across providers. Prompt logic doesn’t belong in that layer. Writing a single abstracted prompt template that works identically across providers is a trap. It tends to cost more engineering effort and produce worse output on every provider than maintaining provider-specific prompt variants and migrating them deliberately when needed.

Eval suites as the real portability layer. The single highest-leverage investment against lock-in is a test suite, not an abstraction layer. A team with an automated, provider-agnostic evaluation suite covering its real production tasks can often validate a new provider’s behavior in about a day, and make an informed switch within a week or two. The real number depends heavily on eval coverage, compliance review, and how deep the deployment goes. A team without any such suite has to rediscover, one production incident at a time, every place its prompts were quietly tuned to one provider’s quirks. This is a testing investment, not an architecture investment, and it pays off whether or not you ever switch providers, because it also catches regressions when your current provider silently updates a model version underneath you.

Open-weights model choice as structural insurance. Running an open-weights model, served through whichever provider offers the best price and performance at the moment, is one of the deepest forms of portability available. That’s because most of the behavioral calibration described in Section 1 travels with the model, instead of staying trapped inside one vendor’s hosted deployment. This doesn’t eliminate lock-in entirely. Quantization choices, tensor-parallelism and other serving-engine differences, and tokenizer revisions between versions can all still shift output for a nominally identical open model. Some hosted providers also inject their own default system prompt or safety instructions ahead of yours, though this varies by provider, so check rather than assume for whichever open-weights host you pick. Open-weights cuts calibration risk; it doesn’t remove it. Even with those caveats, it survives a provider migration far better than a fully proprietary model does, where switching means adopting an entirely different model’s behavior along with the new endpoint. The honest tradeoff is capability. The strongest proprietary models still lead on many hard tasks, and choosing an open-weights model for portability’s sake can mean accepting a real quality gap.

What’s worth thinking twice about. Multi-provider routing on day one, adopting a gateway abstraction layer like LiteLLM before you have more than one provider to abstract across, and premature multi-cloud deployment all share the same problem. They pay a real, ongoing complexity cost today against a hypothetical future benefit that may never materialize, and teams often underestimate that cost because it’s easy to justify in the abstract. A closer look at multi-model routing as an infrastructure decision puts real numbers on that complexity cost: added latency on every request, a new failure surface, and, worst of all, silent misrouting, where a wrong route returns a plausible but wrong and sometimes far more expensive answer with no error raised at all. That said, this is a tradeoff, not a rule. Some organizations get real day-one value from a unified gateway, such as centralized observability and auth, or need multi-provider redundancy from the start for availability or compliance reasons that are already concrete rather than hypothetical. The judgment call is whether the second provider (or the compliance requirement, or the uptime target) is real today or merely plausible someday. Only the former justifies paying the complexity cost now.

None of the above means portability is always worth pursuing. The decision comes down to three variables: how likely you are to migrate, how much that migration would cost given the calibrations described in Section 1, and how much value you’d give up by staying portable instead of using a provider’s specific strengths, be that a frontier proprietary model, favorable caching economics, or integrated tooling your team already relies on.

Accepting lock-in tends to be the right call for an early-stage team where shipping speed dominates every other consideration and portability is realistically a problem for a later, better-funded version of the company. It’s also often right for a workload that depends on a frontier proprietary model with no comparable open-weights alternative, where the capability gap would be the real cost of switching, not just the migration effort. And it’s right when a provider’s specific cost optimizations, aggressive caching discounts or batching support tuned to your traffic pattern, deliver savings that clearly exceed any plausible future migration benefit.

Minimizing lock-in matters more for open-weights workloads, where portability is close to free and there’s little reason not to take it, for cost-sensitive operations at a scale where provider competition is your primary lever for keeping prices down, and for compliance environments where the ability to exit a vendor on short notice is a requirement rather than a nice-to-have.

As a condensed reference: minimize lock-in when you’re running open-weights models, when your spend is large enough that provider competition materially affects your pricing, when your evals are already automated, or when exit capability is a compliance requirement. Accept lock-in when a provider’s specific features deliver measurable value above the realistic migration cost, when you’re pre-product-market-fit and speed matters more than optionality, or when the model you need only exists in one place.

Regardless of which side of that line you land on, a few practices are worth doing unconditionally: keep your evaluation suite portable and automated, log per-request usage and cost data so you can reconstruct your actual spend independent of any single provider’s dashboard, structure prompts so provider-specific elements are isolated rather than scattered through your codebase, and know each of your providers’ model deprecation notice policies before you need that information under time pressure.

1. Does an “OpenAI-compatible” endpoint mean switching providers is just a config change?

No. It standardizes the transport layer, the shape of the request and response, but not what happens above it. Prompt calibration, output parsing, retry logic, caching design, and cost estimates are all tuned to one provider’s specific behavior, and re-tuning them after a switch is real engineering work, not a base URL update.

2. What are the main surfaces where lock-in shows up?

Six: prompt calibration, output format assumptions, error handling and retry logic, caching architecture, cost optimization calibration, and operational tooling. Each one quietly accumulates provider-specific behavior that a straight endpoint swap never touches.

3. Do structured outputs and tool calling work the same way across providers?

They don’t. OpenAI’s strict JSON mode guarantees schema-conformant output up to a size cap. Anthropic’s structured output feature has documented exceptions around truncation and message prefilling. Together and Fireworks require the schema to be restated directly in the prompt text, not just passed as a parameter. Tool-call argument formats and message-ordering rules diverge too.

4. Does prompt caching save money the same way on every provider?

Not the same way. Anthropic charges a write premium with a short default TTL, OpenAI’s caching is largely automatic with a longer retention window, and Google adds a separate storage fee that none of the other providers charge. A prompt structure optimized for one provider’s cache is often noticeably less effective on another’s.

5. Are no-contract providers automatically less risky than proprietary ones like OpenAI or Anthropic?

Not on every dimension. OpenAI and Anthropic publish more detailed model-deprecation and price-change notice policies than several providers marketed as lock-in-free. Contractual freedom and practical portability are different things. A generous deprecation notice doesn’t help if your prompts and retry logic are calibrated to that provider’s behavior.

6. What’s the single best investment for reducing lock-in risk?

An automated, provider-agnostic eval suite covering your real production tasks, not an abstraction layer. It lets a team validate a new provider’s behavior quickly and pays off even if you never switch, since it also catches regressions when your current provider silently updates a model underneath you.

This article tracked lock-in across six surfaces: prompt calibration, output format assumptions, error handling and retries, caching architecture, cost optimization, and operational tooling. Most of that calibration lives above the transport layer that “OpenAI-compatible” claims to standardize. The contractual layer (deprecation notices, price-change policies, egress terms) matters less than the behavioral calibration underneath it, though it’s worth compiling since some vendors document it far better than others. Cost predictability closes the loop. If you can’t price your own workload accurately, you can’t credibly threaten to switch, and pricing stays soft. The practical answer is to keep an automated eval suite, log real usage and cost data, isolate provider-specific code instead of scattering it, and know your providers’ deprecation policies before you need them. Then decide whether accepting lock-in or avoiding it is the better trade for where your team is right now.

Swapping a base URL was never the whole migration. Naming the surfaces turns the rest of it into a scoped project you can plan for. Do that work while nothing is forcing you to, and a provider change becomes a decision you make on your own schedule, not one a deprecation email makes for you.

• OpenAI API documentation
• Anthropic Claude API documentation
• Google Gemini API and Vertex AI documentation
• Together.ai documentation
• Fireworks AI documentation
• DigitalOcean Inference documentation
• Baseten documentation
• Modal documentation
• Nebius AI Cloud documentation