Comparing model latency and cost on DigitalOcean Inference before you commit to one

Question

Every time I put a model behind an endpoint I make the same lazy decision. I pick whatever I used last time, or whatever I read about most recently, and I tell myself I’ll benchmark it properly later. I never do.

So I built the thing that would make me do it: one prompt, fired at six models at once, streaming side by side in columns, with time to first token and cost per run underneath each one. It’s Flask, about 390 lines, no Dockerfile, and it deploys to App Platform on push. The code is on GitHub under MIT: oceanforge/inference-shootout.

What I want to walk through here isn’t really the app. It’s the three things that only showed up once I ran it against the live endpoint, none of which I would have learned from reading the docs. If you’re picking a model for something right now, two of them will probably save you an afternoon.

KFSys · Answer

The integration is two lines, and that’s the boring part DigitalOcean’s inference endpoint speaks OpenAI, so the entireprovider-specific surface area of the app is this: client = OpenAI( base_url="https://inference.do-ai.run/v1/", api_key=os.environ["DIGITAL_OCEAN_MODEL_ACCESS_KEY"], ) Every model below goes through that one client. Llama, DeepSeek, Mistral,Qwen, OpenAI’s open-weight gpt-oss line. Only the model string changes.That is genuinely the pitch, and comparing four providers normally meansfour SDKs, four keys and four invoices, so it is worth something. It isalso the least interesting thing I learned. One thing to get right before you paste that: the credential is a modelaccess key , made under the Gradient AI Platform, not the API token fromSettings. Different object, different page. The catalog lists models your account can’t call The app builds its model picker from GET /v1/models, because hardcodinga model list is how you end up shipping a dead one. That call returned 72 models. My account could call six. Everything from Anthropic, plus GPT-4o and o3, comes back with: 403 - {'error': {'message': 'this model is not available for your subscription tier', 'type': 'forbidden_error'}} Nothing in the /v1/models payload distinguishes them. No availabilityflag, no tier field. You find out by calling it and reading the 403. Which is how I shipped a broken default. My preselected list hadanthropic-claude-haiku-4.5 in it, because it’s in the catalog and it’son the pricing page with a real per-token rate beside it. First live run,that column went red. If you build a picker from that endpoint, assumemost of what comes back is unreachable and handle the 403 per modelrather than per page. Six concurrent streams need threaded workers Worth knowing if you stream anything on App Platform, not just modeloutput. gunicorn’s default sync worker takes one connection per process andholds it until the response finishes. Fine for a 40 ms request. Astreaming response stays open for its whole life, so six concurrentstreams sit on six workers, and with one worker they queue. Time to first token, six streams, default config: mistral-3-14B 1250 ms openai-gpt-oss-120b 5326 ms openai-gpt-oss-20b 7278 ms deepseek-3.2 8347 ms llama-4-maverick 9147 ms Look at the spacing. Each stream’s first token lands about when theprevious one finished. That’s a queue, not slow models. The fix is oneline in the Procfile: web: gunicorn --worker-class gthread --threads 16 --timeout 120 --bind 0.0.0.0:8080 'app:create_app()' Same load, after: four of the six first tokens inside a 1.4 second windowinstead of strung across ten, and wall clock down from 10.7 s to 6.4 s. The reason I’m flagging it: nothing errors. Every request succeeds andreturns correct output. Ship the broken version and you’d conclude themodels are slow, never your own config. A hang would have been easier tocatch. Two smaller App Platform notes. If you don’t commit a Procfile thebuildpack picks a start command for you, and it won’t be tuned forstreaming. And if you use an app factory the target needs theparentheses, 'app:create_app()', not app:app, or you get Failed to find application object, which reads like an import problem. What the numbers actually said Three prompt shapes, six models, three runs each. 54 calls, max_tokens512, nyc region. Medians: model TTFT total cost empty runs mistral-3-14B 533 ms 3.3 s $0.000108 0/9 llama-4-maverick 676 ms 17.9 s $0.000362 0/9 deepseek-3.2 869 ms 5.4 s $0.000416 0/9 openai-gpt-oss-20b 1792 ms 4.6 s $0.000235 2/9 openai-gpt-oss-120b 4797 ms 16.7 s $0.000367 0/9 qwen3.5-397b-a17b 9332 ms 32.0 s $0.000995 7/9 Mistral 14B won on every axis I measured. Fastest to first token, fastestoverall, cheapest per run, and it answered every time. On this workloadthe expensive options bought nothing, which is not what I expected goingin. Time to first token spread 17x across the six, so if a model is goingbehind anything a person waits on, that gap decides whether the featureworks. The failure that doesn’t look like a failure Look at the last column again. All 54 calls succeeded. No exceptions, no non-200s, no timeouts. Nine ofthem returned no readable text at all and billed in full. qwen3.5-397b-a17b did it seven times out of nine, while being the mostexpensive model in the race and the slowest at 32 seconds. It’s areasoning model. It spent 487 of its 512 token budget thinking, ran outbefore writing a word of the answer, and streamed the thinking intodelta.reasoning_content, which is not part of the OpenAI schema and istherefore invisible to any client reading delta.content: completion_tokens: 512 reasoning_tokens: 487 content: 0 characters You can pay full price for silence and have your monitoring call it asuccess. Raise max_tokens and it does answer. The general version isworse than my particular bug though: if your evaluation only watcheslatency and status codes, it cannot see this failure at all. Checkcompletion_tokens_details.reasoning_tokens if you’re putting areasoning model anywhere near a budget. Try it The repo is github.com/oceanforge/inference-shootout . MIT, Flask, noDockerfile, deploys to App Platform from a fork with one encryptedenvironment variable. NOTES.md has the raw build log including the partsthat went wrong, and docs/measurements.json has all 54 runs if you wantto argue with the numbers. The whole experiment cost $0.0185 . That reframed it for me: the reasonnobody measures this before picking a model isn’t cost, and it isn’treally time either. It’s that there’s nothing sitting there ready to run.So now there’s one, and it takes your prompt rather than mine.

alexdo · Answer

Hi there, Nice one, this is exactly the kind of thing that’s annoying to build and genuinely useful once it exists. The three findings in the writeup above, especially the reasoning-model silent-failure one, are the sort of thing you only catch by actually running the traffic, not by reading a spec sheet. Regards

Become a contributor for community

DigitalOcean Documentation

Resources for startups and AI-native businesses

The developer cloud

Start building today

Report this

Comparing model latency and cost on DigitalOcean Inference before you commit to one

Related questions

Tier 2 serverless inference: Daily token bucket caps at 500K

Tier 2 serverless inference: Daily token bucket caps at 500K

Why x-ratelimit-limit-tokens-per-day too little?

What are the best AI frameworks and tools for adding AI features to mobile apps?

Report this

Comparing model latency and cost on DigitalOcean Inference before you commit to one

Related questions

Tier 2 serverless inference: Daily token bucket caps at 500K

Tier 2 serverless inference: Daily token bucket caps at 500K

Why x-ratelimit-limit-tokens-per-day too little?

What are the best AI frameworks and tools for adding AI features to mobile apps?