Full Article
The question every CFO asks after the pilot works: what does this cost at real volume? The honest answer is that it depends on a crossover point most vendors will not compute for you.
The two cost shapes
Hosted APIs are pure variable cost - you pay per token, forever, and the bill scales linearly with adoption. Private deployment is mostly fixed cost: GPU capacity, power, and the engineering time to operate it. Below a certain volume the API is cheaper. Above it, it is not.
What comparisons usually omit
- Idle time. A GPU you rent by the hour costs the same whether it serves 10 requests or 10,000. Utilisation is the single biggest lever on private-deployment economics.
- Operational labour. Someone patches, monitors and upgrades the serving stack. That is a real recurring cost and it rarely appears in vendor comparisons.
- Retry and evaluation traffic. Evaluation runs, retries and prompt iteration can be a large fraction of total tokens in the first months.
- Egress and data movement between your VPC and a third-party endpoint.
How to actually decide
Measure your real token volume over 30 days of production-like traffic before choosing. Then compute cost per 1M tokens for each option at your utilisation, not at a vendor's benchmark utilisation. If the workload is spiky and low-volume, hosted usually wins on cost. If it is steady, high-volume, or the data cannot leave your network, private wins - and where data residency is the binding constraint, cost is not the deciding variable at all.
We will publish measured numbers across GPU options once we can do so reproducibly. Until then, treat any specific rupee-per-token figure - including ours - as something you should reproduce on your own workload.
Tags
