The real cost of running AI features: APIs vs self-hosted models

AI features have a cost structure most teams haven't dealt with before: you pay every time a user clicks the button. A feature that costs a fraction of a cent per request in testing can turn into a serious monthly bill once thousands of people use it every day. And one of the most common reasons AI projects get cancelled is that costs grew faster than value.

The fix is not to avoid AI features. It's to estimate costs before you build, design to keep them low, and know when a different hosting model makes sense. This guide walks through all three.

Note: prices change often and differ widely between providers and models. All prices below are illustrative round numbers for the maths. Plug in your provider's current rates.

Where the cost of a request comes from

Every lever in this diagram changes your bill.

With a hosted API, you pay per token, the chunks of text models read and write. As a rough rule for English text, one token is about four characters, or about three-quarters of a word. Code and non-English text often use more tokens per word.

The basic formula:

cost per request = (input tokens × input price) + (output tokens × output price)
monthly cost     = cost per request × requests per month

Two details matter a lot:

• Output tokens usually cost several times more than input tokens. Long answers are expensive.
• Input includes everything you send: system prompt, retrieved documents, conversation history, tool definitions. Not just the user's message.

Worked example: a support assistant

Assume these example prices:

And a typical request:

Large model, per request: 6,100 × $3 / 1,000,000 = $0.0183 input, plus 400 × $15 / 1,000,000 = $0.0060 output, for about $0.024.

At 10,000 requests a day (about 300,000 a month): about $7,300 a month.

Small model, same request: about $0.0039, or about $1,200 a month.

Same feature, roughly a sixfold difference, purely from model choice. And notice where the tokens are: the user's message is under 2% of the input. Context size, not user input, drives the bill.

The hidden costs teams miss

• Agents multiply calls. An agent that takes 8 steps makes 8+ model calls, each re-sending a growing transcript. Agent tasks can cost tens of times more than a single chat answer.
• Retries and failures. Timeouts, invalid JSON and retries all cost tokens.
• Evals and testing. Running your test suite against a large model on every change adds up.
• Embeddings and reranking. Usually cheap, but not free at scale.
• Long conversations. History grows every turn; without trimming, turn 30 costs far more than turn 1.
• Engineering time. Prompt tuning, monitoring and incident handling are real costs, especially for self-hosting.

Six levers that cut costs (in order of effort)

1. Trim the context

The cheapest token is the one you don't send. Retrieve fewer, better documents; summarise old history; strip unused tool definitions; shorten the system prompt. Cutting input from 6,100 to 3,000 tokens nearly halves input cost.

2. Limit output length

Ask for concise answers, set a maximum output token limit, and avoid making the model restate the question. Because output tokens are pricier, this has an outsized effect.

3. Use prompt caching

Most providers discount repeated input prefixes. Put stable content (system prompt, tool definitions, reference docs) first and identical across requests, and dynamic content last. For apps with long, stable prompts, this can cut input costs substantially and reduce latency.

4. Cache whole responses

Many requests are near-duplicates: the same FAQ question asked a hundred ways. Cache answers for identical (or, with care, semantically similar) questions. A cache hit costs almost nothing.

5. Route between models

Send easy requests to a small, cheap model and hard ones to a large model. A simple classifier, or even rules like message length and topic, can route most traffic to the cheap model while keeping quality where it matters.

6. Batch non-urgent work

Many providers offer discounted asynchronous batch processing for jobs that don't need an instant answer: nightly summaries, document tagging, bulk extraction.

Self-hosting: when does it make sense?

Running an open-weight model on your own GPUs swaps per-token pricing for fixed infrastructure cost. That changes the maths completely:

A simple break-even calculation

monthly API cost       = requests × cost per request
monthly self-host cost = GPU servers × monthly price + engineering/ops time
 
break-even requests    = self-host cost ÷ API cost per request

Example: suppose a GPU server that can handle your load costs $1,500/month (illustrative), plus about $1,500/month of engineering time to run it: $3,000 total. Against the small-model API cost of $0.0039 per request, break-even is about 780,000 requests a month. Against the large model at $0.024, it's about 125,000 requests, if an open model you can host matches the large model's quality for your task.

That "if" is the crux. Self-hosting only wins when all of these are true:

1. Volume is high and steady, so the GPUs stay busy.
2. An open model is good enough for your task. Test it with your evals, not benchmarks.
3. You have the skills to run inference servers, scaling and monitoring.
4. Or data residency, privacy or latency requirements rule out third-party APIs.

For most early-stage products, hosted APIs are cheaper and far simpler. A common path is: start with APIs, optimise with the six levers above, and revisit self-hosting when volume is high, steady and well understood.

The middle option: hosted open models

Several providers host open-weight models and charge per token, often at lower prices than frontier models. This gives you open-model pricing without running GPUs, and makes a later move to self-hosting easier because the model is the same.

Build a cost model before you build the feature

Answer these in a spreadsheet before writing code:

1. How many requests per user per day? How many active users?
2. Average input tokens and output tokens per request, measured from a prototype, not guessed?
3. Which model, at which price, for which share of requests?
4. Expected cache hit rate?
5. Agent steps per task, if any?
6. Cost per user per month, compared to what that user pays you?

That last line is the one that matters. If an AI feature costs more per user than the user is worth, no optimisation trick will save it. Price the feature, cap usage, or redesign.

Monitor costs like you monitor errors

• Log input tokens, output tokens, model and cost for every request.
• Break cost down by feature and by customer.
• Set budget alerts and per-user rate limits.
• Watch the trend per request: costs creep up as prompts and contexts grow.

Cost per user: the number that decides if a feature survives

Total monthly cost is useful, but cost per active user is what you compare with revenue. Continuing the example above with the large model:

monthly cost         ≈ $7,300
monthly active users = 5,000
cost per user        ≈ $1.46 / month

If users pay $10 a month, that's affordable. If the feature is in a free tier with 50,000 users, the same maths gives the same per-user cost but no revenue to cover it, which is why free tiers usually get smaller models, lower limits or both.

Also look at the distribution: a small share of heavy users often drives most of the cost. Per-user limits protect you from the long tail.

Latency vs cost

Cost and speed usually move together, but not always:

• Smaller models are cheaper and faster, which is the best of both.
• Shorter contexts are cheaper and faster, because less input to process means a quicker first token.
• Caching helps both: cached prefixes reduce cost and time to first token.
• Batch processing is cheaper but slower, so it's only for work that can wait.
• Self-hosting can be fast if the GPUs are close to your app and not overloaded, but under-provisioned servers get slow at peak times.

Decide which features need instant answers (chat, autocomplete) and which can wait seconds or hours (reports, tagging), and optimise each accordingly.

AI cost checklist

• Tokens, model and cost logged per request
• Cost per feature and per user tracked weekly
• Context budget per source, with trimming
• Output length limits set
• Stable prompt prefix for caching
• Response cache for repeated questions
• Routing to the smallest model that passes evals
• Per-user rate limits and budget alerts
• Batch processing for non-urgent jobs

Frequently asked questions

How much does it cost to run an AI feature?

It depends on the number of requests, the tokens per request and the model's price. Cost per request equals input tokens times the input price plus output tokens times the output price. Multiply by monthly requests for your bill; a large context and a large model can make the same feature several times more expensive.

How many tokens is a word?

For typical English text, one token is roughly four characters, or about three-quarters of a word, so 1,000 tokens is around 750 words. Code, unusual words and many non-English languages use more tokens per word.

Is it cheaper to self-host an LLM?

Only at high, steady volume, when an open model is good enough for your task and you can operate GPU infrastructure. Self-hosting has fixed costs whether or not the GPUs are busy, so at low or spiky volume, hosted APIs are usually cheaper.

How can I reduce LLM API costs?

Send less context, limit output length, use prompt caching for stable prefixes, cache responses to repeated questions, route easy requests to smaller models, and batch non-urgent jobs. Measuring tokens per request first shows where the biggest savings are.

Why are output tokens more expensive than input tokens?

Generating output requires the model to produce tokens one at a time, which uses more compute per token than reading input, which can be processed in parallel. Providers price output higher to reflect that cost.

Should I use a small or a large model?

Use the smallest model that passes your evaluation tests for each task. Many apps route simple requests such as classification and short answers to a small model, and reserve large models for complex reasoning, keeping quality high while cutting costs.

How do I calculate the cost per user of an AI feature?

Divide the monthly AI cost of the feature by its monthly active users. Compare that with what each user pays and look at the distribution, because a small group of heavy users often drives most of the cost; per-user limits help keep it predictable.

What is the cheapest way to add AI to an app?

Use a small hosted model, send only the context you need, cap output length, cache repeated answers and prompt prefixes, and batch non-urgent work. Measure tokens per request from a prototype before choosing a model or hosting approach.