What Does It Cost to Run an AI Feature in Production?

What does it cost to run an AI feature in production? For most apps it's a per-request cost of a fraction of a cent to a few cents, driven almost entirely by how many tokens go into and out of the model on each call. A chatbot answering short questions might cost $20-100 a month at low traffic. The same feature with a long system prompt, large retrieved context, and no caching can cost ten times that for identical usage. The build is a one-time number. The running cost is the one that surprises people, and it's almost always fixable.

I've built AI features that cost almost nothing to run and inherited ones that were quietly burning hundreds of dollars a week on traffic that didn't justify it. The difference is rarely the model choice. It's usually a few decisions about context, caching, and model tier that nobody revisited after launch.

Here's how the money actually works, with real math instead of "it depends."

How Is an AI Feature Actually Billed?

Almost every hosted model (OpenAI, Anthropic, Google, and the rest) bills you per token. A token is roughly ¾ of a word. You pay for input tokens (everything you send: the system prompt, the conversation history, any retrieved documents, the user's message) and output tokens (what the model generates back). Output tokens usually cost more than input, sometimes three to five times more.

So a single request's cost is:

(input tokens × input price) + (output tokens × output price)

Both prices are quoted per million tokens. As of mid-2026, a mid-tier general-purpose model sits somewhere around a dollar or two per million input tokens and a few dollars per million output tokens. The cheapest small models are a fraction of that. The top-end reasoning models are several times more. These numbers move — usually downward — every few months, so check the current pricing page of whichever provider you're using before you commit to an estimate.

The trap is that the numbers look tiny in isolation and add up fast at volume. A request costing half a cent is nothing. A hundred thousand of them is $500.

What Does a Real Request Actually Cost?

A typical support-assistant request that answers from your help docs costs about one cent: roughly 5,100 input tokens (system prompt, retrieved context, conversation history, the question) and 400 output tokens, at around $1.50 per million input and $6 per million output. Run 30,000 of those a month and the model bill is about $300. Here's where each of those tokens comes from.

Say the assistant uses RAG to pull answers from your documentation. A typical request breaks down as:

• System prompt and instructions: ~800 tokens
• Retrieved doc chunks pulled in as context: ~3,000 tokens
• Conversation history so far: ~1,200 tokens
• User's new question: ~100 tokens
• Input total: ~5,100 tokens
• Model's answer: ~400 tokens output

That's the $300/month figure above. Now change two things. Drop the retrieved context from 3,000 tokens to 1,200 by retrieving fewer, more relevant chunks, and enable prompt caching on the static system prompt so you're not re-billed full price for those 800 tokens every call. The same 30,000 requests now cost closer to $120/month. Same feature, same traffic, less than half the bill, and often a better answer because the model isn't wading through loosely relevant context.

That's the whole game. The model you pick matters less than what you feed it.

Which Costs Blow AI Budgets in Production?

A handful of patterns account for most of the surprise bills I've seen:

• Unbounded conversation history. Sending the entire chat back on every turn means a long conversation's 20th message costs many times what its 2nd did. Cap the history, summarize older turns, or both.
• No prompt caching. If your system prompt and instructions are stable across requests (they usually are), caching them cuts the input cost on that portion dramatically. Leaving it off is money on the floor.
• Over-retrieval in RAG. Pulling 10 document chunks "to be safe" when 3 would answer the question triples the input cost of every single call and can make answers worse.
• Using a flagship model for everything. Routing simple classification or extraction to a top-tier reasoning model is like hiring a senior engineer to reset passwords. Cheap small models handle a lot of real work.
• Retry storms. A flaky pipeline that retries failed calls without backoff can multiply your bill during an incident. Retries need limits.
• Agent loops. An agent that calls itself repeatedly with no step cap can burn tokens fast — and if it can also spend money, the cost isn't only tokens. AI agent security — what can go wrong covers the limits an agent needs.
• Streaming that never stops. An agent stuck in a loop, or a generation with no max-token cap, can run up a large single request. Always set output limits.

None of these are exotic. They're all things that get set once during the build and never looked at again.

What About the Costs That Aren't Tokens?

Tokens are usually the biggest line, but not the only one. For a RAG feature you also pay for:

• Embeddings. Turning your documents into vectors costs tokens too, though embedding models are cheap and it's mostly a one-time cost per document plus a tiny cost per query. A knowledge base of a few thousand documents costs a few dollars to embed, not hundreds.
• Vector storage. A managed vector database runs anywhere from free at small scale to tens or low hundreds of dollars a month as your document count and query volume grow. For small knowledge bases, the vector store built into Postgres (pgvector) is often enough and adds no separate bill.
• Hosting the app itself. The API route or worker calling the model still runs on your normal infrastructure. For most apps this is already covered by what you're paying for the rest of the backend.

For a typical small-to-mid production feature, non-token costs are usually $0-50/month. They only get significant at large scale or with a big, frequently-updated knowledge base.

When Does Self-Hosting a Model Actually Save Money?

This is the question everyone asks and the answer is: later than you think.

Running an open-weights model (Llama, Mistral, Qwen, and similar) on your own GPU infrastructure trades a per-token cost for a fixed hourly cost. As of mid-2026, a capable GPU instance runs somewhere in the range of $1-4 an hour, which is roughly $750-3,000 a month if it's always on. Plus the engineering time to set it up, keep it running, handle scaling, and stay on top of model updates.

That fixed cost only beats per-token API pricing once your volume is high and steady enough to keep the GPU genuinely busy. As a rough line: if your monthly API bill for a given feature is under about $500-1,000, self-hosting will almost certainly cost more, not less, once you count the infrastructure and the time. Above a few thousand dollars a month of steady, predictable usage, it starts to be worth modelling seriously.

There are non-cost reasons to self-host too (data can't leave your environment, you need a fine-tuned model, you want no external dependency), and those are often the real driver rather than the money. But "it'll be cheaper" usually isn't true until you're much bigger than a first production feature.

How Do You Estimate the Cost Before You Build?

You can get a usable estimate in about ten minutes:

1. Write out a realistic single request: system prompt, expected context size, history, user input, expected output length. Add up the tokens (most providers have a tokenizer tool, or estimate at ¾ word per token).
2. Multiply by the model's input and output prices to get a per-request cost.
3. Multiply by your expected monthly request volume.
4. Add a rough non-token line ($0-50/month for most small features, more for a large knowledge base).
5. Double the whole thing as a safety margin for the first few months while you tune it.

If that number is comfortable, build it. If it's alarming, the fix is almost always in step 1 (smaller context, caching, cheaper model for part of the work), not "don't build it."

Where This Fits With What I Build

If you're adding an AI feature to an existing product and want the running cost sane from day one, that's AI Integration for Existing Apps — the integration work includes the caching, context, and model-tier decisions that keep the bill down. If it's a new build that's AI-native from the start, that's Full Stack AI Development. And if the "feature" is really an automated workflow rather than a user-facing thing, the cost math is different again — n8n vs a custom AI agent covers that side.

Pricing for the build is scoped per project, not a fixed rate. Tell me what the feature does and roughly how much traffic you expect, and I'll give you a real estimate of both the build and what it'll cost to run.