Enterprise AI Costs 101

The pilot-to-production surprise

Most AI projects start the same way: a small team, a few thousand test queries, a monthly bill that barely registers. Then the decision gets made to roll it out. Usage goes from hundreds to hundreds of thousands of queries per month. And the bill that felt like nothing is suddenly a line item someone has to explain.

This guide is for the executive who is about to approve that rollout — or who is already asking "why does this cost so much?" It explains how AI is actually priced, where costs hide, and what your deployment will realistically cost before you find out the hard way.

How the whole thing actually works

Before any of the cost makes sense, it helps to see the machine end to end. Strip away the branding and every AI product on the market runs the same four steps.

1. Your text is broken into pieces — the unit you're billed in.
2. The pieces go to a model that's already sitting in memory — expensive memory, standing by whether anyone asks it anything or not.
3. The model predicts the next piece. Once. That's genuinely all it does.
4. It repeats step 3 until the answer is finished.

That's it. No database of answers, no retrieval of a stored reply. Every response is computed from scratch, word by word, at the moment you ask.

What a "token" actually is — and why it costs you more than you'd guess

A token is roughly three-quarters of a word. That's the approximation to carry around, and it's the unit every AI price list is quoted in. Here's the part that costs money: AI models can't read words at all. They read pieces, drawn from a fixed dictionary of roughly 100,000 to 250,000 fragments, depending on the model, decided before it was ever trained.

Common English words — "the", "policy", "customer" — each got their own entry in that dictionary, so they cost one piece. Anything the dictionary didn't anticipate gets chopped into fragments until it fits. Your product codes, customer IDs, internal jargon, and any language that isn't English were not anticipated.

Type into the box below to see it happen. Each coloured block is one piece — one unit on your bill.

Approximate. Real tokenisers use a fixed 100,000-entry dictionary that can't be shipped to a web page, so this reproduces the behaviour rather than the exact vocabulary. Counts land within roughly 10–15% on ordinary prose — the right precision for a budgeting conversation, not for reconciling an invoice.

Ordinary English runs at roughly 1.2–1.3 tokens per word. That's the number every published price list quietly assumes. Now click the Hindi and Tamil samples and watch the ratio move.

The same effect hits anything else the dictionary didn't expect. Order IDs and SKUs shatter into fragments. Structured output — the JSON your systems actually consume — pays for every brace, quote and colon. Specialist vocabulary in pharma, law and insurance fragments harder than everyday English. None of this is waste you can eliminate, but all of it is cost you should see coming.

What "inference" actually means — and what one query takes

There are exactly two things anyone does with an AI model, and only one of them is on your invoice.

• Training is building the model: feeding it a large fraction of the written internet over months, across thousands of chips, to produce a finished set of numbers. It costs well over $100 million for a frontier model, happens once, and you never see it as a line item. It's amortised into the per-token price everyone pays.
• Inference is using the finished model to answer a question. It happens every single time anyone touches your system. This is your bill. Every cost in this guide is an inference cost.

The distinction matters because the intuition most executives carry is that AI is expensive to build and then basically free to run, the way software normally works. Software is written once and copied for nothing. AI is different: every single answer is freshly computed. There is no library of pre-written replies to hand out. You can cache the repeated setup — more on that shortly, and it matters — but the answer itself is new work every time.

What one query costs you

When a query arrives, three things have to happen:

1. The model has to already be in memory. Hundreds of gigabytes of numbers, loaded onto the chips and standing by. This is a standing cost, paid whether or not a query ever arrives. It's the floor under everything.
2. The model reads your prompt. It takes the whole thing in one go rather than word by word, which makes reading far more efficient than writing — though a longer prompt still costs proportionally more.
3. The model writes the reply, one word at a time. And here's the part that determines your bill: each word requires another full trip through the model. Not a lookup, not a shortcut. Modern models are built to wake only the portion of themselves a given word needs, but that still means an enormous quantity of numbers hauled out of memory, hundreds of times per answer.

That's why the two halves of your query are priced so differently:

That asymmetry explains a line on every invoice you'll ever see: output costs about 5× more than input. It also hands you a cheap lever. Telling the system "answer in three bullet points, not three paragraphs" is not a style preference — it's a direct cut to the expensive half of the bill. Verbose AI is expensive AI, and most teams never tune this.

The price ranges above span the cheapest models to the most expensive; Section 2 explains what sets your position in that spread.

Why it's affordable at all: you're carpooling

If every query needed its own dedicated machine, none of this would be economic. It works because the provider runs many customers' queries through the same pass simultaneously. The expensive part — hauling the model's numbers out of memory — happens once and serves dozens of unrelated users at the same instant.

You are carpooling on someone else's chip, and the per-token price is your share of the ride. Nearly every cost dynamic in this guide falls out of that one fact: prices drop when providers pack the car more efficiently, and self-hosting is expensive because you'd be driving alone.

A worked example: what one customer query really contains

Tokens and inference in one picture. When a customer types a 6-word question to your AI support bot, the AI reads far more than 6 words — and this single diagram explains why pilots feel cheap and deployments don't.

Every production AI system has three layers of hidden "overhead" on top of the user's message:

1. The instruction manual — also called the "system prompt." This is where you tell the AI how to behave: your brand tone, what it can and can't say, escalation rules. It's typically 500–1,500 words, and the AI reads it on every query.
2. The knowledge context — if your AI is pulling from your product docs, policies, or knowledge base, those relevant sections are also passed to the AI on each call.
3. Conversation history — in a multi-turn chat, the AI receives the full prior conversation every time to maintain context.

The result: a "simple chatbot" that handles 5,000 queries per day is not sending 5,000 short messages. It's sending 5,000 large packets, each several times bigger than the user's actual words. The sticker price assumes none of this. Your real bill reflects all of it.

How the meter runs

You now know what a token is and what inference does. Pricing follows directly from both: every AI interaction is billed like a courier shipment, charged by weight, in both directions.

You pay to send the package — your question, plus everything the system attaches to it. Then you pay again for the reply to come back. Nobody asks how valuable the contents are; the meter only measures how much was moved. A one-line question wrapped in a 600-word instruction manual is billed as the whole bundle, not the one line.

And as Section 1 showed, the return leg is the expensive one.

The model you choose determines 90% of the cost

Not all AI is priced the same. There's a 100× price difference between the cheapest and most expensive models from major providers. Think of it like flights: economy gets you there, business class is more comfortable, and a private jet costs orders of magnitude more. Most of the time, economy works fine.

Where the money physically goes: GPUs

The chips that run inference are GPUs, and the economics of your bill are the economics of that hardware.

A single top-end AI chip costs roughly $25,000–$40,000 to buy and rents for a few dollars an hour, more through the big cloud providers. Large models don't fit on one — they need several chips wired together, standing by as a unit. Your per-token price is the provider taking that hourly cost and dividing it across everyone sharing the machine. When you hear that AI prices keep falling, that's the reason: better hardware and denser packing of queries onto each chip, not generosity.

Why memory, not speed, is the real constraint

Here's the thing most cost conversations miss entirely. The scarce resource inside an AI system isn't processing speed. It's memory on the chip — and it gets consumed in two places.

First, the model has to sit there. We covered this as the standing cost, but it's worth naming what it buys you: this is most of why the flagship tier costs 100× the economy tier. A flagship model is simply a much larger pile of numbers that has to be held in expensive memory and hauled through on every word it writes. You're not paying 100× for a cleverer answer so much as for a far bigger thing being kept resident and re-read constantly. Which is exactly why matching the tier to the task matters so much: on a routine classification job, you're renting a warehouse to store an envelope.

Second, every live conversation holds a working memory of its own. As the model reads your prompt, it builds up notes on everything it has seen so it doesn't have to re-read the conversation from scratch for every new word it writes. Those notes live in chip memory for the duration of the conversation, and they grow with every turn.

This single mechanism explains most of the behaviour you've noticed and been unable to explain:

• Why context windows have limits. The advertised limit isn't arbitrary — it's partly how much working memory one conversation is allowed to occupy, and partly the point beyond which the model was never trained to stay reliable.
• Why long conversations get slower and more expensive. Turn twenty genuinely costs more than turn one, on the same model, for the same question length.
• Why "just give the AI all our documents" is expensive advice. Every document you stuff into the prompt occupies memory on every query, forever. Retrieving the three relevant paragraphs beats sending the whole handbook.

Why this needs a special kind of memory (HBM)

Here's the part that connects everything above to a physical component you'll see in the news. Remember that writing an answer means running the entire model again for every single word. That means the chip has to read hundreds of gigabytes of numbers, from memory, per word.

Ordinary computer memory cannot feed a chip that fast. It isn't a matter of capacity — it's a matter of how quickly bytes can be moved. So AI chips use High Bandwidth Memory, or HBM: memory stacked in layers and bonded directly onto the processor package, sitting millimetres away instead of centimetres, with a far wider path between the two.

This is why inference is described as memory-bound rather than compute-bound: the chip usually isn't thinking hard, it's waiting for numbers to arrive. That is precisely why providers pack many customers into the same pass — it's the one thing that puts a waiting chip back to work. It also explains three things you may have read about without the connection being made. HBM is the genuine bottleneck in AI hardware — the constrained component the entire industry is fighting over, and a large share of what an AI chip costs to build. It's why memory manufacturers became AI stocks. And it's why your per-token price is what it is: you are renting bandwidth to a very expensive piece of silicon, shared with everyone else on that chip.

For why this constraint is reshaping the whole hardware market, see our companion piece: The Memory Boom.

The work that isn't the model

Here's where most cost models break. Everything so far has priced the AI. Almost none of the work in a real deployment is the AI.

A demo takes an afternoon: point a model at some documents, ask it questions, show the room. It works because a demo has no wrong answers, no unauthorised users, no systems of record, and no auditor. Production has all four.

What the demo quietly skipped

• Your documents are not ready. They're scanned PDFs, three conflicting versions of the same policy, and a SharePoint nobody has pruned since 2019. If the AI retrieves the superseded policy, it will state it with total confidence. Cleaning and structuring this is usually the single largest work item, and it is unglamorous enough that nobody scopes it.
• Permissions are the hard part. This is where enterprise projects actually stall. Your existing access rules live in your applications — they do not automatically apply to a search index built from your documents. Build the retrieval layer without rebuilding entitlements into it, and the system will eventually show someone a salary band, a board pack, or another customer's file. Not as a bug, but as designed behaviour nobody specified.
• It has to write back. An assistant that reads is a toy. One that raises the ticket, updates the CRM, and books the follow-up is a system — and now you're integrating with systems of record, which brings their release cycles and their change boards with it.
• You need a way to know it's right. AI needs a graded set of real questions with known-good answers, re-run on every change. Teams that skip this cannot tell whether last week's prompt tweak improved things or quietly broke them.
• Somebody has to be in the loop. Where does it escalate, who reviews low-confidence answers, and what does the user see when it doesn't know? Answer this early or your support team inherits it.
• Your monitoring doesn't cover this. Existing tools watch uptime and latency. They do not watch token spend per feature, or answer quality drifting after a model update. Both will surprise you, one on the invoice and one in front of a customer.

Where the first year's money goes

Everything priced in this guide so far is the API bill. For a first serious deployment, that is typically the smaller share of what you spend in year one. Rough shape of a first-year budget:

Indicative shares for a first enterprise deployment. The API share rises in later years as build costs fall away and usage grows — which is exactly when the model-tier and caching decisions in Section 2 start compounding.

The layered stack, before and after

Your enterprise stack has looked roughly the same for two decades. GenAI does not replace those layers. It adds six of them, and changes the character of three you already run.

The three that will bite you

1. Your application layer stops being deterministic. This is the deep one. Traditional software is a promise: the same input produces the same output, forever, and you test it with assertions that pass or fail. A model gives you a different answer to the same question on Tuesday than it did on Monday — and a model update you didn't control can shift behaviour across every feature at once. Testing becomes statistical rather than binary: not "does it pass" but "is it still right often enough." That changes QA, your release process, and what your on-call engineer can even do at 2am.

2. Access control moves down the stack. As above: your existing rules live in the application, and the retrieval index does not inherit them. Retrofitting this after launch is painful, and it is a common reason pilots never reach production.

3. Caching stops working the way you know. You can't cache answers when every question is phrased differently. What you cache is the repeated input instead — and your CDN knows nothing about it.

Four use cases, four cost shapes

"AI" isn't one thing on your bill. Each use case assembles the same pieces into a different shape, and each shape has its own cost signature. Here's what's running behind the four most common enterprise deployments — and where the meter actually sits in each.

What it costs by use case — real numbers at 5,000 queries/day

Here is the API bill on its own, at a realistic production volume. Keep the budget table from Section 3 in mind: this is the 15–30% row, not the whole invoice. It is, however, the row that scales with every new user you add — and the only one you can change with a configuration decision rather than a project.

Computed from published list prices (July 2026) at 5,000 queries/day, or ~150,000/month, carrying the production context overhead from the worked example in Section 1 — roughly 1,900 tokens in and 260 out for a chat query. Tiers priced on Claude Haiku 4.5, Sonnet 4.6 and Opus 4.8 respectively; other providers sit in similar bands. Agentic estimates assume ~8 AI calls per task. Ranges are lean vs. full-context implementations.

Three questions to answer before you approve the budget

You don't need to understand any of the machinery in this guide to run a good approval conversation. You need three questions, and the confidence to keep asking until the answers are specific.

The bottom line

The economics here are genuinely good. A support bot that costs a few hundred dollars a month against a contact centre that costs thousands is not a marginal case, and the models keep getting cheaper for the same quality.

The surprise is not the price of the AI. It's that the AI is the small part. Most of what you will spend goes on getting your documents in order, rebuilding who-can-see-what into a new layer, wiring the thing into systems that already exist, and learning to tell whether it is still right. None of that appears on the invoice you have been shown, and all of it appears on the one you eventually pay.

So when the number comes back and someone has to explain it: ask which of those it covers. That single question is the whole guide.