The Real Cost of "Cheap" LLM APIs

GPT-4o-mini costs $0.15 per million input tokens. That sounds almost free. At a projected 10 million tokens per day, your CFO calculates $45/month and signs off. Six months later, the actual AI line item is $8,000/month and nobody understands why. The per-token price is real but it is the smallest component of your actual cost.

The Hidden Multipliers

System Prompt Overhead — Your 1,500-token system prompt is sent with every single request. If users send 100-token queries, 94% of your input tokens are system prompt. At 100,000 requests per day, your system prompt alone consumes 150 million tokens daily.

RAG Context Injection — Retrieval-augmented generation adds retrieved documents to every prompt. A typical RAG system injects 2,000–5,000 tokens of context per query. At scale, this dwarfs both the system prompt and the user query.

Retry and Fallback Costs — Model errors, rate limits, and timeout retries add 5–15% to your base token consumption. Most cost projections ignore this.

Embedding Costs — Every RAG query requires an embedding call. Every document ingestion requires embedding generation. These are cheap per call but add up at volume.

Evaluation and Monitoring — If you are running automated quality checks (using an LLM to evaluate another LLM's output), that is additional inference cost that does not appear in your initial projections.

The Engineering Cost

Token costs are only the API line item. The total cost includes engineering time to build and maintain the AI pipeline, infrastructure costs (databases, caching, compute), monitoring and observability tools, and ongoing prompt engineering and quality maintenance. For most startups, engineering costs exceed API costs by 3–5x.

How to Calculate True Cost

Model your cost per user-facing query, not per token. Include system prompt tokens, RAG context tokens, output tokens, embedding calls, retry overhead, and evaluation calls. Multiply by projected daily query volume. Add infrastructure and engineering costs. That is your real number. It is always higher than the back-of-napkin calculation — but knowing it prevents budget surprises.