LLM Cost Optimization: A Practical Inference Playbook

What is LLM cost optimization?

LLM cost optimization is the practice of minimizing the cost of a successful AI task, not merely the price of a model call. The useful unit is usually cost per resolved support case, approved document, qualified recommendation, or completed workflow.

A cheaper request that fails and requires a retry, a human correction, or a second model can be more expensive than a strong first response. Optimize cost, quality, latency, and operational effort together.

Start with a unit-economics model

For each workflow, record:

task cost =
  input token cost
  + cached input cost
  + output token cost
  + tool and retrieval cost
  + retry cost
  + human review cost

Segment by route, model, customer tier, and outcome. A single monthly API total cannot show whether long retrieval context, verbose output, failed tools, or the wrong model is driving spend.

The model catalog and pricing change over time, so keep prices in configuration rather than application logic. Store the model identifier with every trace so historical cost remains explainable.

Which optimization should come first?

1. Remove context that does not help

Inspect what the model actually receives. Common waste includes repeated policy text, entire conversation histories, oversized retrieval chunks, duplicate tool descriptions, and JSON fields the model never uses.

Use retrieval thresholds and reranking to send the smallest evidence set that still supports the answer. Summarize old conversation turns, but retain exact facts and decisions that later steps depend on.

2. Control output length

Output tokens are often priced differently from input tokens and add visible latency. Ask for the shortest format that completes the task. Use structured fields for machine consumers and let the interface expand details on demand.

Do not cut the response so aggressively that the user needs another turn. “Concise” should mean complete without repetition.

3. Reuse stable prompt prefixes

Prompt caching rewards repeated prefixes on supported models. Place stable instructions, schemas, and shared context before variable user content. Keep the prefix byte-for-byte consistent where the provider’s caching design requires it, and inspect usage fields rather than assuming a cache hit.

4. Route by difficulty

A classifier or deterministic rule can send extraction, formatting, or narrow classification tasks to a smaller model while reserving stronger models for ambiguous reasoning, high-risk decisions, and complex tool use.

Routing only works with evaluations. If the smaller route produces more retries or silent errors, the nominal saving is false.

How should a model router be designed?

Prefer explicit features—task type, document length, required tools, risk class—over asking another expensive model to route every trivial request. Add a fallback when the first route reports low confidence, invalid structure, or an evaluation-defined failure.

How do caching and batching help?

Prompt caching is suited to repeated long prefixes, such as a stable system policy or shared codebase context. Track cached tokens and time to first token. A prompt that changes near the beginning may prevent reuse even when most later content is identical.

Batch processing is suited to work that does not need an immediate response: nightly classification, offline enrichment, evaluation runs, and bulk migrations. OpenAI’s Batch API documentation describes asynchronous processing with a completion window. The operational benefit is not only price; batching also separates background throughput from interactive latency requirements.

How does retrieval affect inference cost?

Retrieval can reduce hallucination and keep knowledge current, but careless retrieval inflates every prompt.

• Chunk around semantic boundaries rather than fixed size alone.
• Filter by tenant, product, language, and time before vector search.
• Rerank a wider candidate set, then send only the strongest evidence.
• Deduplicate overlapping passages.
• Cite the selected evidence so unsupported context can be detected.
• Cache retrieval results only when permissions and freshness allow it.

Measure answer quality as context shrinks. The target is not the fewest tokens; it is the smallest context that preserves supported answers.

When should you distill or fine-tune?

Consider fine-tuning when the task is stable, repeated, and difficult to specify efficiently with prompts—for example a consistent classification taxonomy or output style. It does not replace live knowledge retrieval.

Distillation can use a strong model to produce or label examples for a smaller model, but the resulting dataset needs review, diversity, and held-out evaluation. Do not train a smaller model on unverified production mistakes.

An optimization loop that preserves quality

1. Define the successful task and risk threshold.
2. Build a representative evaluation set.
3. Record tokens, latency, tools, retries, and outcome.
4. Change one lever: context, output, cache layout, route, or model.
5. Compare cost and quality on the same cases.
6. Canary the change in production.
7. Roll back automatically when quality or reliability crosses its limit.

This loop turns cost work into controlled engineering. Without it, teams often celebrate a lower token bill while moving cost into support tickets and human review.

Sources and further reading

• OpenAI cost optimization guide
• OpenAI prompt caching guide
• OpenAI Batch API guide
• OpenAI model catalog