Optimizing LLM Token Costs in High-Volume Automation Pipelines

Running an AI pipeline at volume — a support bot answering thousands of tickets a day, a lead qualifier scoring every website visitor, a document processor chewing through PDFs — turns "which model should I use" into a real line item. The levers that actually move that number are well documented at this point: caching, routing, batching, and compression, roughly in that order of effort-to-payoff. This is what each one is worth, with real pricing and real reported results.

Lever 1: prompt caching — the highest ROI, least effort

Both major providers now discount repeated context heavily. On Anthropic's API, cache reads cost roughly 10% of the standard input price — a 90% discount — with a small write surcharge (1.25x base price for a 5-minute cache, 2x for a 1-hour cache) that a single subsequent cache hit already recoups (Respan, Finout). Concretely, on a model priced at $5/MTok input, a cache read costs about $0.50/MTok.

OpenAI's cached-token discount varies by model tier: roughly 75% off on the GPT-4.1 family, up to 90% off across the current GPT-5.x line, and as high as 98.75% off on realtime audio models like gpt-realtime-2.1. Starting with GPT-5.6, OpenAI began charging a 1.25x cache-write surcharge on both automatic and explicit caching modes, where earlier models had offered free cache writes (AI Cost Check).

What actually gets cached

Caching works on a prefix match: everything up to a cache breakpoint has to be byte-identical across requests, or the cache misses silently. Practical implication for pipeline design — put stable content (system prompt, tool definitions, reference documents) first, and volatile content (timestamps, per-request user IDs, the actual user message) last:

# Anthropic example: cache the large, stable system prompt
response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_STATIC_SYSTEM_PROMPT,  # knowledge base, instructions, etc.
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": user_query}  # varies per request — goes last
    ]
)

# Confirm it's actually hitting:
print(response.usage.cache_read_input_tokens)  # should be > 0 after the first call

A support-bot case study with a 4,000-token system prompt and a 6,000-token RAG context reported a 76% total API cost reduction at a 95% cache hit rate (AI Cost Check). If cache_read_input_tokens stays at zero across repeated calls, something in the prefix is changing on every request — a common culprit is an unsorted JSON object or a datetime.now() call inside the system prompt.

Lever 2: model routing — send the easy 80% to the cheap model

Not every request needs your best model. Lead qualification, FAQ triage, and document classification are frequently well within reach of a small, cheap model; only the ambiguous or high-stakes cases need to escalate.

RouteLLM, a UC Berkeley research project (ICLR 2025), demonstrated over 85% cost reduction on MT-Bench while retaining 95% of GPT-4-level performance — by sending only 14% of queries to the expensive model (DEV Community). Production routing platforms report a similar range in practice: organizations using model routing report 40–85% cost reductions with no measurable quality loss (IntuitionLabs). At meaningful scale — 100,000 to 500,000 daily active users — one industry estimate put the routing-vs-no-routing cost gap at $200,000–$400,000 per month (Inworld AI).

The price gap that makes routing worth building: on current Anthropic pricing, the flagship tier costs roughly 5x the cheap tier on both input and output tokens (IntuitionLabs).

A minimal routing pattern — classify complexity cheaply first, then dispatch:

def route_request(query, context):
    # Cheap, fast pre-classification (small model or heuristic)
    complexity = classify_complexity(query)  # "simple" | "complex"

    if complexity == "simple":
        model = "claude-haiku-4-5"
    else:
        model = "claude-sonnet-5"

    return client.messages.create(model=model, messages=[...])

Lever 3: semantic caching — beyond exact-match

Prompt caching only fires on an identical prefix. Semantic caching goes further: it embeds the incoming query, checks it against previously answered queries for similarity, and returns the stored answer — skipping the LLM call entirely — when a new query is close enough in meaning to one already answered, not just textually identical.

Reported results are substantial. A VentureBeat-documented case study saw a company cut LLM API costs from $47,000 to $12,700 per month — a 73% reduction — while improving its cache hit rate from 18% to 67% (VentureBeat). AWS research reported an 86% cost reduction in LLM inference with 90%+ cache hit rates at a 91% response accuracy threshold (VentureBeat). Broader survey figures put the typical range at 40–80% cost reduction with response speedups up to 250x on a cache hit, since no inference call happens at all (Percona).

The tradeoff is accuracy risk: too aggressive a similarity threshold returns a stale or wrong-context answer for a query that only looks similar. This is why AWS's reported figures pair the hit rate with an accuracy number — semantic caching needs its similarity threshold tuned against real traffic, not assumed.

Lever 4: prompt compression

LLMLingua (Microsoft Research) compresses prompts by having a small model estimate which tokens actually carry information, then dropping the rest — achieving up to 20x compression with under 2% quality loss on benchmarks including CoQA, HotpotQA, and TriviaQA (GitHub — microsoft/LLMLingua). It also cuts response latency by roughly 20–30% as a side effect of the shorter input (NeuralTrust). Even at 20x compression on a 2,400-token example reduced to 115 tokens, LLMLingua's compressed prompt tracked close to full-prompt performance and clearly beat a naive-truncation baseline (PromptHub).

This is a different lever from caching — compression shrinks what you send in the first place; caching discounts what you send repeatedly. They compose: compress a long retrieved document before it enters the prompt, then cache the compressed version if it recurs across requests. It's most useful in RAG pipelines with long retrieved contexts, where every token counts against both cost and the context window.

Lever 5: batch processing for non-latency-sensitive work

If a workload doesn't need a synchronous response — bulk document classification, offline evaluation runs, overnight report generation — both major providers offer a flat 50% discount for asynchronous batch processing. OpenAI's Batch API and Anthropic's Message Batches API both cut the per-token price in half in exchange for results landing within 24 hours instead of immediately (MakeUseOf). Anthropic's batch endpoint accepts up to 100,000 requests or 256 MB per batch, with results downloadable for 29 days (Respan).

The batch discount stacks with prompt caching — a nightly document-processing job with a shared system prompt can combine the 50% batch discount with a 90% cache-read discount on the repeated prefix, pushing per-document cost down further than either lever alone (MakeUseOf).

Putting it together: an approximate pricing comparison

Applied cumulatively to a hypothetical 1,000-request/day pipeline with a large shared system prompt, these levers are not additive percentages (they compound on different parts of the bill), but the ordering of impact based on the figures above is roughly:

Output tokens deserve the same scrutiny as input tokens

Most cost-optimization writing focuses on input — caching, compression, routing — because input tokens are the ones you control most directly before a request goes out. But output tokens are typically priced several times higher than input tokens on every major provider (roughly 4–5x on Anthropic's current lineup), so a verbose response is disproportionately expensive relative to its token count. Two practical habits catch this: capping max_tokens to what the task actually needs rather than leaving generous headroom by default, and instructing the model explicitly toward terse output for high-volume classification or extraction tasks where a JSON object or a short label is the entire deliverable. Extended-reasoning modes compound this further — a model "thinking" through a simple classification task before answering burns output tokens on reasoning that a cheaper, non-reasoning model tier would skip entirely, which is itself an argument for routing simple tasks away from reasoning-heavy models rather than just cheaper ones.

The market backdrop

None of this is optional busywork in a market where usage keeps outpacing price drops. Even as per-token prices have fallen sharply — one industry estimate put the decline at roughly 80% between 2025 and 2026 — enterprise LLM API spend passed $8.4 billion in 2025 and is on track to keep growing, because usage volume is rising faster than price is falling (Wavect). Cost optimization matters more, not less, as pipelines scale — a 90% discount on a 10x-larger bill is still a bigger absolute number to manage than it was at last year's volume.

Where to start

For a pipeline that hasn't touched any of this yet, the practical order is:

1. Cache the stable prefix first. It's the least code, highest immediate return, and every other lever benefits from having it in place.
2. Turn on batch processing for anything that doesn't need a live response. It's a flat discount with no architecture change beyond accepting a 24-hour turnaround.
3. Add routing once you understand your traffic mix. This requires knowing which fraction of requests are genuinely "simple" — measure before building the classifier.
4. Reach for semantic caching and compression once the above are captured. They require more infrastructure (embeddings, vector stores, a compression pipeline) and are worth it primarily at higher volume or with longer, more repetitive contexts.

The exotic end of the spectrum — cache eviction policies, quantization, custom attention mechanisms — is real, but it's infrastructure-team territory reserved for teams that have already captured the four levers above and are still bottlenecked on cost.

Sources: Respan — Claude Prompt Caching Pricing, Finout — Anthropic API Pricing in 2026, AI Cost Check — Prompt Caching Savings 2026, DEV Community — LLM Model Routing in 2026, IntuitionLabs — AI Model Routing: Cost and Quality Optimization Guide, Inworld AI — AI Model Routing Explained, VentureBeat — Semantic caching cuts LLM API costs by 73%, Percona — Semantic Caching for LLM Apps, GitHub — microsoft/LLMLingua, NeuralTrust — Prompt Compression Guide, PromptHub — Compressing Prompts with LLMLingua, MakeUseOf — Anthropic and OpenAI 50% Discounts, Respan — Anthropic Message Batches API, Wavect — How to Cut LLM Token Costs in 2026