How to Cut LLM Application Costs With Token Budgets

An LLM bill is tokens in, tokens out. Tokens are the pieces of text a model counts, roughly word fragments. Two habits drive most of the bill: resending the same big prompt on every call, and guessing at costs instead of reading them from the API response.

This post shows how to see that in your own numbers, then four levers in the order I would try them. I ran the token-counting and cost code below with Python 3.12 and tiktoken. I did not call any provider API. Prices and model names change faster than blog posts, so there are none here: copy rates from the provider’s pricing page when you need them.

Step 1: Count what you resend #

Every call to a chat model carries the system prompt, the tool definitions, the conversation so far, and the new message. Here is a toy support bot: a 601-token system prompt and short turns (13 tokens from the user, 15 back), counted with tiktoken’s o200k_base encoding.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
n = lambda s: len(enc.encode(s))

SYSTEM = "You are a support assistant for Acme. Answer from the policy text. " * 40
USER_TURN = "Where is my order #1234? It was due Tuesday."
ASSISTANT_TURN = "It shipped Monday and is with the carrier; tracking is in your email."

history = resent = 0
for turn in range(1, 21):
    resent += n(SYSTEM) + history + n(USER_TURN)    # everything is resent
    history += n(USER_TURN) + n(ASSISTANT_TURN)

Measured output: the input for one call grows from 614 tokens at turn 1 to 866 at turn 10 and 1,146 at turn 20. Across all 20 calls the bot sends 17,600 input tokens. The fixed system prompt alone accounts for 12,020 of them, 68%.

That result surprised me, and it changes the advice. The usual first suggestion is to trim or summarize conversation history. In this test, keeping only the last four exchanges saved 3,248 of 17,600 tokens (18%), because the turns were short. Your numbers will differ: a bot whose users paste documents has long turns and history matters more. The point is to run this count on your real prompts before choosing a lever.

Step 2: Read cost from the response, not from an estimate #

Providers return token counts with each response. Use those, not a local estimate, for billing math. With Anthropic’s API the usage block has three input counters: cache_creation_input_tokens, cache_read_input_tokens, and input_tokens, which counts only the tokens after the last cache breakpoint, not all input (prompt caching docs). A cost function has to add all three:

from dataclasses import dataclass

@dataclass(frozen=True)
class Rates:            # dollars per million tokens, from the provider's pricing page
    input: float
    output: float
    cache_write: float
    cache_read: float

def request_cost(usage, rates: Rates) -> float:
    return (usage["input_tokens"] * rates.input
            + usage["cache_creation_input_tokens"] * rates.cache_write
            + usage["cache_read_input_tokens"] * rates.cache_read
            + usage["output_tokens"] * rates.output) / 1_000_000

I ran this with made-up rates (input 1.00, output 5.00, cache write 1.25, cache read 0.10) and two usage blocks: a cold call (20 uncached input tokens, 600 written to the cache, 15 output; cache_read_input_tokens 0) cost $0.000845, and the same call reading the 600 tokens from the cache (cache_creation_input_tokens 0) cost $0.000155. Log this number per request with a feature name and a user or tenant id, and you can answer “which feature costs the most” from your own database. Anything beyond that, a dashboard or an alert, is yours to build; I have not tested one.

Step 3: Prompt caching #

Caching lets the provider reuse the processed form of a prompt prefix instead of recomputing it, at a lower price per token. For a bot where the fixed prefix is 68% of input, this is the lever that matches the shape of the cost.

What Anthropic’s documentation says:

• A request can turn caching on with one top-level "cache_control": {"type": "ephemeral"} field, and the cache point moves forward as the conversation grows.
• Cache reads cost 0.1x the base input price on most models (per-model exceptions are in a table on that page). Writing with the default 5-minute lifetime costs 1.25x. A 1-hour lifetime costs 2x.
• A prompt shorter than the model’s minimum cannot be cached, and no error is returned. To see whether caching happened, check the usage fields: if both cache_creation_input_tokens and cache_read_input_tokens are 0, nothing was cached.

OpenAI’s prompt caching guide describes the same idea with differences in lifetime settings and minimum prefix length that depend on the model, and reports cached_tokens in the usage block. Read the page for your model rather than trusting a summary.

How much does it save? Two sources that measured it:

• Lumer et al. ran 500+ agent sessions with 10,000-token system prompts across OpenAI, Anthropic and Google. They report caching cut API cost by 41-80% and time to first token by 13-31%. They also found that putting dynamic content at the end of the system prompt and excluding dynamic tool results from the cached part gave more consistent gains than naively caching the whole context (arXiv 2601.06007).
• Anthropic’s own cost guide reports that over a day of real traffic its agent loops read a median 84% of their input from the cache, and advises looking for something breaking the cache if your share falls below about 80% (Optimizing for cost and intelligence).

Both are agent workloads with long, stable prefixes. A bot with a short prompt and no repeated prefix will see less. The cause of a poor hit rate is nearly always the same: caching matches a prefix, so anything that changes early in the prompt, such as a timestamp, a user name or a reordered tool list, invalidates everything after it. Put the stable text first and the changing text last.

Step 4: Batch what can wait #

Work nobody is waiting for, such as nightly classification or backfills, does not need an instant answer. Anthropic lists batch processing at 50% off for work that can wait up to 24 hours (cost guide). The trade is latency for price. Check your provider’s batch page for its terms.

Step 5: Do not pay twice for an identical request #

If the same prompt really does repeat, such as a product description generated from the same fields, cache the answer yourself:

import hashlib, json

def cached_complete(cache, model, prompt, complete):
    key = hashlib.sha256(json.dumps([model, prompt]).encode()).hexdigest()
    if key not in cache:
        cache[key] = complete(model, prompt)
    return cache[key]

I tested it: 1,000 calls with the same prompt plus one different prompt made 2 calls to the provider. In production, cache would be Redis or Rails.cache, with an expiry, rather than a dict.

The condition that matters: only use this when the prompt fully determines the answer you want. If the prompt contains per-user data, the key is different for each user and you gain nothing. If the answer should vary between calls, you have changed the behavior. Matching on similar prompts (semantic caching) serves one user’s answer to a different question when the match is wrong, so test it against real traffic before turning it on. I have not measured it here.

Step 6: The cheapest model that passes your test #

Not every call needs the largest model. Classification, extraction and routing are common candidates for a smaller one. The rule that holds up is to write 30 to 100 real examples with the answers you expect, run each model tier against them, and pick the cheapest tier that passes. Vendor claims about which model is good enough for your task are not a substitute. I have no measured numbers for model tiers in this post, so I have given none.

What to do Monday #

1. Count tokens for your real prompts as in Step 1 and find your fixed prefix.
2. Log cost per request from the usage block, tagged with feature and tenant.
3. Turn on prompt caching and check the cache-read counter.
4. Move stable text to the front of the prompt.
5. Batch the jobs nobody waits on.

Sources #

• Anthropic: prompt caching - top-level cache_control, price multipliers, usage fields, silent minimum
• Anthropic: optimizing for cost and intelligence - cache-read share in agent loops, batch at 50% off
• OpenAI: prompt caching - how caching works on OpenAI models and the cached_tokens counter
• Lumer et al., Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks - 41-80% cost and 13-31% time-to-first-token reduction across three providers
• tiktoken - local token counting for OpenAI encodings

Related: building RAG in Rails with pgvector.