Stop Throwing Money at LLM APIs: The Honest Truth About Where Your Costs Actually Live

I spent three weeks last year optimizing database queries that were costing us maybe $200/month in infrastructure. Meanwhile, I was casually firing off API calls to Claude and GPT-4 with zero visibility into what we were actually spending. When my co-founder finally pulled the AWS bill, we realized our LLM costs had silently become our biggest infrastructure expense. That's when I realized I'd been thinking about this problem completely backwards.

Most developers treat LLM API costs like a force of nature—something you accept and move on. But after digging into how token pricing actually works, I found that 60% cost reductions aren't some fantasy. They're just the result of paying attention to where the money actually flows. And the answer isn't what most people think.

The Token Cost Breakdown Nobody Talks About

Here's what surprised me: everyone obsesses over output token costs, but that's rarely where the problem lives. In real production systems, you're sending the same system prompt, the same retrieved documents, the same few-shot examples with every single request. That's redundant input tokens, multiplied across thousands of calls per day.

I started instrumenting every LLM call in my projects—logging token counts, latency, and estimated costs. Without this data, you're genuinely just guessing. I built a simple middleware that tracks call types and their cost impact:

@dataclass
class LLMCallRecord:
    model: str
    call_type: str
    input_tokens: int
    output_tokens: int
    latency_ms: float
    cached: bool = False

    @property
    def estimated_cost_usd(self) -> float:
        rates = {
            "gpt-4o": {"input": 0.0000025, "output": 0.00001},
            "gpt-4o-mini": {"input": 0.00000015, "output": 0.0000006},
            "claude-sonnet-4-6": {"input": 0.000003, "output": 0.000015},
        }
        rate = rates.get(self.model, {"input": 0.000003, "output": 0.000015})
        return (self.input_tokens * rate["input"]) + (self.output_tokens * rate["output"])

Once you have a week of data, the real culprits become obvious. For us, it was document retrieval queries with identical system prompts getting asked slightly differently by users. That's where semantic caching became a game-changer.

Semantic Caching Actually Works (If You Get It Right)

The principle is simple: cache responses not by exact string match, but by semantic similarity. When a user asks "What's your refund policy?" and another asks "Can I get my money back?", these should hit the cache, not trigger separate API calls.

The implementation uses embeddings to find similar queries above a threshold. After a few weeks of traffic, we were seeing 30%+ cache hit rates on user-facing queries. The embedding calls for cache lookups cost maybe 1-2% of a full LLM completion.

But here's where I had to be careful: the similarity threshold matters enormously. 0.95 works fine for factual queries. For generative or creative tasks, you probably shouldn't cache at all—users don't want to receive each other's generated content.

System Prompts Are Silent Cost Killers

This one genuinely frustrated me. System prompts grow organically. You add edge case handling. You add examples. You add clarifications. Six months later, a 200-token prompt has become 1,500 tokens, and you're paying that tax on every single call.

I started doing quarterly audits of prompts for redundancy. Most of the time, you find instructions that are either redundant with earlier ones or no longer relevant. I've cut system prompts by 30-40% just by being ruthless about what's actually necessary.

My Take: Visibility Changes Everything

What resonates with me about this approach is that it starts with measurement, not assumptions. Too many teams reach for model switching or aggressive prompt compression without knowing what's actually expensive. That's backwards.

I'm skeptical of one thing though: semantic caching assumes your cache backend can handle large embedding indexes efficiently. If you're using Redis with Python lists, you'll hit performance walls pretty quickly. I've started experimenting with vector databases for this, but that's adding complexity.

The real insight is this: most of your LLM bill isn't about choosing between GPT-4 and GPT-4 mini. It's about redundant context being sent repeatedly. Fix that first, and the math takes care of itself.

What's Your Hidden Expense?

I'm genuinely curious what other developers are finding when they instrument their LLM calls. Are you seeing the same pattern of redundant input tokens? Or are you hitting a different cost bottleneck? Drop me a note if you implement any of these techniques—I'd love to know what actually moves the needle for your workload.

Source: This post was inspired by "How We Reduced Our LLM API Costs by 60%: What Actually Worked" by Dev.to. Read the original article