AI Cost Optimization — Token-Saving Engineering Stack

What's in this pack

When the monthly LLM invoice crosses five figures, every engineer suddenly has an opinion about caching. This pack is the boring, ordered playbook that actually moves the number: measure before you optimize, cache before you route, route before you compress, compress before you fine-tune. Most teams that touch every layer in order book 10–50% savings without users noticing — the upper end shows up in support chat (high cache hit) and bulk classification (small fine-tuned models), not in greenfield agent reasoning where you should be careful.

Install in this order — measure first, fine-tune last

# Layer 1: measure (do this before changing anything)
tokrepo install tokencost                  # offline price model
tokrepo install litellm-cost-tracking      # live per-project dashboard

# Layer 2: cache (highest ROI, lowest risk for repetitive workloads)
tokrepo install helicone-cache             # drop-in response cache
# or: tokrepo install cloudflare-ai-gateway  # edge cache, free tier

# Layer 3: route (cheap-model-first with safe escalation)
tokrepo install litellm-proxy              # self-hosted, BYOK
# or: tokrepo install openrouter-unified-api  # hosted, 300+ models
tokrepo install manifest-smart-router      # semantic router on top

# Layer 4: compress (only after layers 1–3 are baselined)
tokrepo install llmlingua                  # 2–20× prompt compression

# Layer 5: fine-tune (last resort — costs engineering time)
tokrepo install fireworks-fine-tune        # LoRA on Llama, narrow task

The order matters more than the picks. Skip ahead and you'll either burn engineering time fine-tuning a model that a cache would have replaced, or you'll silently degrade quality with prompt compression you can't attribute because you never instrumented the baseline. The unglamorous truth: the biggest savings usually come from layer 2 (cache) and layer 3 (route), not the fashionable layer 5.

Layer 1 — measure. Install TokenCost as a library so every PR prints before/after token math on staging. Install LiteLLM cost tracking (or Portkey) so production has a per-call ledger by project / user / model. Don't move on until you can answer 'what does one user session cost' to two significant figures.

Layer 2 — cache. Helicone gives you exact-match caching via a single proxy header and semantic caching via embedding similarity. Cloudflare AI Gateway gives you the same at the edge with a free tier. For chatbots, FAQ, RAG Q&A, and idempotent classification, hit rates of 30–70% are realistic. For agent planning loops and creative generation, they are not — don't try.

Layer 3 — route. LiteLLM Proxy is the self-hosted default — one OpenAI-compatible URL maps to Anthropic, Bedrock, Vertex, OpenAI, with failover chains and per-key budgets. OpenRouter is the hosted equivalent. Manifest sits on top to classify prompts and route cheap models first, escalating only on low confidence. Portkey adds enterprise features (SSO, audit, virtual keys, guardrails).

Layer 4 — compress. LLMLingua compresses prompts up to 20× by token-level importance scoring. Quality loss depends on task: tolerated on summarization, extraction, and classification; risky on math, code generation, and complex reasoning. Always A/B against an eval suite. Treat compression ratio as a budget, not a target.

Layer 5 — fine-tune. Fireworks serverless LoRA on Llama replaces a frontier model for one narrow, high-volume task in roughly 30 minutes of training. Worth doing when you have ≥10k labeled or LLM-generated examples, the task is stable, and the frontier bill on that one task justifies the engineering time. Don't fine-tune to save 5% on a low-volume endpoint.

How they fit together

client app
   │
   ▼
┌──────────────┐  cache hit   ┌────────────┐
│ Helicone /   │─────────────▶│ cached     │
│ Cloudflare   │              │ response   │
│ AI Gateway   │              └────────────┘
└──────┬───────┘ cache miss
       │
       ▼
┌──────────────────┐  classify  ┌─────────────────┐
│ Manifest router  │───────────▶│ cheap model     │
│ (semantic)       │            │ (Llama / Haiku) │
└──────┬───────────┘            └─────────────────┘
       │ low confidence / escalate
       ▼
┌─────────────────┐  optional   ┌──────────────┐
│ LiteLLM /       │────────────▶│ LLMLingua    │
│ OpenRouter /    │  compress   │ pre-compress │
│ Portkey gateway │             └──────┬───────┘
└──────┬──────────┘                    │
       │                               │
       ▼                               ▼
  frontier model (Opus / GPT-4 / Gemini Ultra)
       │
       ▼
LiteLLM cost ledger + TokenCost reconciliation

Tradeoffs (the honest part)

• Cache hit rate vs freshness. 70% hit on support chat is a fortune; 70% on a stock-price assistant is a customer-trust disaster. Set TTL per route, not globally.
• Router latency overhead. A semantic router adds 50–200 ms (embedding + classification). Invisible on chat, visible on streaming voice. Measure end-to-end p95 before and after.
• Compression quality loss. LLMLingua at 5× is mostly free on summarization; at 20× it starts dropping facts on extraction. Pair every rollout with a held-out eval set.
• Cheap-model misrouting. Routing a math problem to Haiku because the router thought it was 'simple Q&A' is a silent regression that surfaces a week later. Log routing decisions with the trace, review bottom-decile confidence weekly.
• Fine-tune lock-in. A LoRA against Llama 3.1 70B is yours to host anywhere; a fine-tune on a proprietary model isn't. Pick the base deliberately.
• Observability isn't free either. Break-even is usually around 1M calls/month — below that, the free tiers are fine.

Common pitfalls

• Optimizing before measuring. Engineers fine-tune 'because GPT-4 is expensive' without ever instrumenting the top-spend endpoint. Eight times out of ten the bill is one feature, not the whole product.
• Caching private content by mistake. A semantic cache keyed on prompt text alone will serve user A's medical chat to user B if queries embed similarly. Always scope cache key by user / tenant / auth-context.
• Routing the wrong task to the wrong model. Tool use and JSON-mode structured output break on many cheap models. Run the routing classifier against real production traffic distribution before rollout.
• Confusing 'tokens saved' with 'dollars saved'. Input tokens are 3–5× cheaper than output tokens at most providers. Track dollars, not tokens.
• Treating cost optimization as one-shot. Provider prices change monthly; new cheaper models ship quarterly. Re-run the routing benchmark every quarter.
• Skipping the eval gate. Every layer 3–5 change must ship behind an eval suite. 'Save 20% on tokens, lose 4% on accuracy' is rarely the trade you wanted.

Pair with these packs

This pack is the cost layer. Pair with Agent Observability + Tracing for debugging — you cannot optimize spend you can't attribute to a span. Pair with LLM Eval & Guardrails so every routing and compression change ships behind a quality gate. Pair with Vector DB + RAG if retrieval context is what is making prompts long; sometimes the cheapest token is the one you don't send.