Table of Contents
TL;DR
Most companies think their AI bill is high because models are expensive. Usually, that’s not it. The real driver is how those models get used, endless retries, repeated system prompts, and top-tier models doing work a cheaper one could handle just fine.
- Cache prompts instead of resending the same context every time
- Break big prompts into smaller tasks and route simple ones to cheaper models
- Match the model to the task, save your best model for real reasoning
- Cuts cost by up to 90% on routine work, no quality lost
Executive summary
Many enterprises are wasting significant budget on AI tokens – infinite loops, repetitive system prompts, and overuse of frontier models for simple tasks. This blog by Wishtree Technologies outlines practical steps to reduce AI spend without sacrificing quality – prompt caching, micro‑service decomposition, and task‑appropriate model selection. It is written for business, engineering, and finance leaders who need a clear way to improve AI ROI.
Introduction
Across the industry, AI budgets are growing fast. Finance teams are approving significant spend on:
- LLM tokens and API calls.
- AI platforms and orchestration tools.
- Pilot projects and proof‑of‑concept work.
Yet we can see a common pattern emerging.
- Engineers and employees blindly overuse AI tokens.
- Simple tasks (data formatting, basic summarisation) are handled by expensive frontier models.
- Infinite loops and repetitive system prompts burn tokens without adding value.
The result – higher AI costs for lower‑quality outcomes.
Our blog today outlines how to re‑engineer AI infrastructure for financial efficiency.
The core bottleneck: Tokenmaxxing
“Tokenmaxxing” – the blind overuse of AI tokens – shows up in several ways.
- Infinite loops
- Agents that retry failed operations indefinitely, burning tokens on each attempt.
- Workflows that call the same tool multiple times due to poor error handling.
- Repetitive system prompts
- Long, verbose system prompts that are sent with every request, even when the context has not changed.
- Duplicate instructions and examples that increase token count without improving quality.
- Overuse of frontier models
- Using the most expensive model for every task, including simple data formatting, basic classification, or templated responses.
- No differentiation between high‑value reasoning tasks and low‑value routine tasks.
The financial impact is significant.
- Excess token consumption
- Over‑provisioned models
- Extended pilot cycles that delay ROI
Three practical fixes
1. Deterministic AST extraction (local & free)
What it is
Prompt caching allows you to cache system prompts and context so they are not sent with every request.
How to implement (Anthropic or OpenAI)
- Anthropic:
- Use the cached_prompt or equivalent feature to cache system prompts and long context.
- Send only the variable parts (for example, user query) in each request.
- OpenAI:
- Use the extra_headers or equivalent feature to enable prompt caching where available.
- Structure prompts so that static context is cached and only dynamic content is sent.
Financial impact
- Reduces token count per request by 30-70% for workflows with long system prompts.
- Lowers latency by reducing the amount of data sent per request.
| Scenario | Tokens per request | Cost impact |
| Before (no caching) | 2,000 tokens (1,500 static + 500 variable) | Baseline |
| After (with caching) | 500 tokens (only variable content sent) | ~75% reduction |
2. Decompose workflows into micro-services and route by task complexity
What it is
Break large, monolithic prompts into smaller, single-purpose sub-tasks, then route each sub-task to the least expensive model capable of completing it.
How to implement
- Split monolithic prompts into distinct steps (e.g., classification, retrieval, reasoning, data formatting, safety checks).
- Assign low-cost or rule-based models to simple tasks (formatting, simple extraction) and reserve expensive frontier models strictly for multi-step reasoning.
- Execute independent sub-tasks in parallel rather than running a single massive sequential loop.
Financial impact
- Lowers average cost per workflow execution by 50-80%.
- Prevents spending frontier-model prices on basic data formatting or template assembly.
Before: A single giant prompt (3,000 tokens) sent entirely to a frontier model ($0.08 per run).
After:
- Retrieval & Guardrails: 800 tokens via a small, cheap model ($0.002)
- Core Reasoning: 1,500 tokens via a frontier model ($0.04)
- JSON Formatting: 200 tokens via a micro model or deterministic template ($0.0005)
Net result: Equivalent output quality at less than half the execution cost.
3. Establish defensive orchestration and retry guardrails
What it is
Implement hard execution limits, exponential backoff, and fallback handling to prevent broken agent loops or failing API calls from silently burning through your budget.
How to implement
- Enforce maximum loop counts (e.g., max 3 retry attempts) and total token caps per user session or workflow.
- If a frontier model fails or returns a bad tool call, fall back to a structured validation script or a smaller, specialized parser rather than blindly retrying the original prompt.
- Set automated alerts and circuit breakers at the API orchestrator level to halt run-away agent loops before they drain your daily balance.
Financial impact
- Eliminates runaway budget spikes caused by bug-driven infinite loops or infinite agent retries.
- Protects engineering and finance teams from unexpected off-hours API billing surges.
Stop using a multi‑billion dollar frontier model to do basic data formatting
If a task can be done with a template, a rule, or a small model, do not use a frontier model.
Examples
- Data formatting (for example, converting JSON to a table)
- Basic classification (for example, spam vs. not spam)
- Templated responses (for example, “Your ticket has been received”)
These tasks do not require frontier‑level reasoning. Using a frontier model for them is like using a supercomputer to do a calculator’s job.
What this means for leadership
For leaders, the practical implications are:
- Treat AI spend as an engineering problem.
Invest in prompt caching, micro‑service decomposition, and task‑appropriate model selection. - Measure cost per task.
Track metrics such as cost per task, token count per request, and model mix. - Encourage frugal AI design.
Reward teams that reduce cost without sacrificing quality.
Wishtree Technologies helps enterprises design and implement AI workflows across their existing cloud, data, and application environments. We work with teams to re‑engineer AI infrastructure for financial efficiency, reducing token waste and improving ROI.
Contact us today to get started.
FAQs
Why is our AI budget higher than expected?
A major driver is often tokenmaxxing – infinite loops, repetitive system prompts, and overuse of frontier models for simple tasks.
What should leaders prioritize – model cost or infrastructure optimization?
Prioritize infrastructure optimization (prompt caching, micro‑services, task‑appropriate models) for high‑cost workflows. A cheaper model with poor infrastructure often costs more in the long run.
How do we implement prompt caching?
Use native caching features on Anthropic or OpenAI to cache system prompts and long context. Send only variable parts in each request.
What if our workflows are too complex to decompose?
Start with one high‑cost workflow and decompose it into sub‑tasks. Use that as a pattern for subsequent workflows.
Who should own this work?
Typically, a cross‑functional team, i.e platform engineering, data engineering, finance, and the business owner of the workflow. Risk or compliance may need to be involved for higher‑impact use cases.
Is this only relevant for large enterprises?
No. Any organization that wants AI to work reliably at scale needs to solve the financial efficiency problem. The complexity will vary, but the principle is the same.
