Prompt engineering wins on cost for most production workloads, because inference dominates lifetime spend and prompt-side levers — caching, compression, model routing — cut that bill without any training run. Fine-tuning pays back only in a narrow band: high-volume, stable tasks where shrinking the prompt removes more token cost than the fine-tuned inference premium adds. The decision sharpened when OpenAI is winding down its fine-tuning platform, and existing users can create training jobs only for the coming months — which turns fine-tuning from a default tactic into a vendor-risk calculation. If you are still weighing both routes, the cloud AI cost optimization playbook covers the broader spend levers; this piece isolates the tune-versus-prompt decision.
What Fine-Tuning Actually Costs
Training itself is cheap; the per-call premium that follows is what bites. On the OpenAI platform, GPT-4o mini training costs $3 per million tokens, and inference on the fine-tuned model runs $0.30 per million input tokens and $1.20 per million output tokens — exactly double the base model’s $0.15 input and $0.60 output rates. A realistic starter job is small: 500 examples at roughly 500 tokens each, run for three epochs, bills about 750,000 training tokens, or $2.25. Teams do not fail at fine-tuning because of the $2.25.
The failure mode is the doubled inference rate multiplied by volume. Premium-model tuning is a different budget entirely: current hosted training on o4-mini is metered at $100 per hour, with discounted per-token inference available only if you enable data sharing on the job. That makes mini-class tuning an experiment and frontier-class tuning an infrastructure line item with a procurement conversation attached — and a platform that is closing to new users on top of it.
The Prompt-Side Cost Levers
The prompt route has three levers fine-tuning cannot touch. First, prompt caching: on Anthropic’s API, cache hits are billed at 0.1x the base input price on every current Claude model except Fable 5.1 and Mythos 5.1, so a stable 50,000-token system prompt costs 90% less on every hit after the first write. OpenAI’s cache discount is shallower — cached input runs at roughly half the base rate — but it applies automatically. Second, prompt compression: cutting few-shot examples from eight to two routinely removes 60–70% of input tokens on classification-style tasks with modest quality cost. Third, model routing: sending the easy majority of calls to a small model and only the hard tail to a frontier model is the single largest saver in most real stacks.
Pricing on the prompt side is also more stable than teams fear. Claude Sonnet 5 now carries $2 per million input and $10 per million output as its standard price, after a scheduled increase to $3 and $15 was cancelled. And when your customization lives entirely in prompts, a model version change carries no retraining debt: swap the model identifier and re-run your eval suite. Anthropic’s platform offers no general-purpose tuning option at all, which is why its customization economics concentrate in caching and long context.
When Tuning Beats Prompting
Run the comparison on identical traffic: one million calls per month, each with 500 fresh input tokens and 200 output tokens, against a 2,000-token few-shot prefix.
| Route | Upfront cost | Per-call cost | Monthly cost at 1M calls |
|---|---|---|---|
| Fine-tuned GPT-4o mini, no few-shot | ~$2.25 per starter job | $0.000390 | $390 |
| Base mini + 2,000-token few-shot, uncached | $0 | $0.000495 | $495 |
| Base mini + 2,000-token few-shot, cache hits | ~$0 beyond cache writes | $0.000345 | $345 |
The honest reading surprises most teams: at mini-class pricing, prompt caching alone beats fine-tuning on pure cost. The tuned model sits between the cached and uncached prompt routes. Fine-tuning still wins on cost when three conditions hold: your prefixes vary so much that caching rarely hits; the task is output-heavy and the tuned model emits shorter, more reliable outputs; or tuning lets you drop a model class — a tuned mini replacing an untuned frontier call changes the arithmetic by an order of magnitude, not a percentage. If none of those apply, the tuning premium is a permanent tax.
Decision Checklist Before Retraining
- Baseline first: measure cost per resolved task with the base model and your current prompt for two weeks. Without that number every later comparison is guesswork — wire it into the same dashboards you use for production LLM observability.
- Exhaust caching: move stable instructions and reference material into a cached prefix; on Anthropic that is a tenth of base input on hits, on OpenAI roughly half.
- Compress and route: trim few-shot examples, then send easy calls to a smaller model before touching training.
- Re-check the gap: only if the remaining delta exceeds about 20% of monthly model spend and the task distribution has been stable for months does tuning beat prompting.
- Budget the full lifecycle: training cost, doubled inference, eval re-runs on every base-model update, and a migration plan if the tuning platform shuts its doors.
For EU teams there is one more line in the ledger: data residency. Training jobs export your examples to the provider’s platform, which for regulated workloads means a DPIA conversation that prompt-only pipelines with regional endpoints never trigger. Given the wind-down announcement, treat any new fine-tuning dependency as technical debt with an expiry date, and keep the prompt layer — versioned, cached, routed — as the portable core of your stack.