Pick by what changes. If the knowledge changes after training, use RAG. If the behaviour needs to change — format, tone, refusal policy, a smaller model doing the job — fine-tune. If the whole relevant corpus fits in the window, stays small, and the task is not length-sensitive, use long context. Most teams say fine-tune when they mean RAG.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026.
Dated arithmetic, primary sources, and the three conditions that flip the answer. Engineering account, not procurement advice.
What is the actual decision rule?
The change test: name the thing that changes. Knowledge changing after training means retrieval. Behaviour changing means fine-tuning. Neither, with a corpus that fits, means long context.
The retrieval and fine-tuning branches are clean. The long-context branch needs a second condition, because "it fits" is necessary and not sufficient — the task must also survive length, and most reasoning tasks do not. Twenty-four products across fifteen sectors, from pharma cold-chain to aviation certification retrieval, and not one needed a fine-tune to know something. Retrieval knew things. Fine-tuning changes manners, not facts.
The pages ranking on the query explain the three options competently and close on "it depends". A rule that resolves nothing is decoration.
Why do teams fine-tune when they mean RAG?
Because "train it on our data" is the phrase executives use, and nobody in the room translates it. Microsoft researchers measured the gap in arXiv:2312.05934 (December 2023): on a current-events task requiring genuinely new facts, RAG scored 0.875 and 0.876 against fine-tuning's 0.504 and 0.511. More than double, same base models.
Do not overclaim from a single paper. Soudani, Kanoulas and Hasibi (arXiv:2403.01432, SIGIR-AP 2024) ran twelve models and found fine-tuning improves factual performance across all entity popularity bands — retrieval surpasses it by a large margin specifically on the least popular knowledge. The long tail is the entire enterprise use case.
Fine-tuning changes how a model behaves. Retrieval changes what it knows, and does it better on exactly the long-tail facts an enterprise has. Confusing the two is the most expensive mistake in enterprise AI, and it is nearly always made in a meeting rather than in a codebase.
Can you still fine-tune in 2026?
On the two largest first-party APIs, mostly not. OpenAI announced a phased wind-down on 7 May 2026: from that date, organisations that had never fine-tuned lost access to training; from 2 July 2026, organisations without inference on a fine-tuned model in the previous 60 days lost it; from 6 January 2027, remaining active customers can no longer create new fine-tuning jobs. Inference on existing fine-tunes continues until the base models are deprecated. Anthropic's first-party API has never carried a fine-tuning endpoint.
Fine-tuning in mid-2026 means open-weight models on your own or a managed GPU stack, or a Bedrock/Vertex tuning path. Price it as GPU hours, an eval harness, a serving bill and a retraining cadence. Anyone quoting a per-token training rate is pricing a door that is closing.
What does each option actually cost per query?
Roughly 40× for uncached long context, roughly 4× cached, against retrieval — not the 20–24× that gets copied between blog posts without a source or a date. Anthropic's published rates, retrieved 22 July 2026, Claude Opus 4.8 at $5 per million input tokens:
- Long context, uncached, 200,000-token corpus resent every query: 200,000 × $5 / 1M = $1.00 per query.
- Long context, warm cache: a cache read costs 0.1× base input, so the same corpus is $0.10 per query.
- RAG turn — eight chunks of ~500 tokens plus a ~1,000-token system prompt, ≈5,000 input tokens: $0.025 per query.
At 100,000 queries a month on Opus 4.8: ≈$100,000 uncached, ≈$10,000 cached, ≈$2,500 for retrieval plus its infrastructure. On Sonnet 5 at the introductory $2 per million input (in force to 31 August 2026, then $3): ≈$40,000, ≈$4,000, ≈$1,000. Output tokens are excluded throughout, because output is priced identically across all three architectures and its volume is set by the task, not the architecture — comparing total spend is the error in nearly every published comparison.
Three details that move the numbers more than the architecture choice does. A 5-minute cache write costs 1.25× base input and then refreshes free on every use, so continuous traffic pays one write, not one per window. The Batch API takes 50% off input and output at both major providers, and Anthropic documents that it stacks with caching. And the long-context surcharge is gone on Anthropic's current models — the pricing page states that a 900k-token request bills at the same per-token rate as a 9k-token request. Provider-dependent: Google still charges a long-context premium on its Pro tier — Gemini 3.1 Pro goes $2 to $4 input and $12 to $18 output above 200K, and crossing the threshold reprices the whole request, not just the excess. Flash tiers are flat. Check the provider's own page or the multiple is wrong.
Any cost multiple quoted without a caching assumption, a model name and a date attached is a guess wearing a number's clothing.
Where does long context genuinely beat RAG?
On accuracy, when you can afford it. The Google study behind Self-Route (arXiv:2407.16833) is blunt: "when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage." Nine LongBench and ∞Bench datasets, three frontier models of the day.
Self-Route is also the escape hatch. Long context and retrieval produce identical predictions for over 60% of queries, so routing per query — retrieval first, escalate on self-reported failure — cut cost against pure long context by 65% on Gemini-1.5-Pro and 39% on GPT-4O at comparable quality. Route rather than standardise.
Two further flips, both from LaRA (arXiv:2502.09977, 2,326 test cases). Model tier flips it: Mistral-Nemo-12B gained 38.12% from retrieval at 128k while GPT-4o and Claude 3.5 Sonnet consistently preferred the window. Task type flips it: retrieval is strongest at refusing unanswerable questions and weakest at cross-document comparison, where chunking severs the synthesis.
Where does long context collapse?
On length alone, long before the window is full, even when retrieval is perfect. arXiv:2510.05381 varied context from 0 to 30K across five models on maths, QA and coding: performance degraded substantially with length, independent of retrieval quality and of distraction, most of the drop inside the first 7K tokens. Inserting 25,000 whitespace characters — nothing to be distracted by — still drove models to wrong answers after they had correctly extracted every condition. The authors state the consequence plainly: adding documents to a RAG prompt can hurt even when all of them are relevant.
The phenomenon was established earlier by NoLiMa (arXiv:2502.05167), which defines effective length as the longest context where a model holds at least 85% of its base score. GPT-4o fell from 99.3% to 69.7% at 32K — an effective length of 8K against a claimed 128K window. Ten of 22 models dropped below half their baseline at 32K. Honest caveat: 2024-era models on a deliberately adversarial associative benchmark — proof the gap exists, not a forecast for current frontier models.
Advertised context windows are marketing. Effective windows are engineering. Almost nobody measures the second before committing an architecture to it.
Which of the three can cite its sources?
Any architecture where the source document is present at inference time — retrieval or a stuffed window. Not a fine-tune. With citations enabled on document blocks, Anthropic's API returns text blocks carrying a citations array: each citation holds the quoted span plus a typed location — character indices for text, 1-indexed page numbers for PDF. The pointer resolves against a document the caller supplied.
A fine-tuned model can emit a citation-shaped string, and never a verifiable one, because no supplied document exists for the pointer to resolve against. The distinction is mechanical, not stylistic, and the test is the one that decides every AI system: can you rebuild the answer without calling the model? If not, you don't have a system. You have a demo with a try/except around it.
Operational footnote: citations and structured output are incompatible on that API and return a 400 together. More on citation-or-refuse in RAG over certification regulations.
How do the three compare on the things that decide it?
| RAG | Fine-tune | Long context | |
|---|---|---|---|
| What it changes | What the model knows | How the model behaves | What the model sees |
| Freshness | Index time, incremental | Frozen at training | Request time |
| Verifiable citation | Yes — span into a supplied document | No | Yes — same mechanism |
| Input cost per query | $0.025 | Base rate, plus training and serving | $0.10 cached, $1.00 uncached |
| Latency | Adds a retrieval round trip | The only lever that lowers it via model size | Prefill scales with corpus unless cached |
| Available on first-party APIs | Yes | Winding down at OpenAI; never offered by Anthropic | Yes |
Costs are Claude Opus 4.8 input tokens on a 200,000-token corpus, 22 July 2026.
What breaks this in production?
Five named failures, four of them silent.
- The knowledge fine-tune. Training to teach facts. Costs a GPU bill and delivers half the accuracy of retrieval on new knowledge.
- The silent cache miss. Prompts below the minimum cacheable prefix — 512 tokens on Fable 5, 1,024 on Opus 4.8 and Sonnet 5, 4,096 on Opus 4.6 and Haiku 4.5 — do not cache. No error is raised. The bill arrives at 10× the modelled figure.
- The tokenizer drift. Newer Anthropic models produce roughly 30% more tokens for the same text. A corpus measured on Sonnet 4.6 is not that size on Opus 4.8, and every downstream cost estimate is quietly low.
- The promo budget. Sonnet 5's introductory rate holds to 31 August 2026. A twelve-month business case built on it prices September onward 50% below the real rate.
- The orphaned fine-tune. After 6 January 2027 an OpenAI fine-tune keeps serving and stops being retrainable, so staleness becomes terminal rather than recurring.
Two days against a real corpus with real traffic is what The First Look buys.
How do you run the test on your own numbers?
Fork the file, replace the arguments, run it before the meeting rather than after.
# change_test.py — decide, then price it.
# Rates: Anthropic published pricing, retrieved 22 July 2026. Re-check before trusting.
RATES = { # USD per 1M input tokens
"opus-4.8": {"input": 5.00, "cache_read": 0.50},
"sonnet-5": {"input": 2.00, "cache_read": 0.20}, # introductory, to 31 Aug 2026
}
def change_test(knowledge_changes, behaviour_changes,
corpus_tokens, window_tokens, length_sensitive):
"""The change test: name the thing that changes."""
verdict = []
if behaviour_changes:
verdict.append("FINE-TUNE — behaviour only, never to add facts")
if knowledge_changes:
verdict.append("RAG")
if corpus_tokens <= window_tokens and not length_sensitive:
verdict.append("LONG CONTEXT — cache the prefix or pay 10x")
return verdict or ["NEITHER — a plain prompt is the answer"]
def monthly_input_cost(model, corpus_tokens, rag_tokens, queries, cached=True):
"""Input only. Output is priced the same across all three architectures."""
r = RATES[model]
lc_rate = r["cache_read"] if cached else r["input"]
return {"long_context": corpus_tokens * queries * lc_rate / 1e6,
"rag": rag_tokens * queries * r["input"] / 1e6}
if __name__ == "__main__":
print(change_test(True, False, 200_000, 1_000_000, True))
print(monthly_input_cost("opus-4.8", 200_000, 5_000, 100_000))
# ['RAG'] · {'long_context': 10000.0, 'rag': 2500.0}
The three options are not a menu. Each answers a different question, and only one of the questions is yours.