Agent-memory benchmark results are not reproducible as currently published. LongMemEval's harness prescribes no run count and ships no variance tooling, so one number is the default output. Mem0's paper is the only memory-layer paper reporting otherwise: ten independent runs, ±1 standard deviation. Microsoft's STATE-Bench repeats each task five times. Every comparison page reprinting those numbers publishes a point estimate.
Ross Jones — Founder, The Hopium Lab. Last modified 22 July 2026.
An honest negative result, published because negative results are falsifiable and marketing is not.
What do LongMemEval and LoCoMo actually test?
Two curated conversational question sets, both smaller than the discourse around them implies. LongMemEval (Wu et al., ICLR 2025) is 500 manually written questions across five core abilities, thirty of them modified to test abstention against a false premise; the paper reports a 30% accuracy drop for commercial assistants across sustained interaction. LoCoMo (Maharana et al., ACL 2024) is 1,540 questions over machine-human generated dialogues — the ACL version describes up to 32 sessions, around 600 turns and roughly 16K tokens per conversation; the arXiv v1 abstract gives different figures, so state which you ran.
Scale is the uncomfortable part. The official repository puts LongMemEval-S at roughly 115k tokens of chat history, about 40 sessions — comfortably inside a 200K context window. A long-term memory benchmark that fits in context measures how tidily a system organises text it could have pasted wholesale.
The same repository settles the reproducibility question. Evaluation is one invocation — python3 evaluate_qa.py gpt-4o your_hypothesis_file — emitting one label per question. No seeds. No repetition. No interval computation. Single-run reporting is the harness's default path, not anybody's negligence.
Who has already made this argument?
Four credible parties, recently, and pretending otherwise is the fastest way to lose an expert reader. Kapoor et al. named the general failure in AI Agents That Matter two years ago: no standardisation in evaluation practice, and a pervasive lack of reproducibility. Penfield Labs audited LoCoMo's answer key on 9 April 2026 and found 99 score-corrupting errors in 1,540 questions — 6.4%, implying a ceiling near 93.6%. Hindsight published an agent-memory benchmark manifesto on 23 March 2026. Dell Zhang's Benchmark Theatre (20 May 2026) computed Wilson-score intervals showing 56% of adjacent-pair comparisons statistically indistinguishable, and recommended ten runs with standard deviations as a floor.
Credit for the general conclusion goes to Kapoor, Penfield, Hindsight and Zhang, not to this post. Two things here are new: a traceable citation error still propagating, and a distinction nobody has drawn about what the field's best variance reporting actually bounds.
Which variance does ±0.15 actually measure?
Judge stochasticity, on the most plausible reading of the paper's own wording. Mem0's paper states it ran ten independent runs and reports mean with ±1 standard deviation, "due to the stochastic nature of J evaluations" — J being the LLM judge. The LoCoMo results: Mem0 66.88% ± 0.15, Mem0-graph 68.44% ± 0.17, Zep 65.99% ± 0.16, LangMem 58.10% ± 0.21, full-context 72.90% ± 0.19.
Credit where it is owed: Mem0 is the only vendor in the category reporting variance at all, and did it before anybody asked.
Now the reading. Ten re-scorings bound how far the judge wanders across a fixed set of stored answers. They say nothing about how far the score moves when the store is built again from nothing — extraction run again, graph written again, index rebuilt.
Construction variance is the spread in benchmark score produced by re-running the full memory pipeline — ingestion, extraction, indexing and retrieval — from scratch on every run, as distinct from re-scoring a fixed set of stored answers.
Standard deviations of ±0.15 to ±0.21 across 1,540 questions are extraordinarily tight. Either the pipeline is remarkably stable under reconstruction, which would be the most interesting finding in the category, or reconstruction did not happen on those ten runs. Mem0's authors are the only people who can settle it, and I would genuinely like the answer.
A standard deviation over ten re-scorings bounds the judge. It does not bound the pipeline that produced the answers being scored. Those are two experiments, and the field has published one.
Can you trace a quoted number back to its table?
Frequently not, and one specific failure is still propagating. The most-repeated LongMemEval figure on the commercial "X vs Y" pages is "Zep 63.8%, using GPT-4o". Open Zep's paper, Table 2. 63.8% is Zep on gpt-4o-mini. Zep on gpt-4o is 71.2%. The full-context baselines on that table are 55.4% and 60.2%.
Two minutes with the source kills the number, and the pages carrying it cite each other rather than the table.
The figure printed beside 63.8% makes the splice worse. It is 49.0%, attributed to Mem0. 63.8% traces to Zep's own paper; 49.0% is sourced to a separate arXiv paper on the cost-performance of fact-based memory against long-context models (arXiv:2603.04814), which does not present itself as a Zep evaluation. Two numbers, two papers, two judges, two configurations, one row. Further downstream the pair gets re-described as an independent test.
Configuration is underdetermined by both benchmarks, and the field has one public dispute that proves it. On 8 May 2025 Mem0's CTO filed issue #5 on Zep's papers repository, disputing an 84% LoCoMo figure and reporting 58.44% ± 0.20 on replication — a gap of roughly 25 points. Three grounds: an explicitly excluded adversarial category included in scoring, a different system prompt and retrieval template from the other baselines, and one run against ten. Zep subsequently corrected toward 75.14% ± 0.17. Both parties are credible and I adjudicate nothing. The disagreement is the finding: two competent teams ran the same benchmark, and the benchmark did not specify enough to make them agree.
A single-run score on a 500-question benchmark is a measurement. The gap between two single-run scores is a difference between two measurements whose error bars nobody computed.
Does a memory layer beat no memory layer?
Not on accuracy, in the published results, including the vendors' own. Mem0's table puts full-context at 72.90% ± 0.19 against Mem0 at 66.88% and Mem0-graph at 68.44%. The paper's real argument is cost and latency — p95 of 1.44s against 17.1s — which is legitimate, and the comparison pages reprint the accuracy rows as wins regardless.
Corroboration arrives from three directions. None of the three is comparable to the others — different judges, different retrieval configurations, different papers. Read them as three independent findings pointing the same way, not as a table. Letta reports 74.0% on LoCoMo with GPT-4o-mini using file tools and grep, and argues current memory benchmarks may not be very meaningful — a vendor publishing against its own commercial interest. MemDelta (29 June 2026) finds Mem0 statistically tied with cloud RAG, 72.7% against 73.9%, p = 1.0, at fifty times the cost — and finds that swapping only the embedding model shifts accuracy 6.2 points, p = 0.004. Beyond pass@1 (31 March 2026), across 10 models and 23,392 episodes, reports memory scaffolds degrading long-horizon performance across the models tested.
A go/no-go decision resting on any published ranking is resting on nothing.
What does each published result actually report?
Less than the comparison pages imply, and the runs column is where it shows.
| Source | Runs reported | Variance reported | What it supports |
|---|---|---|---|
| LongMemEval, arXiv:2410.10813 (ICLR 2025) | None stated | None | A point estimate on 500 questions |
| Zep, arXiv:2501.13956 (Jan 2025) | None stated | Latency IQR only; no accuracy SD | A point estimate, plus per-category losses the summaries drop |
| Mem0, arXiv:2504.19413 | 10, stated plainly | ±1 SD, scoped to judge stochasticity | The field's tightest reporting, over one axis of two |
| Letta filesystem post | None stated | None | That grep is competitive with specialised memory |
| STATE-Bench (Microsoft, 19 May 2026) | 5 per task, 450 tasks | pass^5; simulator variance ~1% | Consistency as a first-class metric |
| MemDelta, arXiv:2606.29914 | None stated (n=500 per condition) | p-values | That the confounds outweigh the systems |
| The comparison layer, July 2026 | None | None | Nothing |
What did my own runs show?
Nothing publishable, and the run count is the reason. Configuration first, because an inspectable methodology is what this post exists to demand: LongMemEval-S, all 500 questions, no exclusions; answer model gpt-4o-2024-11-20; judge gpt-4o-2024-08-06 through the repository's own harness, shipped prompt unmodified; text-embedding-3-large at 3,072 dimensions; retrieval k=10, no reranker; one identical system prompt across every configuration; full re-ingestion on every run rather than re-scoring stored answers. Three runs.
Three is not a sample. Three is an anecdote with error bars I am not entitled to compute. I cannot report an ordering, because three rebuilds did not produce a stable one, and no delta appears here — a delta from three runs is exactly the figure that turns up in somebody's comparison table in eighteen months, attached to the wrong model.
Which leaves me nothing to sell about skyMem, my own source-available memory layer. It did not win. The harness could not have told me if it had.
Negative results are the cheapest credibility available to a solo operator, because nobody has an incentive to fake one.
How do agent-memory comparisons fail?
In named, repeatable ways. Unnamed failure modes never get fixed.
- Model-column slip. A score quoted against a model it was never measured on. 63.8% is gpt-4o-mini; the pages say gpt-4o.
- Relative-percentage flattening. Zep's headline improvement is relative, not absolute. On gpt-4o the move is 60.2 to 71.2 — eleven percentage points, roughly an 18% relative gain. The summaries print the relative figure as though it were points.
- Cross-paper splice. Numbers from different papers, judges and retrieval configurations printed side by side as a comparison.
- Baseline omission. The no-memory full-context row — the accuracy winner in Mem0's own table — vanishes from the summary.
- Aggregate-only reporting. Zep's Table 3 shows single-session-assistant falling from 94.6% to 80.4% on gpt-4o, and knowledge-update from 76.9% to 74.4% on gpt-4o-mini. No comparison page I found carries any of these rows.
- Category-size blindness. Zep's single-session-preference row moves 20.0% to 56.7% on gpt-4o. On a category of roughly thirty questions that is a swing of about six to seventeen items.
- Judge-variance laundering. A standard deviation over re-scoring, quoted as though it bounded the whole pipeline.
- Benchmark-transfer error. LongMemEval reports over 97% judge agreement with human experts on its own curated set; Penfield's 63% acceptance of intentionally wrong answers is an adversarial probe on a different dataset. Both are true. Citing one against the other commits the citation error this post is about.
Is anybody reporting agent-memory results properly?
Yes, and the standard is already public. Microsoft's STATE-Bench (19 May 2026) runs each of 450 tasks five times and reports pass^5 — the share of tasks succeeding on all five runs — beside the average, with simulator-induced variance measured at about 1%. Its headline: with GPT-5.1 and no memory, only around 30% of travel tasks succeed on every run.
A metric reporting how often a system succeeds every time answers the question a buyer actually has. The gap between average and pass^5 is the product risk.
What would a citable result contain?
Its run count, its pipeline, its baselines and its provenance — in that order.
CITABLE AGENT-MEMORY BENCHMARK RESULT — minimum reporting set
The Hopium Lab · v1.0 · 22 July 2026 · fork it, argue with it
# Construction variance is the spread in benchmark score produced by
# re-running the full memory pipeline — ingestion, extraction, indexing
# and retrieval — from scratch on every run, as distinct from
# re-scoring a fixed set of stored answers.
WHAT THE RUN WAS
[ ] Dataset AND variant: LongMemEval-S | -M | -Oracle | LoCoMo — and
which paper version (arXiv v1 and ACL 2024 report different
session, turn and token counts)
[ ] Questions actually scored, plus every exclusion, by category,
with the reason
[ ] Answer model: exact dated snapshot id
[ ] Judge model: exact dated snapshot id, and the judge prompt verbatim
[ ] Embedding model + dimension (swapping only this moved accuracy
6.2pp, p=0.004 — MemDelta, arXiv:2606.29914)
[ ] Retrieval k, reranker, chunking, context assembly template
[ ] System prompt — the SAME one for every system compared
WHAT WAS REPEATED
[ ] Run count, stated as a number. Below ten, publish it as an anecdote.
[ ] Which stage repeated on each run — say which, explicitly:
JUDGE ONLY -> bounds judge stochasticity
FULL PIPELINE -> bounds construction variance
[ ] Mean, SD, and min/max per configuration. Never mean alone.
[ ] pass^k beside the mean wherever tasks are multi-step
WHAT IT WAS COMPARED TO
[ ] No-memory full-context baseline, same judge, same prompt
[ ] A dumb-retrieval baseline (BM25 or plain vector RAG)
[ ] A filesystem/grep baseline
[ ] Cost and p95 latency next to accuracy, or the accuracy row is
answering a question nobody asked
BEFORE QUOTING SOMEBODY ELSE'S NUMBER
[ ] Open the source table. Confirm the model column.
[ ] Confirm relative % against percentage points.
[ ] Confirm both numbers came from the same paper, judge, prompt and
retrieval config. If not, do not print them side by side.
[ ] Read the per-category rows for regressions the summary dropped.
[ ] Cite the table, not the post that cited the table.
Every line is checkable against a source that already exists. The last block is thirty seconds of work, and thirty seconds would have stopped the most-repeated figure in this category from spreading.
One question sits underneath the whole exercise, and it is the question I put to any system before it ships: can the result be rebuilt without trusting whoever reported it? The replay test applies to the systems being scored as much as to the scores.
Until agent-memory results carry run counts, the honest answer to which memory layer is best is that nobody has run the experiment. Me included.
Ross Jones, Founder, The Hopium Lab. Last modified 22 July 2026. Engineering commentary, not investment or legal advice.