Evals
Last updated: 11 October 2026
Memory is easy to demo and hard to get right. These are the public benchmarks we run Recall on, with how we ran them, what they cost, and where Recall is still weak.
Summary
all 400 questions
all 700 questions
all 500 questions
75 held-out questions
same 75 questions
at the default level
How Recall compares
Recall next to other memory systems, and next to the model alone reading the whole history with no memory system at all. Each chart first lists the systems graded the same way as Recall (LongMemEval-S's GPT-4o judge; BEAM's own scoring), whose numbers compare directly, then every other published figure, which used other judges, answer models or question sets. Hover a bar, or open the table under a chart, for its answer model, judge and question count; every figure links to where it was published.
Other systems' scores are as their makers or the papers published them, each with its own answer model, judge and harness; none is a rerun, so a few points either way is noise. Recall answers with Recall Lumen and builds memory with Recall Loom. Recall's figure is graded by LongMemEval's official judge (gpt-4o-2024-08-06 with the benchmark's own prompts), the GPT-4o judge that Honcho and most figures here also report, so it compares more directly than a figure from our own rubric would; the other figures are still self-reported. Newer frontier models reading the whole history score higher still (Gemini 3 Pro reached 92.0% in Plastic Labs' run), at a far higher cost per question.
- 1Mastra, Observational Memory (Feb 2026)
- 2Recall, this page (Oct 2026)
- 3Plastic Labs, Benchmarking Honcho (19 Dec 2025)
- 4Supermemory research
- 5Hindsight benchmarks (Mar 2026)
- 6Mem0 memory-benchmarks (Apr 2026)
- 7Zep research (Sep 2026)
- 8MemOS README (Jul 2026)
- 9LongMemEval paper (Wu et al., ICLR 2025), Figure 3b
Show as a table
| System | Kind | Score | Answer model, judge, questions | Source |
|---|---|---|---|---|
| Mastra (Observational Memory) | LongMemEval-S · same judge | 94.9% | answer model gpt-5-mini, judge gpt-4o; 500 questions; the average of the six question-type scores, not of all questions | 1 |
| Recall | LongMemEval-S · same judge | 91.0% | all 500 questions; answer model Recall Lumen, memory built by Recall Loom; judge gpt-4o-2024-08-06 with LongMemEval's official prompts | 2 |
| Honcho | LongMemEval-S · same judge | 90.4% | answer model Claude Haiku 4.5, ingestion gemini-2.5-flash-lite, judge GPT-4o with the paper's prompt; 500 questions | 3 |
| Supermemory | LongMemEval-S · same judge | 85.2% | answer model gemini-3-pro, judge gpt-4o | 4 |
| Hindsight | LongMemEval-S · other judge | 94.6% | answer model gemini-3.1-pro-preview, judge gemini-2.5-flash-lite; 473 of 500 on Vectorize's own harness | 5 |
| Mem0 (platform) | LongMemEval-S · other judge | 94.4% | answer model and judge gpt-5, top-200 retrieval; as stated in the README (its per-question results file gives 93.4%, 467 of 500) | 6 |
| Zep | LongMemEval-S · other judge | 90.2% | answer model and judge gpt-5.4; 451 of 500 (variant not named) | 7 |
| MemOS | LongMemEval-S · other judge | 89.2% | models not disclosed; self-reported, 500 questions | 8 |
| Claude Haiku 4.5, whole history | model alone, whole history | 62.6% | the model reading every session itself; run by Plastic Labs, judge GPT-4o | 3 |
| GPT-4o, whole history | model alone, whole history | 60.6% | the model reading every session itself, no chain-of-note; judge GPT-4o | 9 |
| Llama 3.1 70B, whole history | model alone, whole history | 33.4% | the model reading every session itself, no chain-of-note; judge GPT-4o | 9 |
Other systems' scores are as their makers or the papers published them, each with its own answer model, judge and harness; none is a rerun, so a few points either way is noise. Recall answers with Recall Lumen and builds memory with Recall Loom. BEAM scores are the mean of ten abilities, 0 to 1.
Show as a table
| System | Kind | Score | Answer model, judge, questions | Source |
|---|---|---|---|---|
| Recall | BEAM 100K · same judge | 0.68 | all 400 questions, 20 conversations of about 125,000 tokens; answer model Recall Lumen, memory built by Recall Loom; BEAM's own evaluation code and judge prompts, graded by gpt-4.1-mini as BEAM's code does; all 400 scored | 1 |
| Honcho | BEAM 100K · same judge | 0.63 | 400 questions; answer model Claude Haiku 4.5; judge per the BEAM paper, model not named | 2 |
| Cognee | BEAM 100K · other judge | 0.79 | 20 questions of one conversation, mean of 4 rounds; answer model and judge gpt-5; marked preliminary | 3 |
| Hindsight | BEAM 100K · other judge | 0.73 | 400 questions; answer model gemini-3.1-pro-preview, judge gemini-2.5-flash-lite (not BEAM's reference judge) | 4 |
| Claude Haiku 4.5, whole history | model alone, whole history | 0.53 | run by Plastic Labs | 2 |
| Qwen2.5-32B, whole history | model alone, whole history | 0.28 | vanilla long context, the paper's own judge | 5 |
| Gemini 2.0 Flash, whole history | model alone, whole history | 0.24 | vanilla long context, the paper's own judge | 5 |
| Llama-4-Maverick, whole history | model alone, whole history | 0.24 | vanilla long context, the paper's own judge | 5 |
| GPT-4.1-nano, whole history | model alone, whole history | 0.24 | vanilla long context, the paper's own judge | 5 |
Other systems' scores are as their makers or the papers published them, each with its own answer model, judge and harness; none is a rerun, so a few points either way is noise. Recall answers with Recall Lumen and builds memory with Recall Loom. Conversations of about 500,000 tokens each.
Show as a table
| System | Kind | Score | Answer model, judge, questions | Source |
|---|---|---|---|---|
| Recall | BEAM 500K · same judge | 0.67 | all 700 questions; answer model Recall Lumen, memory built by Recall Loom; BEAM's own evaluation code and judge prompts, graded by gpt-4.1-mini as BEAM's code does; all 700 scored | 1 |
| Honcho | BEAM 500K · same judge | 0.65 | answer model Claude Haiku 4.5; judge per the BEAM paper, model not named (the blog gives 0.649) | 2 |
| Hindsight | BEAM 500K · other judge | 0.71 | 700 questions; answer model gemini-3.1-pro-preview, judge gemini-2.5-flash-lite (not BEAM's reference judge) | 3 |
| Llama-4-Maverick, whole history | model alone, whole history | 0.28 | vanilla long context, the paper's own judge | 4 |
| Gemini 2.0 Flash, whole history | model alone, whole history | 0.26 | vanilla long context, the paper's own judge | 4 |
| Qwen2.5-32B, whole history | model alone, whole history | 0.20 | vanilla long context, the paper's own judge | 4 |
| GPT-4.1-nano, whole history | model alone, whole history | 0.19 | vanilla long context, the paper's own judge | 4 |
BEAM 100K
BEAM (ICLR 2026) asks 400 questions about 20 long conversations across ten abilities, from recalling a detail to ordering events, resolving contradictions and knowing when to say "I don't know". We ran all 400 questions, 20 conversations of about 125,000 tokens against the hosted service, with Recall Lumen answering, Recall Loom building memory, and one model call per answer. The scoring is BEAM's own evaluation code, ported from the BEAM repository (commit b2da22e) with its judge prompts unchanged: each rubric item of a question is judged on its own and the question scores their mean, and event ordering scores Kendall tau-b, normalised to 0-1, after the judge matches the answer's items to the reference events. The judge is gpt-4.1-mini at temperature 0, the model BEAM's code uses. All 400 questions were scored.
Strongest: following a user's preferences and standing instructions, and saying "I don't know" when the conversation never answered. Weakest: putting events in order, where BEAM's metric credits only the answer items its judge matches to a reference event, and summarizing long stretches of conversation; this is where we are working now.
Against the paper's systems
The BEAM paper reports its own systems at 100K: long-context models, retrieval (RAG) and LIGHT, the paper's memory architecture. Recall's bar is the run above (Recall Lumen answers, Recall Loom builds memory), scored with BEAM's own evaluation code and gpt-4.1-mini, the judge that code uses. The paper's systems answer with other models, and none of their figures is a rerun, so treat the gap as indicative rather than exact.
BEAM 500K
The same benchmark with conversations five times longer, about 500,000 tokens each. We ran all 35 of its conversations, 20 questions each, with the same models (Recall Lumen answers, Recall Loom builds memory) and the same scoring: BEAM's own evaluation code, graded by gpt-4.1-mini. All 700 questions were scored. Saying "I don't know" when the conversation never answered is the strongest ability at this length; ordering events and summarizing are again the weakest. The best system in the BEAM paper scores 0.359 at this length.
LongMemEval-S
LongMemEval-S gives each question its own chat history of about 50 sessions and 100,000 tokens, then asks about it: facts the user or the assistant mentioned, preferences, updates, dates, and things that were never said. We ran all 500 questions, one workspace each, with Recall Lumen answering and Recall Loom building memory. The answers are graded by the benchmark's official judge, gpt-4o-2024-08-06 with the prompts from LongMemEval's own evaluation script, the judge most published LongMemEval-S figures report; that makes this figure more comparable than our earlier sample, which our own rubric graded. Recall answered 455 of 500 correctly (95% interval 88.2-93.2%). Two questions were never asked because a background memory job failed in each workspace (a provider error), and they count as wrong. The weakest types are questions that combine several sessions (102 of 121) and preferences (26 of 30); questions about what the assistant itself said are among the strongest (54 of 56).
LoCoMo
On 75 held-out LoCoMo questions (15 per conversation: single-hop, multi-hop, temporal and open-domain), self-hosted with Recall Loom for every task and judged by gpt-4.1, Recall answered 60.0% correctly under a strict judge and 79.2% under a lenient one, with a median answer time of 1.6 s. Published LoCoMo scores are usually higher because of lenient judges and an easier question mix.
Method
- System under test. The hosted service at api.recallmem.dev, exactly as customers use it: messages go in through the API, background reasoning runs, then each question is asked through
chatat the defaultlowlevel. The memory for these runs was built on 10 October 2026 and every question asked on 11 October. - Models. Answers by Recall Lumen; memory (reasoning over messages, summaries) built by Recall Loom; embeddings by Recall Embed. These are the names of the models the hosted service runs.
- Scoring. BEAM is scored with BEAM's own evaluation code (protocol
beam-ref-b2da22e: ported from the BEAM repository at commit b2da22e, with its judge prompts verbatim). Each rubric item is judged 0, 0.5 or 1 with BEAM's judge prompt and a question scores the mean of its items; event ordering scores Kendall tau-b normalised as (tau + 1) / 2, after the judge aligns the answer's items with the reference events; an ability is the mean of its questions and the overall score the mean of the ten abilities. The judge isopenai/gpt-4.1-miniat temperature 0, the model BEAM's code uses, and every question was scored (400 at 100K, 700 at 500K). LongMemEval-S uses the benchmark's official judge,gpt-4o-2024-08-06with the prompts from itsevaluate_qa.py; an answer counts as correct when the judge says yes. - Isolation. One workspace per BEAM conversation and per LongMemEval-S question, so no memory is shared between them.
- Between runs. The answering rules follow general lessons we wrote down after reading earlier runs' mistakes (name conflicting statements with their dates, say "I don't know" when the record is silent, read the conversation log for overview questions). Nothing is specific to a question or an answer.
- Single runs. Each figure is one run, with no repeats. The 95% intervals are 0.653-0.713 on BEAM 100K, 0.643-0.695 on BEAM 500K and 88.2-93.2% on LongMemEval-S, and wider per ability or type.
What's next
Event ordering and summaries on BEAM, and LongMemEval-S questions that span several sessions, are what we are improving now. The 1M split follows. New results land on this page, including the ones that do not flatter us.