brainbox was rebuilt in September 2026. The benchmarks from the old deployment are not the fresh runtime, and a tiny synthetic retrieval baseline is not a judged answer-accuracy score. So this page has two clearly separated halves: a frozen synthetic baseline for the fresh rebuild (30 September 2026, local harness), and archived results from the July 2026 Neo4j deployment — root-inspected on 30 September 2026, not rerun. Raw counts, dates and environments throughout; what was not tested is listed next to what was.
Compiled 30 September 2026 · Fresh section: local synthetic baseline, run 2026-09-30 · Historical section: archived July 2026 artifacts from the previous deployment · Independent review and native visual checks pending before publication.
Scope first: this is a retrieval-only benchmark of the fresh rebuild on a frozen synthetic corpus of 30 fictitious memories — real cached bge-m3 vectors, the actual PostgreSQL retrieval verbs, running against a local Docker harness. It is not a generated-answer evaluation, not hosted-service latency, not real user data, and not comparable to the archived judged-answer scores further down this page.
| Query type | Queries | Relevant items | Retrieved@5 | Item recall@5 | Queries with ≥1 hit | MRR |
|---|---|---|---|---|---|---|
| Exact wording | 6 | 6 | 6 | 6/6 | 6/6 | 1.000 |
| Time / “who now, who before” | 6 | 6 | 6 | 6/6 | 6/6 | 0.806 |
| Paraphrase | 12 | 12 | 7 | 7/12 | 7/12 | 0.454 |
| Multi-source (combine several) | 6 | 16 | 7 | 7/16 | 3/6 | 0.417 |
| Overall | 30 | 40 | 26 | 26/40 (65%) | 22/30 | 0.626 |
“Retrieved@5” counts relevant items present in the top 5. Exact and temporal queries did well; paraphrase and multi-source queries are where this baseline is weak — none of the three-item multi-source questions surfaced all three items.
Ten unanswerable questions were asked (five near-topic, five out-of-domain). Seven still returned results — e.g. asking for gym opening hours at a building returns other memories about that building. All seven carried weak/partial flags, but the strict lesson stands: retrieving a related memory about an entity does not establish the missing attribute. Only the three out-of-domain questions came back truly empty. Abstention flags and empty result sets are different measures, and this baseline scores them separately.
Correction replaces stored text (old unique wording stops being the only path; new wording retrieves it). Forgetting hard-deletes rows, re-remembering tombstoned text is rejected, and a stale-snapshot restore does not resurrect corrected-away or forgotten content — while untouched items restore intact, vectors included. Cross-tenant connections are rejected in both directions. 17 of 17 assertions passed.
One observation is deliberately not scored: after a correction, the old-wording query still ranked the item first via vector similarity — semantic drift, reported as an observation rather than a pass/fail.
Recall p50 ≈ 2.0 ms, p95 ≈ 3.1 ms (n=47); writes p50 ≈ 2.7 ms, p95 ≈ 6.9 ms (n=33). These are local cached-vector database executions — embeddings were pre-fetched into a cache and resolved in-process. Hosted latency (network inference plus cloud transport) is unmeasured here and will be slower. Treat these as plumbing numbers, not product numbers.
Frozen thresholds (vector floor 0.68, RRF k=60, 48 vector candidates, top-5) inherited from the legacy calibration — no tuning was performed on this benchmark, and the held-out improvement work is still open: this baseline must not be re-labelled as a tuned score. An independent Python pass re-derived the headline counts from the raw cases and matched. The full stripped summary — method, counts, assertions, environment, provenance hashes and limits — is downloadable: brainbox-synthetic-v1-summary.json.
These results come from the previous brainbox deployment (Neo4j era, July 2026). They are archived artifacts, inspected on 30 September 2026 and not rerun since — they are not claims about the fresh runtime. Every available same-turn artifact is listed, not just the best one: later runs exist beyond the one the old page published, and one early cold run is shown at its unflattering actual value.
Same-turn experiment, as published at the time: 150 gold facts with unique canary tokens,
each given a near-twin distractor; every gold probed keyed (contains the canary)
and semantic (paraphrase); tiers = fresh write, interference after the full
corpus, and a later cold session. 30 negative probes asked for combinations that don’t exist.
| Artifact (UTC) | Recorded commit† | Tier | n | hit@1 | hit@5 | MRR | Median s | p95 s* |
|---|---|---|---|---|---|---|---|---|
| 2026-07-12 21:54 run | f8e32bf | fresh | 75 | 61 | 75 | 0.890 | 1.42 | 2.09 |
| 2026-07-12 21:54 run | f8e32bf | interference | 150 | 58 | 150 | 0.634 | 1.37 | 1.93 |
| 2026-07-13 03:24 run | f8e32bf | fresh | 75 | 61 | 75 | 0.890 | 1.43 | 2.08 |
| 2026-07-13 03:24 run | f8e32bf | interference | 150 | 59 | 150 | 0.633 | 1.45 | 1.98 |
| 2026-07-13 04:17 run | f8e32bf | fresh | 75 | 70 | 75 | 0.967 | 1.36 | 1.96 |
| 2026-07-13 04:17 runpublished then | f8e32bf | interference | 150 | 135 | 150 | 0.950 | 1.34 | 1.76 |
| 2026-07-14 18:41 run | 7b9a003 | fresh | 75 | 68 | 74 | 0.947 | 3.07 | 5.38 |
| 2026-07-14 18:41 run | 7b9a003 | interference | 150 | 132 | 149 | 0.934 | 3.21 | 5.11 |
| 2026-07-15 06:49 run | 34d941c | fresh | 75 | 69 | 75 | 0.960 | 2.61 | 2.99 |
| 2026-07-15 06:49 run | 34d941c | interference | 150 | 134 | 150 | 0.947 | 2.10 | 2.67 |
| 2026-07-13 03:03 cold | — | cold | 150 | 34 | 150 | 0.525 | 1.42 | 2.01 |
| 2026-07-13 05:39 coldpublished then | — | cold | 150 | 135 | 150 | 0.950 | 1.04 | 1.15 |
| 2026-07-14 19:38 cold | — | cold | 150 | 134 | 149 | 0.943 | 2.31 | 3.51 |
| 2026-07-15 07:35 cold | — | cold | 150 | 137 | 150 | 0.957 | 2.16 | 2.79 |
† Recorded commit metadata is not verified as the deployed worker revision. Three early runs share the same recorded commit despite changing results — no causal claim is made from this field. “Cold” artifacts were recorded in separate later sessions against the then-deployed pipeline.
* Latency is recorded wall-clock seconds at run time, through the then-live pipeline. p95 here is the nearest-rank recomputation from the 2026-09-30 inspection; the original harness interpolated percentiles. The two methods are not silently substituted.
Keyed mode (query contains the memory’s unique token) scored 148–150 of 150 hit@1 across all nine artifacts; the 14 July run is the only sub-perfect one (148/150 hit@1, 149/150 hit@5, with two fresh-tier abstentions). Across dates, interference-tier semantic rank-1 moved from 58/150 (12 July) to 135/150 (13 July) and stayed 132–137/150 in the later runs — recorded outcomes only; the artifacts do not establish why.
| Run (UTC) | Queries | Abstain flag set | Returned context |
|---|---|---|---|
| 2026-07-12 21:54 | 30 | 0 | 30 |
| 2026-07-13 03:24 | 30 | 0 | 30 |
| 2026-07-13 04:17 | 30 | 0 | 30 |
| 2026-07-14 18:41 | 30 | 0 | 30 |
| 2026-07-15 06:49 | 30 | 30 | 30 |
The 15 July run sets the abstain flag on all 30 negatives while still returning context in all 30 — abstention flags and empty result sets are different measures, and no archived run shows empty-result abstention on negatives. The old page’s “0 of 30 abstain” disclosure described the July 12–14 runs; it is kept here in its full timeline instead of being quietly updated.
One archived LongMemEval-S result exists from the old deployment: a saved judge artifact
scored by gpt-4o-2024-08-06. 326 of 499 questions correct (65.33%).
One of the 500 dataset questions was never scored. It was not rerun or rejudged on
30 September, and the original online calls were not independently witnessed — this is the
recorded artifact, nothing more.
| Category | Scored | Correct | Rate |
|---|---|---|---|
| single-session-assistant | 56 | 53 | 94.6% |
| single-session-user | 70 | 57 | 81.4% |
| knowledge-update | 77 | 55 | 71.4% |
| temporal-reasoning | 133 | 88 | 66.2% |
| multi-session | 133 | 69 | 51.9% |
| single-session-preference | 30 | 4 | 13.3% |
| Overall (499 of 500 scored) | 499 | 326 | 65.33% |
Preference was, and remains, the weak category — 4 of 30. Category counts are as recorded in the artifact; they are not an exact proportional sample of the suite. This is a judged answer-accuracy score on the old pipeline; it cannot be compared to the fresh baseline’s retrieval counts above, nor to vendor headline numbers without matched dataset, reader, retrieval and judge protocols — so no vendor comparison is shown. The earlier published page’s competitor table was removed on 30 September 2026 for exactly that reason.
Benchmarking is not “complete”, and this page will not pretend otherwise. Open items, in the order the work is planned:
The 30 September baseline is frozen at its original scores. Improvement work runs against a held-out set and will be published only as root-reviewed results — no self-upgraded numbers.
Same-turn write→recall, round-trips and multi-client choreography through actual connected clients — the fresh baseline measures retrieval in one local harness, and the archived runs measured one pipeline end to end; neither is a cross-client measurement.
The archived 65.33% belongs to the old deployment. A fresh run on the rebuilt system needs approval and budget for judge calls; until then the archived number stays archived.
Everything current on this page that smells like latency is local cached-vector execution. Hosted end-to-end latency and cost per call are unmeasured.
How retrieval behaves when memories age a month — untested in both eras, and on the list.
brainbox is a personal preview today; invitations open gradually. Register interest if you want to be part of the next stage — and hold every future number on this page to the standard this one sets: raw counts, dates, environment, and limits in the open.
Register interest What is brainbox?