A good result can still teach a bad rule.
ServeLearnBench gives agents chronological task streams whose hidden policies change between windows. Serving tasks return a score. Held-out test tasks return no feedback. The benchmark then measures whether the agent learned the current rule without losing behavior that still works.[1]
That split matters outside a benchmark. A support agent may receive a high score because it guessed the right exception, not because it found the durable policy. If that episode becomes trusted memory at once, one lucky answer can steer later work.
Do not connect the score line straight to the memory write.
Learning and capability are different tests.
The paper reports a large gap between following a disclosed policy and inferring one from feedback. Across the benchmark, models averaged 95.4 on hidden tasks when the policy was disclosed. The blind condition averaged 14.1. On the hardest retail tier, the best learning pair reached 59.6.[2]
The paper also found a regression on already-specified retail tasks. Every tested learning method scored below the blind baseline's 98.9 percent, with results between 89.5 and 95.5 percent. New memory did not only help with unknown rules. It also disturbed known work.[1]
Use two test windows.
Keep one window for tasks that need the new rule. Keep another for tasks that worked before the memory update. The first window asks whether the agent adapted. The second asks what the adaptation broke.
Add a cost limit because the benchmark found that learning can require far more tokens and money than following a disclosed rule. Record the memory version and keep a rollback target. A passing average without these records is hard to inspect and harder to reverse.
Promote memory like a release.
Google's SRE workbook defines a canary as a partial, time-limited deployment evaluated against a control before the rollout continues. The same release shape fits agent memory. The candidate memory runs on a bounded slice. The prior memory remains the control. A named rule decides whether the candidate advances or returns to quarantine.[3]
CANDIDATE=[MEMORY VERSION OR SKILL DIGEST] NEW_POLICY_WINDOW=[PASS / FAIL] KNOWN_TASK_WINDOW=[PASS / FAIL] COST_LIMIT=[PASS / FAIL] ROLLBACK_TARGET=[VERSION] DECISION=[HOLD / SHADOW / REVIEW FOR PROMOTION]
The open-source benchmark can exercise this idea through an OpenAI-compatible endpoint. Its command-line runner writes task rewards and trajectories to JSONL. That makes the project inspectable, but it does not make the published scores universal. They belong to the benchmark's scenarios, methods, models, and September 25 result snapshot.[4]