Memory is the part of an agent people only notice when it breaks. A model can be excellent on a single turn and still feel like a stranger on the second one, and nothing in a single-turn benchmark will warn you.
Writing things down is trivial. The hard part happens at retrieval, when the system has to decide what counts as relevant to a request the user has not fully stated. An evaluation that checks whether explicitly stated facts come back out is measuring a database, not a memory.
Three properties keep coming up when we throw out a memory eval:
- It has to run over conversations long enough that the interesting failures have room to happen, not over synthetic pairs of turns.
- It has to distinguish forgetting from declining to mention, which look identical from the outside and have opposite fixes.
- It has to be robust to the agent getting the right answer for the wrong reason, which a scored-output-only harness will happily reward.
The direction we keep returning to is evaluating the retrieval decision directly rather than the text that follows it — grading what the agent chose to bring into context, separately from what it then said. It makes the eval harder to write and much easier to read.