Skip to content

convomem-arena

convomem-arena turns Salesforce ConvoMem (paper, reference code) into tasks for evaluation and reinforcement learning. Each task asks a question about a user's earlier conversations with an assistant: a fact the user stated, something the assistant recommended, a detail that later changed, a preference to respect, or something that was never said at all. The answer is buried in a history that also holds many unrelated conversations from the same person.

The same tasks reach the agent in three ways, so you can measure long-context recall and agentic memory use on identical data:

memory_mode What the agent gets Tests
context (default) The full history in the prompt, using ConvoMem's official answer prompt Long-context recall (the paper's baseline)
tools Only the question; the history sits behind a per-rollout MCP memory server Agentic retrieval: deciding what to look up, and when
files Only the question; the history is written to memory/conversation_NNN.txt Memory as a filesystem, with general-purpose tools

Answers are graded by ConvoMem's official RIGHT/WRONG judges, one per category, and in tools mode the environment also measures whether the agent actually retrieved the evidence.

Get started

  • Evaluate a model


    Install, check that tasks load, and run your first episodes against a hosted or fully local model.

    Model quick start

  • Evaluate a personal assistant


    Let an assistant live through a persona's history, build its own memory, then answer from it. Vellum is the worked example.

    Assistant quick start

Explore