convomem-arena¶
convomem-arena turns Salesforce ConvoMem (paper, reference code) into tasks for evaluation and reinforcement learning. Each task asks a question about a user's earlier conversations with an assistant: a fact the user stated, something the assistant recommended, a detail that later changed, a preference to respect, or something that was never said at all. The answer is buried in a history that also holds many unrelated conversations from the same person.
The same tasks reach the agent in three ways, so you can measure long-context recall and agentic memory use on identical data:
memory_mode |
What the agent gets | Tests |
|---|---|---|
context (default) |
The full history in the prompt, using ConvoMem's official answer prompt | Long-context recall (the paper's baseline) |
tools |
Only the question; the history sits behind a per-rollout MCP memory server | Agentic retrieval: deciding what to look up, and when |
files |
Only the question; the history is written to memory/conversation_NNN.txt |
Memory as a filesystem, with general-purpose tools |
Answers are graded by ConvoMem's official RIGHT/WRONG judges, one per category, and in tools mode the environment also measures whether the agent actually retrieved the evidence.
Get started¶
-
Evaluate a model
Install, check that tasks load, and run your first episodes against a hosted or fully local model.
-
Evaluate a personal assistant
Let an assistant live through a persona's history, build its own memory, then answer from it. Vellum is the worked example.
Explore¶
-
Concepts
How the environment maps onto verifiers, what the dataset holds, how a task is built, and how it is scored.
-
Quality
Audit the data and stress-test the judges before you trust a number.
-
Guides
Recipes for real experiments, reading traces, and running against a Vellum assistant.
-
Reference
Every configuration flag, the package architecture, and known limitations.