Get started
Quick start: evaluate a personal assistant¶
The model quick start hands the history to the model. A personal assistant is never handed its history: it hears each conversation once, decides what to remember, files it away, and has to find it again later. To test that, the assistant has to live through the conversations first.
convomem_arena.vellum_eval does this for Vellum, an open-source personal assistant with its own memory system. The same four stages apply to any assistant with a memory you can fill and query.
How it works¶
- Plan. Pick N questions per category for one persona, and merge their evidence conversations (plus filler from the same persona) into one time-ordered history, one day apart and ending yesterday. One memory then answers every question.
- Build. Import the history into a clean Vellum instance, let Vellum's own memory retrospective review each conversation and call
remember, force a consolidation pass that files the facts into concept pages, and save the result as avellum backupsnapshot. This is the slow part, so it is cached. - Ask. With memory writing switched off, ask each question in a new conversation. The reply is read from the turn's MLflow trace, and the injected memory from Vellum's memory-v3 pool log.
- Score. Grade with ConvoMem's official judge prompts (any OpenAI-compatible endpoint; default local Ollama), plus two rough diagnostics for fact categories: is the answer anywhere in memory (capture), and was it in what got injected (retrieval).
Run it¶
python -m convomem_arena.vellum_eval.run \
--source /path/to/vellum-assistant \
--instance vellum-rsi-eval --persona Small_Business_Owner \
--per-category 2 --history-size 16
--source is a vellum-assistant checkout (with cli/ and assistant/), and --instance a hatched instance. To run the stages separately, use --stages build, then --stages ask,score --run-dir outputs/vellum-eval/<run-id>. Each run writes plan.json, build.json, answers.jsonl, scores.jsonl and a report.md with accuracy per category.
Before you start¶
Use a dedicated instance
The build restores snapshots over the instance's workspace. Never point it at an assistant you use day to day.
- A freshly hatched instance with
mlflowTracing.enabled, connections and profiles for every call site, and a Claude Code token stored if call sites use theclaude-codeprovider. A snapshot restore wipes that token's value, so set it again after each restore. - A clean shell environment. The harness strips
VELLUM_WORKSPACE_DIRand*_IPC_SOCKET_DIRinherited from your shell; a profile that exports another instance's values would otherwise make daemons bind that instance's socket. - One persona per instance for now.
vellum backupbundles exclude the Qdrant vector index, and a restore keeps the live one, so restoring persona B over persona A leaves A's vectors behind. Restoring a persona's own snapshot is fine. - Time. Retrospectives take about 3–5 minutes per conversation on a local 30B model, so a 17-conversation history takes about an hour to build.
Reading the result¶
A low score has four possible causes, and the diagnostics help separate them: the fact was never captured, consolidation dropped it, retrieval didn't inject it, or the model had it and still answered wrong. Below ten concept pages, Vellum's memory v3 runs a lean profile that skips its selector, so a small history never tests retrieval; the report says so when that happens.
Only want to test Vellum's model?
To run the ordinary benchmark with a Vellum assistant answering from a history in its prompt, see Use a Vellum assistant as the model.