Skip to content

Get started

Quick start: evaluate a model

You need Python ≥ 3.11 and uv.

1. Install

git clone https://github.com/rahmansunny071/convomem-arena.git
cd convomem-arena
uv venv --python 3.12
uv pip install -e ".[dev]"

2. Check that tasks load

This builds six tasks, one per category, and sets each one up. It makes no model calls and needs no API key.

uv run vf-validate convomem-arena -n 6 --only-setup --runtime.type subprocess

-n 6 already covers all six categories, because tasks are interleaved across categories (why).

3. Run your first episodes

The run needs two models: the agent that answers, and the judge that grades the answer.

By default both use Prime Inference, with the key in PRIME_API_KEY.

export PRIME_API_KEY=...
uv run vf-run convomem-arena -n 6 -r 1 -m <model> \
  --env.agent.harness.id null --env.agent.runtime.type subprocess

Run fully local with Ollama

No API key needed: qwen3:30b-a3b serves as both the agent and the judge (verified end to end: 6/6, mean reward 1.000).

# Ollama must be serving the model at 127.0.0.1:11434 first:
#   ollama pull qwen3:30b-a3b && ollama run qwen3:30b-a3b "reply with one word: ok"

export OLLAMA_FAKE_KEY=unused   # ignored by Ollama; vf-run just needs the variable set

uv run vf-run convomem-arena -n 6 -r 1 -m qwen3:30b-a3b \
  --client.base-url http://127.0.0.1:11434/v1 --client.api-key-var OLLAMA_FAKE_KEY \
  --env.agent.harness.id null --env.agent.runtime.type subprocess \
  --env.taskset.context-sizes 10 \
  --env.taskset.task.judge.base-url http://127.0.0.1:11434/v1 \
  --env.taskset.task.judge.api-key-var OLLAMA_FAKE_KEY \
  --env.taskset.task.judge.model qwen3:30b-a3b

A smoke test, not a measurement

A 30B model judging its own transcript is weaker than a frontier judge on the rubric-graded categories (preferences, implicit_connections). Ten conversations of history is also an easy setting. Treat a perfect score here as "it works", not as a result.

4. Look at the results

vf-run writes one line per episode to traces.jsonl. Reading results shows which fields matter and how to compute accuracy per category.

Next steps

  • Try the other memory modes


    Put the history behind search tools, or in files, instead of in the prompt.

    Memory modes

  • Run real experiments


    Context-length curves, context versus tools, and retrieval versus reasoning failures.

    Running evaluations

  • Check your judge


    Find out whether the judge model grades what you think it grades.

    Judge robustness

  • Evaluate an assistant's memory


    Make a personal assistant live through a persona's history, then ask it.

    Personal assistant quick start