Reference
Configuration¶
All fields are CLI flags (snake_case → kebab-case) and can also be set in TOML under [env.taskset], using the snake_case field names.
Taskset (--env.taskset.*): what gets loaded¶
| Flag | Type | Default | Description |
|---|---|---|---|
categories |
list | all six | Subset of user_facts assistant_facts changing_facts abstention preferences implicit_connections |
evidence-counts |
list[int] | None | all | Keep only items with these evidence counts |
context-sizes |
list[int] | [1, 10, 30] |
Conversations per history; one task per size per item. The paper uses 1 2 3 4 5 6 10 20 30 50 70 100 150 200 300 |
seed |
int | 0 |
Seeds item order and every mix |
repo-id |
str | Salesforce/ConvoMem |
Dataset repo |
revision |
str | e3e9b39… |
Dataset revision (pinned for reproducibility) |
Task (--env.taskset.task.*): how each task runs and is scored¶
| Flag | Type | Default | Description |
|---|---|---|---|
memory-mode |
context | tools | files |
context |
How the history reaches the agent |
max-turns |
int ≥ 1 | 40 |
Model calls per rollout before turn_limit stops it |
fact-scoring |
judge | exact_match |
judge |
How the three fact categories are rewarded: the official judge, or the deterministic exact match with no judge call |
judge.model |
str | verifiers default | Judge model |
judge.base-url |
str | Prime Inference | OpenAI-compatible judge endpoint |
judge.api-key-var |
str | PRIME_API_KEY |
Name of the env var that holds the judge key |
tools.runtime.type |
str | subprocess |
Where the memory MCP server runs (tools mode) |
Common run flags (from verifiers)¶
| Flag | Meaning |
|---|---|
-n / -r |
Number of tasks / rollouts per task |
-m |
Agent model |
--client.base-url, --client.api-key-var |
Agent model endpoint |
--env.agent.harness.id |
null, bash, claude_code, codex, … |
--env.agent.runtime.type |
subprocess, docker, prime (default) |
--select.include.idx 0:100, -s (shuffle), --select.limit |
Task selection |
-o |
Output directory |
Example TOML (convomem.toml, used as uv run vf-run convomem-arena @ convomem.toml -m <model>):
[env.taskset]
categories = ["user_facts", "changing_facts", "abstention"]
context_sizes = [10, 50]
seed = 0
[env.taskset.task]
memory_mode = "tools"
max_turns = 30
[env.agent.harness]
id = "null"
[env.agent.runtime]
type = "subprocess"
vf-validate nests the same settings under [taskset] / [taskset.task] (its flags are --taskset.*) and takes the runtime as --runtime.type.