Skip to content

Concepts

verifiers, prime-envs and the Hub

Piece Role In this project
verifiers (import verifiers.v1 as vf) Framework: tasks, tools, judges, rewards, and the runner that puts an agent on a task and records a trace Every class here subclasses a vf.* base
prime-envs Prime Intellect's repo of first-party environments; the source of conventions (layout, prompting rules, README style) This package follows those conventions and can drop in as environments/<group>/convomem_arena
Environments Hub Registry where environment packages are published and installed by ID (owner/name@version) Packaged as convomem-arena, the ID it would be installed by

verifiers v0 vs. v1. Older tutorials and many Hub environments (listed as Legacy) use the v0 API: load_environment(), vf.SingleTurnEnv, vf.ToolEnv, vf-eval. The current release (verifiers>=0.4.1) ships only v1: Taskset / Task / TaskData, @vf.reward, vf.Toolset, and the vf-validate / vf-run CLIs. This package is v1 only.

verifiers v1 concepts, mapped to this code

verifiers concept What it is Here
vf.TaskData One immutable row: the prompt plus hidden fields for scoring ConvoMemData in models.py
vf.Task Behavior for one row: setup, stop conditions, toolsets, rewards, metrics ConvoMemTask in taskset.py
vf.Taskset Loader that yields tasks from load() ConvoMemTaskset in taskset.py
vf.TasksetConfig Load-time knobs → --env.taskset.* flags ConvoMemConfig
vf.TaskConfig Per-task runtime and scoring knobs → --env.taskset.task.* flags ConvoMemTaskConfig
vf.State Mutable per-rollout state shared between tool servers and scoring MemoryState in models.py
vf.Toolset + @vf.tool Custom tools served over MCP MemoryToolset in servers/memory.py
vf.Judge LLM grader: a prompt template plus a parse function ConvoMemJudge and its subclasses in judges.py
@vf.reward(weight=…) Score that adds score × weight to trace.reward correct
@vf.metric Logged measurement that never affects the reward memory (tool calls, evidence recall)
@vf.stop Predicate that ends the rollout early turn_limit
Harness Agent program that talks to the model (null, bash, claude_code, …), chosen at run time Not fixed by this taskset
Runtime Where the harness runs (subprocess, docker, prime), chosen at run time Not fixed by this taskset
Trace The full record of a rollout, saved to traces.jsonl See Reading results

The taskset ID maps to the module: convomem-arena → import convomem_arena → the class listed in __all__ (ConvoMemTaskset). That's the only thing convomem_arena/__init__.py does.

Rollout lifecycle

ConvoMemTaskset.load() ──yields──▶ ConvoMemTask(ConvoMemData, ConvoMemTaskConfig)
                                          │
  1. Task.setup(trace, runtime)           │  records trace.info["convomem"];
                                          │  files mode: writes memory/conversation_NNN.txt
  2. Task.toolsets(config)                │  tools mode: launches one MemoryToolset per rollout,
                                          │  which loads the history in setup_task(task_data)
  3. Harness runs the model ◀────────────┘  prompt = TaskData.prompt; MCP tools if any;
       │                                     @vf.stop turn_limit checked before each model call
  4. Scoring
       ├─ @vf.metric memory    → memory_tool_calls, evidence_recall, conversations_retrieved
       └─ @vf.reward correct   → ConvoMem judge for the category → RIGHT = 1.0, WRONG = 0.0
                                          │
  5. Trace written to traces.jsonl ◀──────┘