Skip to content

Reference

Design decisions

Decision Reason
verifiers v1, not v0 The current release ships only v1; v0 environments are listed as Legacy on the Hub
Mix on the fly instead of using the 25.7 GB pre_mixed_testcases Same official algorithm, any context size, and a few MB per small run instead of ~40 MB files
Same-persona filler, evidence order preserved Matches the reference generator; ordering defines the answer for changing_facts
Official answer and judge prompts, verbatim Comparable with the paper; the instructions match the reward
Judge failures raise Infrastructure errors must not be scored as wrong answers
Per-task toolset (not shared) Each rollout has a different history; per-rollout servers isolate state
History out of the prompt in tools/files mode The agent must actually use memory; the history lives in hidden TaskData fields
Recall as a metric, not a reward Observing retrieval without incentivizing history dumps
Generic tool descriptions prime-envs guidance: describe what a tool does, not how to use it for this task
files mode prime-envs prefers general harness tools over custom ones; this lets any coding agent use its own
Lazy generator plus round-robin -n stays cheap and still covers every category
No role prompt ("You are a memory assistant…") prime-envs guidance for RL environments