ProtoQA and HotPotQA training dynamics of Campaign 3, in the format of the Campaign 2 page: our framework followed on the development set across checkpoints and through its training stream beside Multi-RLCR on the same batches. Status on 2 October: ours is complete (272 steps); Multi-RLCR is complete (272 steps). HotPotQA ours is complete (417 steps) and shown beside Campaign 2's Multi-RLCR run; HotPotQA RLVR is still training. The frozen-test evaluation of the finished runs is running; its results will be added here.
The Campaign-3 ProtoQA ours run followed through training, in two views. Development set: the 100 development questions evaluated at every in-training checkpoint (0, 20, …, 260; eight rollouts each), as population trends and question by question for ten sampled questions. Training stream: the 69,632 training rollouts themselves, as weight statistics pooled by fifth of the single pass and group by group at every optimizer step, with the trainer's advantages and the reward breakdown.
Every point and every cell below comes from the in-training evaluations on the fixed 100-question development set: the same questions at checkpoints 0, 20, …, 260, eight rollouts each, rescored with the training reward. This is the paired view of learning.
Each point is one in-training evaluation checkpoint (0, 20, …, 260): the mean over the 8 × 100 evaluation rollouts of that checkpoint, not a running average over training. The evaluator pads the 100 × 8 rows to 1,024 by generating 28 questions twice; each question keeps its first 8 rollouts here, while W&B's eval reward averages the padded 1,024.
Sampled with a fixed seed to spread over cluster counts. Move the pointer over a bar to see the cluster's aliases, its crowd probability, its base-model prior and the forecast mass placed on it. The rollout dropdown in each question's header switches all 14 checkpoint cells of that question between the average over the eight rollouts and one chosen rollout.
Everything below comes from the 69,632 training rollouts themselves, 272 optimizer steps × 32 questions × 8 rollouts, each question generated once in the single pass. First pooled by fifth of the pass, then one group at a time.
Each valid rollout's weight vector (its listed weights, up to five, and Other, rounded to .01) is counted across the training rollouts. The policy converges to a handful of templates, so within a group of 8 the weights barely differ; the only shape decision left is which answer occupies which slot. The last two columns compare the advantage variance over all groups with the variance inside groups whose 8 rollouts list exactly the same gold clusters, where coverage is held fixed and only weights, Other and unmatched mass can differ.
Means or medians over the 256 rollouts of each optimizer step, ours and the Campaign-3 Multi-RLCR run (no KL) on the same batches, with a 9-step centered moving average. Unlike the checkpoint charts in Part 3 A, these are the training rollouts themselves, so the questions change at every step.
Pick an optimizer step (every 2nd step and the last one are stored) and one of its 32 questions. The first chart is the crowd distribution; then, row by row, our rollout k beside Multi-RLCR's rollout k on the same question at the same step (the Multi-RLCR run campaign3_protoqa_marlcr_seed42 walks the data in nearly the same order: 70,408 of its 70,656 training rollouts fall in ours' batch for the same step; it was resumed from checkpoint 60 after a GPU fault and a preemption, and a step generated twice shows its later generation). Rollouts are not paired across arms. Multi-RLCR's reward is 0.5 · format + 0.5 · (number of distinct gold clusters hit) − 0.5 · candidate Brier over its five candidates, with the same exact-first WordNet matching; its advantage is reward minus group mean. In parentheses after every advantage is the standardized advantage, (reward − mean) / (std + 10⁻⁴), what TRL's default group scaling would have used. Our rollouts show the trainer's logged reward; the advantage is recomputed as reward minus the group mean (DAPO, no reward scaling), as the trainer computes it. The reward is split into the listed-gold term, the unlisted-gold term (Other × prior + ε), the normalization penalty and the format-fail reward, recomputed with the reward code of the training commit d0f0d36. Each rollout's <hypotheses> block can be expanded; on every 16th step and the last one (marked in the step list) the full chain of thought of every rollout can be expanded too. The full user message of each question can be expanded under its title. These are training questions, each seen once during the single pass, unlike the paired development set above. A rollout is flagged when the training parser accepted it (weights within 0.01 of one) but the current parser, which requires the weights to sum to one, does not, or when the recomputed terms do not add up to the trainer's reward.
The Campaign-3 HotPotQA run of our framework (campaign3_hotpotqa_ours_seed42, 417 optimizer steps, no KL) beside the KL-free Multi-RLCR run of Campaign 2 (campaign2_hotpotqa_marlcr_seed42_nokl_v1), which Campaign 3 keeps as its Multi-RLCR arm. Both runs walk the 25,000 training questions in the same order: the 60-question batch of every stored step is identical in both runs (Multi-RLCR's log lacks steps 193–200 and 287–290). That Multi-RLCR run predates the Campaign-3 fix that synchronizes the two training ranks. Eight rollouts per question. Our reward is log(qgold + ε) with ε = 5 × 10⁻⁴, qgold the weight on exactly-matching listed answers or Other × prior when none matches, minus 0.2 |Σ weights − 1|; a malformed output receives log(prior + ε) − 3. It is reproduced exactly on every valid rollout. Multi-RLCR's reward is 0.5 · format + 0.5 · (any candidate correct) − 0.5 · candidate Brier. Advantages are reward minus the group mean in both runs.
Means over the 480 rollouts of each step, both arms on the same steps; a 9-step centered moving average is drawn over the raw per-step values. Hit rate is the share of valid rollouts with a matching candidate. Multi-RLCR series stop where its log has gaps.
Every 16th step and the last one are stored (27 steps × 60 questions × 2 arms × 8 rollouts, with full chains of thought). Pick a step and a question: the two columns show the eight ours rollouts and the eight Multi-RLCR rollouts of that very question at that very step, with the trainer's advantage (and, in parentheses, the standardized advantage (reward − mean) / (std + 10⁻⁴) that TRL's default group scaling would have used), the reward and its terms, the length of the completion (tokens, reasoning characters, answer characters, and whether it hit the 1,792-token cap), the chain of thought in a scrollable box, and the final answer block. Rollouts are not paired across arms; row k is simply rollout k of each run.