Let's Think Through the Possibilities · Campaign 4 · Qwen3-8B · seed 42

Campaign 4 QA Results

ProtoQA training dynamics of the Campaign-4 parameter search of our framework, in the format of the Campaign 3 page: four cells, prior scale {1, 0.1} × ε {5 × 10⁻⁴, 5 × 10⁻⁵}, no KL regularizer (β = 0), 272 optimizer steps each, every cell shown beside the Campaign-4 Multi-RLCR rerun (campaign4_protoqa_marlcr_seed42) on the same batches. Pick a cell in the selector; every chart and table below follows it. The regularized stage of the search (β = 0.01) was not completed and is not shown. Status on 14 October: the four cells and the Multi-RLCR rerun are complete; the HotPotQA runs of Campaign 4 are still training and will be added here.

Selection

Cell of the parameter search

Part 1

ProtoQA training dynamics

The selected cell followed through training, in two views. Development set: the 100 development questions evaluated at every in-training checkpoint (0, 20, …, 260; eight rollouts each), as population trends, question by question for ten sampled questions, and for any question of your choice. Training stream: the 272 × 32 × 8 training rollouts themselves, as weight statistics pooled by fifth of the single pass and group by group at every optimizer step, with the trainer's advantages and the reward breakdown.

Reading the cells

Reward. The training objective is Σ_c p_c log(q_c + ε) over gold clusters, with q_c the listed mass on cluster c after the written weights are divided by their total S, or Other × prior_c × prior scale when c is unlisted, minus 0.5 · min(|S − 1|, 1). ε and the prior scale are the selected cell's. A malformed output, or one that names the same answer surface twice, receives log ε − 1 (−8.60 nats at ε = 5 × 10⁻⁴, −10.90 at 5 × 10⁻⁵) whatever the question's priors.

Listed term / unlisted term. The same sum split by whether the cluster was named. A missed cluster of mass p costs at most p · |log ε| nats (≈ 7.6 p at ε = 5 × 10⁻⁴, ≈ 9.9 p at 5 × 10⁻⁵), less when Other × prior already covers it, so naming it is worth at most that much; reweighting named clusters moves the reward by at most about a nat.

Chart. Wide grey bars are the crowd distribution. The narrow bar inside each is the forecast on that cluster: solid blue when a listed answer matched it (the normalized weight the policy wrote), dashed orange when none did (Other × prior × prior scale, usually too small to see on a linear axis). Then the Other bucket as written (outline) and the weight on listed answers matching no cluster (hatched). The lower panel shows each cluster's loss −p·log(q+ε) on a log axis in the same colours. "Mean of 8" averages the valid rollouts' forecasts and losses.

Scores under each chart. Mean over the eight rollouts of the logged reward, and of the listed and unlisted terms over valid ones; the table below each strip adds coverage, Other, unmatched mass, top weight, mode correctness and the gold-support KL at floor 1e−4.

crowd plisted weight on a matched clusterbacked off: Other × prior × prior scaleOther as writtenunmatched listed weightlisted termunlisted termformat failures

Part 1 · A

Development set, followed across checkpoints

Every point and every cell below comes from the in-training evaluations on the fixed 100-question development set: the same questions at checkpoints 0, 20, …, 260, eight rollouts each, rescored with the training reward. This is the paired view of learning. A checkpoint missing from a cell's strip was lost to a preemption before its evaluation was logged.

All 100 dev questions

Population trends across checkpoints

Each point is one in-training evaluation checkpoint: the mean over the 8 × 100 evaluation rollouts of that checkpoint, not a running average over training. The evaluator pads the 100 × 8 rows to 1,024 by generating 28 questions twice; each question keeps its first 8 rollouts here, while W&B's eval reward averages the padded 1,024.

Ten sampled questions

Question by question

Sampled with a fixed seed to spread over cluster counts. Move the pointer over a bar to see the cluster's aliases, its crowd probability, its base-model prior and the forecast mass placed on it. The rollout dropdown in each question's header switches all checkpoint cells of that question between the average over the eight rollouts and one chosen rollout.

crowd probability p listed: weight the policy gave to a listed answer matching this cluster (blue, solid) backed off: no listed answer matched, Other × prior (orange, dashed) the Other bucket as written by the policy (outline) weight on listed answers matching no cluster (unmatched)numbers under a bar: crowd p (grey) above the forecast mass on the cluster, coloured by its componentlower panel, log scale: each gold cluster's loss −p·log(q+ε), same colour as its component above; their sum is minus the reward before the penalty

Any of the 100 dev questions

One question, every checkpoint

Pick a development question: its checkpoint cells show the forecast averaged over the eight rollouts or one chosen rollout, and under each cell every rollout's hypotheses block and chain of thought can be expanded, so the generated answer can be read checkpoint by checkpoint.

Part 1 · B

Training stream

Everything below comes from the training rollouts themselves, 272 optimizer steps × 32 questions × 8 rollouts, each question generated once in the single pass. First pooled by fifth of the pass, then one group at a time.

All training rollouts, pooled by fifth of the single pass (optimizer steps 1–54, 55–108, 109–162, 163–216, 217–270)

Weight templates

Each valid rollout's weight vector (its listed weights, up to five, normalized to total one, and Other, rounded to .01) is counted across the training rollouts. The last two columns compare the advantage variance over all groups with the variance inside groups whose 8 rollouts list exactly the same gold clusters, where coverage is held fixed and only weights, Other and unmatched mass can differ.

Training stream: 272 optimizer steps × 32 questions × 8 rollouts

Population trends across training steps, both arms

Means or medians over the 256 rollouts of each optimizer step, the selected cell and the Campaign-4 Multi-RLCR rerun (no KL) on the same batches, with a 9-step centered moving average. Unlike the checkpoint charts in Part 1 A, these are the training rollouts themselves, so the questions change at every step.

Training groups, step by step

Pick an optimizer step (every 2nd step and the last one are stored) and one of its 32 questions. The first chart is the crowd distribution; then, row by row, the cell's rollout k beside Multi-RLCR's rollout k on the same question at the same step (the Multi-RLCR rerun walks the data in nearly the same order: … of its training rollouts fall in the cell's batch for the same step). The cells were preempted and resumed several times; a step generated twice shows its later generation. Rollouts are not paired across arms. Multi-RLCR's reward is 0.5 · format + 0.5 · (number of distinct gold clusters hit) − 0.5 · candidate Brier over its five candidates, with the same exact-first WordNet matching; its advantage is reward minus group mean. In parentheses after every advantage is the standardized advantage, (reward − mean) / (std + 10⁻⁴), what TRL's default group scaling would have used. Our rollouts show the trainer's logged reward; the advantage is recomputed as reward minus the group mean (DAPO, no reward scaling), as the trainer computes it. The reward is split into the listed-gold term, the unlisted-gold term (Other × prior × prior scale + ε), the normalization penalty and the format-fail reward, recomputed with the reward code of commit 57a5a78, which reproduces the trainer's reward on every rollout of every cell. Each rollout's <hypotheses> block can be expanded; on every 16th step and the last one (marked in the step list) the full chain of thought of every rollout can be expanded too. The full user message of each question can be expanded under its title. These are training questions, each seen once during the single pass, unlike the paired development set above.