Frozen-test results as reported in the ICLR 2027 draft (Overleaf, 21 September 2026), followed by a per-problem explorer of every ProtoQA test forecast: our framework and Multi-RLCR, before and after training, against the crowd distribution.
Transcribed from overleaf-iclr2027/main.tex, tables campaign2-validation-partial, campaign2-hotpot-partial and campaign2-proto-partial. Numbers and qualifications are Codex's; nothing here is recomputed.
| Run | Validation metric (%) | Checkpoint 0 | Best (step) |
|---|---|---|---|
| HotPotQA RLVR | Single-answer accuracy | 45.9 (n=896) | 57.0 (160; n=896) |
| HotPotQA Multi-RLCR | Gold in support | 46.9 (n=896) | 61.2 (160; n=896) |
| HotPotQA ours* | Gold in support | 45.4 (n=896) | 66.7 (416; n=896) |
| ProtoQA Multi-RLVR* | Gold mass in support | 17.3 (n=1024) | 36.7 (220; n=1024) |
| ProtoQA Multi-RLCR | Gold mass in support | 25.4 (n=1024) | 43.1 (240; n=1024) |
| ProtoQA ours* | Gold mass in support | 19.0 (n=1024) | 40.3 (160; n=1024) |
*Training KL coefficient .01; otherwise zero. n: logged attempts, including evaluation-batch padding, not independent questions.
| HotPotQA arm | Checkpoint | Format (%) | Accuracy (%) | Coverage (%) | Mode (%) | Brier | Total mass |
|---|---|---|---|---|---|---|---|
| RLVR, no KL | 0 | 69.0[68.3, 69.6] | 29.4[28.6, 30.2] | — | — | — | — |
| RLVR, no KL | 160 | 99.4[99.3, 99.5] | 43.9[42.9, 44.9] | — | — | — | — |
| Our framework, KL .01§ | 0 | 63.3[62.7, 63.9] | — | 31.6[30.8, 32.3] | 27.0[26.3, 27.7] | .874[.857, .890] | 1.00[1.00, 1.00] |
| Our framework, KL .01§ | 417† | 70.1[69.3, 70.8] | — | 36.1[35.3, 36.9] | 25.4[24.6, 26.1] | .826[.814, .838] | 1.00[1.00, 1.00] |
| Multi-RLCR, no KL | 0 | 80.1[79.6, 80.7] | — | 36.9[36.1, 37.8] | 30.6[29.8, 31.4] | .151[.148, .153] | .957[.955, .959] |
| Multi-RLCR, no KL | 380‡ | 99.9[99.9, 100] | — | 54.6[53.6, 55.6] | 44.3[43.3, 45.4] | .101[.0990, .104] | .671[.665, .676] |
Brackets are 95% percentile intervals from 10,000 question-ID bootstrap resamples; framework Brier is truncated. †417 proxies winner 416; ‡380 proxies validation winner 384. Values use three significant digits. Forecast-score valid attempts (0/selected) are 39,895/44,197 for our framework and 50,533/63,000 for Multi-RLCR. Total mass averages stated listed weights and, for our framework, stated Other over valid outputs. §Known bug: this run was trained with the lenient parser (scoring.extract_hypothesis_list) but scored with the strict one (qa_parsing.extract_forecast). At checkpoint 417, 33 / 30 / 25% of completions per stratum fail only the strict duplicate-answer rule, which the training reward never applied (validation validity under the training parser was 99.8 to 100% at every eval step); cap hits are 0.1%. Format, coverage and mode for this arm are therefore not comparable to the other arms until rescored under one parser. Multi-RLCR is unaffected: its training gate already rejected duplicates. Multi-RLCR checkpoint selection was frozen on validation before test evaluation. Bold: best trained-checkpoint point estimate per metric among arms; Brier is not bolded because the framework's truncated Brier and Multi-RLCR's candidate Brier are different scores.
| Multi-RLVR, KL .01 | Multi-RLCR, no KL | Our framework, KL .01 | ||||
|---|---|---|---|---|---|---|
| Metric · checkpoint | 0 | 230† | 0 | 240 | 0 | 260 |
| Format (%) | 99.63[99.34, 99.85] | 99.71[99.37, 99.96] | 98.90[98.27, 99.45] | 98.49[97.72, 99.15] | 69.70[67.89, 71.50] | 99.48[99.19, 99.74] |
| Gold mass (%) | 20.70[18.55, 22.83] | 40.38[38.07, 42.65] | 28.40[26.19, 30.60] | 45.60[43.29, 47.97] | 19.49[17.69, 21.30] | 43.69[41.39, 45.99] |
| Gold mode in support (%) | 25.00[22.09, 27.87] | 50.44[47.02, 53.83] | 34.90[31.77, 38.03] | 55.27[51.84, 58.73] | 23.60[21.13, 26.11] | 53.31[49.93, 56.63] |
| Correct mode (%) | — | — | 19.48[16.94, 22.05] | 30.67[27.58, 33.80] | 13.95[11.97, 16.02] | 29.53[26.58, 32.47] |
| Lifted Brier | — | — | 3.4806‡[3.4415, 3.5188] | 1.0763‡[1.0597, 1.0924] | 1.0774[1.0640, 1.0908] | .9697[.9596, .9798] |
| KL | — | — | 5.34‡[5.15, 5.54] | 4.22‡[4.03, 4.41] | 5.72[5.53, 5.91] | 4.55[4.38, 4.72] |
| χ² | — | — | 2320‡[2200, 2450] | 1640‡[1520, 1760] | 2310[2170, 2440] | 1650[1530, 1770] |
| Training Brier (MARLCR) | — | — | .4590[.4489, .4687] | .2193[.2113, .2277] | — | — |
| Total mass | — | — | 3.8066[3.7884, 3.8246] | 1.2725[1.2609, 1.2850] | 1.0000[1.0000, 1.0000] | 1.0000[1.0000, 1.0000] |
| Method-specific paired subsets | Multi-RLCR, no KL (n=675) | Our framework, KL .01 (n=670) | ||||
|---|---|---|---|---|---|---|
| Metric · checkpoint | 0 | 240 | Δ | 0 | 260 | Δ |
| Lifted Brier | 3.4762*‡[3.4373, 3.5149] | 1.0754*‡[1.0591, 1.0916] | −2.4008*‡[−2.4354, −2.3664] | 1.0777*[1.0647, 1.0905] | .9692*[.9590, .9794] | −.1085*[−.1184, −.0986] |
| KL | 5.33*‡[5.12, 5.52] | 4.22*‡[4.03, 4.40] | −1.11*‡[−1.29, −.936] | 5.72*[5.54, 5.90] | 4.54*[4.37, 4.72] | −1.17*[−1.32, −1.03] |
| χ² | 2310*‡[2190, 2440] | 1630*‡[1520, 1750] | −677*‡[−789, −571] | 2300*[2170, 2430] | 1640*[1520, 1760] | −657*[−752, −562] |
Brackets are 95% percentile intervals from 10,000 problem-ID bootstrap resamples; upper-panel point estimates keep their pooled-attempt definitions (679 problems, four attempts each). Gold mass: gold crowd mass covered by the hypotheses (modulo folding by equivalence class). †230 proxies checkpoint 220, which was lost. Gold mode in support is one when any listed hypothesis matches any maximum-crowd-mass gold cluster. Correct mode compares highest stated confidence with highest gold crowd mass; first-declared wins ties. ‡MARLCR raw confidences need not sum to one. Gold-support KL and χ² floor each gold coordinate at 1e-4 without renormalizing: mass on unmatched hypotheses or Other stays off the gold support as lost mass (definition corrected 21 September; the earlier version renormalized on gold support, which scored a rollout matching nothing as the uniform forecast). Point estimates and interval endpoints use three significant digits, as in the draft. Lower panel: *within each method, retain every problem with at least one scoreable rollout at both checkpoints, average its available rollouts, and average problems unweighted. Deltas are selected minus checkpoint 0; negative improves. Scoreable outputs (0/selected): ours 1,893/2,702; Multi-RLCR 2,686/2,675. Bold: best trained-checkpoint point estimate per metric among the arms for which it is defined (upper panel) and the larger improvement (lower panel); intervals often overlap, so bold does not imply a significant difference.
679 problems, four rollouts per problem and condition, generated at T=1, top-p .95, 1,792-token cap. Each cell is a histogram over the problem's gold answer clusters plus an Other bucket and an Unmatched bucket. The crowd distribution is drawn as a wide grey bar behind each cluster in every cell. Histograms are built with the repo's own parser, WordNet matcher and forecast code, and reproduce the table means above to four decimals. How these forecasts evolve during training is in Part 3.
epsilon_deciles.yaml) and raw-prior back-off, computed by the training reward function, minus the crowd entropy. It is not normalized on gold support, so an unlisted gold cluster costs about 50 nats times its crowd mass.The ProtoQA ours run followed through training, in two views. Development set: the 104 development questions evaluated at every in-training checkpoint (0, 20, …, 260; eight rollouts each), as population trends and question by question for ten sampled questions. Training stream: the 69,632 training rollouts themselves, as weight statistics pooled by fifth of the single pass and group by group at every optimizer step, with the trainer's advantages and the reward breakdown.
Every point and every cell below comes from the in-training evaluations on the fixed 104-question development set: the same questions at checkpoints 0, 20, …, 260, eight rollouts each, rescored with the training reward. This is the paired view of learning.
Each point is one in-training evaluation checkpoint (0, 20, …, 260): the mean over the 8 × 104 evaluation rollouts of that checkpoint, not a running average over training.
Sampled with a fixed seed to spread over cluster counts. Move the pointer over a bar to see the cluster's aliases, its crowd probability, its base-model prior and the forecast mass placed on it. The rollout dropdown in each question's header switches all 14 checkpoint cells of that question between the average over the eight rollouts and one chosen rollout.
Everything below comes from the 69,632 training rollouts themselves, 272 optimizer steps × 32 questions × 8 rollouts, each question generated once in the single pass. First pooled by fifth of the pass, then one group at a time.
Each valid rollout's weight vector (five listed weights and Other, rounded to .01) is counted across the training rollouts. The policy converges to a handful of templates, so within a group of 8 the weights barely differ; the only shape decision left is which answer occupies which slot. The last two columns compare the advantage variance over all groups with the variance inside groups whose 8 rollouts list exactly the same gold clusters, where coverage is held fixed and only weights, Other and unmatched mass can differ.
Means or medians over the 256 rollouts of each optimizer step, ours and the no-KL Multi-RLCR run on the same 272 batches, with a 9-step centered moving average. Unlike the checkpoint charts in Part 3 A, these are the training rollouts themselves, so the questions change at every step.
Pick an optimizer step (every 2nd step and the last one are stored) and one of its 32 questions. The first chart is the crowd distribution; then, row by row, our rollout k beside Multi-RLCR's rollout k on the same question at the same step (the Multi-RLCR run campaign2_protoqa_marlcr_seed42_nokl_v1 walks the data in the same order: every step's 32-question batch matches ours). Rollouts are not paired across arms. Multi-RLCR's reward is 0.5 · format + 0.5 · (number of distinct gold clusters hit) − 0.5 · candidate Brier over its five candidates, with the same exact-first WordNet matching; its advantage is reward minus group mean. In parentheses after every advantage is the standardized advantage, (reward − mean) / (std + 10⁻⁴), what TRL's default group scaling would have used. Our rollouts are shown exactly as the trainer scored them: the trainer's advantage for the rollout, its reward, and the reward split into the listed-gold term, the unlisted-gold term (Other × prior + ε), the normalization penalty and the format-fail reward. Each rollout's <hypotheses> block can be expanded. These are training questions, each seen once during the single pass, unlike the paired development set above. Reward and advantage are the trainer's; the breakdown is recomputed with the current strict parser and matcher, so a rollout is flagged when the deployed training parser accepted it (it took the last hypotheses block and tolerated drafts in the reasoning) but the current one does not, or when the recomputed terms do not add up to the trainer's reward.
The parser-fixed, KL-free rerun of our framework (campaign2_hotpotqa_ours_parserfix_nokl_seed42, 417 optimizer steps) beside the KL-free Multi-RLCR run (campaign2_hotpotqa_marlcr_seed42_nokl_v1). Both runs walk the 25,000 training questions in the same order: at every step the two 60-question batches are identical (verified for all 405 steps present in the Multi-RLCR log, whose per-step means also match its trainer log; the log lacks steps 193–200 and 287–290). Eight rollouts per question. Our reward is log(qgold + ε) with ε = 4.77 × 10⁻¹⁴, qgold the weight on exactly-matching listed answers or Other × prior when none matches, minus 0.2 |Σ weights − 1|, and −82.16 for malformed output; it is reproduced exactly on every valid rollout. Multi-RLCR's reward is 0.5 · format + 0.5 · (any candidate correct) − 0.5 · candidate Brier. Advantages are reward minus the group mean in both runs.
Means over the 480 rollouts of each step, both arms on the same steps; a 9-step centered moving average is drawn over the raw per-step values. Hit rate is the share of valid rollouts with a matching candidate. Multi-RLCR series stop where its log has gaps.
Every 16th step and the last one are stored (27 steps × 60 questions × 2 arms × 8 rollouts, with full chains of thought). Pick a step and a question: the two columns show the eight ours rollouts and the eight Multi-RLCR rollouts of that very question at that very step, with the trainer's advantage (and, in parentheses, the standardized advantage (reward − mean) / (std + 10⁻⁴) that TRL's default group scaling would have used), the reward and its terms, the length of the completion (tokens, reasoning characters, answer characters, and whether it hit the 1,792-token cap), the chain of thought in a scrollable box, and the final answer block. Rollouts are not paired across arms; row k is simply rollout k of each run.