← BFCL failure explorer · genlm.org/rollouts

Chain-of-thought length vs. answer correctness

On BFCL function-calling, longer reasoning tracks wrong answers — but most of that is problem difficulty, not the thinking itself.

Qwen3-8B & 14B · temperature 1.0 · 13 single-turn categories · 728,200 samples · length = generation output tokens (CoT-dominated) · valid = BFCL AST/exec pass · source: HF clementepasti/bfcl-rollouts

1 · Pass rate falls as reasoning grows

Each point is a 40-token bin; y is the share of samples in that bin that passed. The rightmost point is 800+ tokens. Monotonic collapse from ~92% to ~41%.

Qwen3-8B Qwen3-14B

2 · Wrong answers skew long

Length distribution within each outcome class (each curve sums to 100% of its own class), per model. Correct answers pile up short; wrong answers have a long right tail.

Correct Wrong

Qwen3-8B

Qwen3-14B

3 · Controlling for the problem, the effect is real but small

This is a difficulty control: the collapse above conflates length with difficulty, so here we hold the problem fixed and compare, on the same problem, the samples the model got wrong against the ones it got right (mixed-outcome problems only). The wrong attempts think only slightly longer, winning the "longer" coin-flip just 54% of the time — so most of the aggregate effect was difficulty.

4 · The pattern holds in every category

Point-biserial correlation between output tokens and correctness, per category (all negative). Strongest where the task is open-ended (live_multiple); weakest on relevance/parallel.

5 · Backtracking (“wait”) tracks failure even harder

Count of the word “wait” (case-insensitive, word-boundary; “but wait” is the stronger self-correction marker) per rollout — a proxy for how much the model second-guesses itself. Wrong answers average 3–4× as many.

Qwen3-8B Qwen3-14B

Same difficulty control as §3, on the same mixed-outcome problems: comparing wrong vs right attempts on the same problem, the “wait” and “but wait” signals disappear. The wrong attempt carries more of them only about half the time (or less) — so unlike CoT length (which kept a ~54% tilt), the strong aggregate correlations above are entirely problem difficulty, not a per-attempt cause of failure.

6 · Same problem, right vs. wrong — what the model actually does

Ten mixed-outcome problems per model (both a correct and a wrong answer exist among the 100 samples). Each shows a representative pair — the passing and failing samples nearest the median CoT length for their class, so neither is cherry-picked. The reasoning trace is collapsed; expand it to read what the model was thinking.

7 · BFCL correctness probes — held-out problems (corrected split)

Sanitized evaluation. We split the full set of 3,641 BFCL problem IDs before looking at validity labels: 1,200 train / 200 validation / 400 test problems, category-stratified, with zero problem-ID overlap and the same IDs for Qwen3-8B and Qwen3-14B. Each problem contributes 100 T=1 rollouts, giving 120,000 training / 20,000 validation / 40,000 test rows per generator. All-correct, mixed, and all-wrong problems remain in the evaluation; this is not the earlier mixed-problem-conditioned experiment. The LM judge is the Qwen3-8B LoRA checkpoint at step 500, trained on this same problem-disjoint setup; it sees the BFCL prompt plus candidate final answer but not the CoT.

Test composition · 400 problems	all wrong	mixed	all correct	test rollouts
Qwen3-8B	23 · 5.8%	142 · 35.5%	235 · 58.8%	40,000
Qwen3-14B	20 · 5.0%	120 · 30.0%	260 · 65.0%	40,000

The average within-problem AUROC is defined only on the 142 / 120 mixed test problems; pooled AUROC and selection accuracy use all 400 test problems. Confidence intervals resample whole problems.

Generator · probe	selected representation	pooled AUROC	accuracy@0.5	accuracy@0.66	accuracy@0.75	average within-problem AUROC 95% problem-bootstrap CI	best-of-100	random pick	oracle	ECE
Qwen3-8B · linear	last token · layer 26 · C=0.01	0.8420	0.8238	0.7938	0.7690	0.6849 [0.6452, 0.7247]	0.8900	0.8284	0.9425	0.1003
Qwen3-8B · MLP	last token · layer 26 · epoch 11	0.8531	0.8509	0.8324	0.8164	0.7028 [0.6585, 0.7471]	0.8650	0.8284	0.9425	0.0976
Qwen3-8B · LM judge	Qwen3-8B LoRA · checkpoint 500	0.9008	0.8942	0.8866	0.8819	0.8836 [0.8373, 0.9267]	0.8950	0.8284	0.9425	0.0683
Qwen3-8B · total logprob (baseline)	sequence log-probability · no training	0.6930	—	—	—	0.5564 [0.5127, 0.5999]	0.8200	0.8284	0.9425
Qwen3-14B · linear	last token · layer 40 · C=0.01	0.8549	0.8495	0.8283	0.8094	0.6952 [0.6469, 0.7435]	0.8750	0.8483	0.9500	0.0997
Qwen3-14B · MLP	last token · layer 28 · epoch 23	0.8797	0.8692	0.8617	0.8545	0.7042 [0.6545, 0.7539]	0.8700	0.8483	0.9500	0.0935
Qwen3-14B · LM judge	Qwen3-8B LoRA · checkpoint 500	0.8982	0.9035	0.9063	0.9025	0.8269 [0.7661, 0.8845]	0.8811	0.8483	0.9500	0.0616
Qwen3-14B · total logprob (baseline)	sequence log-probability · no training	0.6793	—	—	—	0.5699 [0.5254, 0.6158]	0.8375	0.8483	0.9500

The updated probes use five-fold cross-validation grouped by problem over the 1,400-problem development pool (1,200 train + 200 validation); the 400 test problems remain untouched until final evaluation. Selection maximizes average within-problem AUROC across mixed development problems. “Best-of-100” selects the highest-scored rollout among all 100 test rollouts for each problem. Random pick includes invalid rollouts; oracle succeeds exactly when a problem has at least one valid rollout. The total-logprob baseline ranks rollouts by the generator’s own sequence log-probability Σ log p (no trained parameters; no length normalization, per JHV’s point that the per-token mean is hard to interpret). It is essentially a length proxy (corr with completion length −0.98 / −0.94 for 8B / 14B) and selects at or below random pick (best-of-100 8B: 0.8200 vs 0.8284). The length-normalized per-token mean behaves similarly (within-problem AUROC 0.5550 / 0.5796; best-of-100 0.8100 / 0.8525).

Best-layer search status. Completed CoT-only winners are 8B linear: last-token layer 26 (CV within-problem AUROC 0.6695); 8B MLP: last-token layer 26 (0.7104); 14B linear: last-token layer 40 (0.7097); and 14B MLP: last-token layer 28 (0.7584). With the final answer included, winners are 8B linear: last-token layer 28 (0.7999), 8B MLP: last-token layer 28 (0.8250), 14B linear: last-token layer 33 (0.7898), and 14B MLP: last-token layer 25 (0.8224).

Qwen3-8B CoT-only linear — layer-wise CV within AUROC

last token · best C per layermean pooled · best C per layer

Selected: last-token layer 26, C=0.01.

Qwen3-14B CoT-only linear — layer-wise CV within AUROC

last token · best C per layermean pooled · best C per layer

Selected: last-token layer 40, C=0.01.

Each bar is the largest five-fold CV mean within-problem AUROC among C∈{0.01, 0.1, 1} for that layer and pooling method. Cross-validation is grouped by problem; mixed development problems receive equal weight.

Embedding-layer results. Layer-0 mean and last-token embeddings are now complete for all 180,000 existing rollouts from each generator. Mean pooling was selected in every embedding-only run; last-token layer-0 representations are constant at the shared end token and consequently give chance-level validation ranking. Embedding-only linear pooled/within AUROC is 0.692/0.594 for 8B and 0.682/0.600 for 14B; embedding-only MLP is 0.783/0.630 and 0.750/0.630. These are informative but clearly weaker than the corresponding residual-stream probes.

Accuracy applies the stated probability threshold to every one of the 40,000 test rollouts. Since every problem contributes exactly 100 scored rollouts, pooled accuracy and the equal-problem mean accuracy are numerically identical.

Qwen3-8B — held-out pooled ROC

linear CoT L26linear embeddings L0 meanlinear + answer L28MLP CoT L26MLP + answer L28LM judgetotal logprobchance

Qwen3-14B — held-out pooled ROC

linear CoT L40linear embeddings L0 meanlinear + answer L33MLP CoT L28MLP + answer L25LM judgetotal logprobchance

Inference runtime per rollout. With generator activations already cached, linear ≈0.3 µs and MLP ≈20–30 µs on CPU. The checkpoint-500 Qwen3-8B LM judge requires a separate full forward pass: completed 5,000-row GPU shards sustained approximately 3.3–5.8 rollouts/s, or ≈0.17–0.30 s/rollout including model loading and output writing. These are pipeline measurements on the Euler hardware used for the experiments, not hardware-normalized kernel benchmarks; activation extraction cost is excluded from the probe figures.

Best-of-N on held-out problems. For each problem and N, sample a size-N subset without replacement from its 100 rollouts and select the highest-scored candidate. Curves are exact expectations over subsets; equal-score maxima are broken uniformly. Shaded 95% intervals come from 10,000 whole-problem bootstrap replicates (seed 0). All 400 test problems—including homogeneous ones—receive equal weight.

Qwen3-8B — held-out best-of-N

linear CoT L26linear + answer L28MLP CoT L26MLP + answer L28LM judgetotal logproboraclerandom

Qwen3-14B — held-out best-of-N

linear CoT L40linear + answer L33MLP CoT L28MLP + answer L25LM judgetotal logproboraclerandom

Mixed-problem conditional view. The plots below repeat the identical calculation after restricting evaluation to the 142 mixed 8B problems and 120 mixed 14B problems. Every retained problem has at least one correct and one wrong rollout, so these curves isolate ranking quality and reach an oracle value of 1 at N=100. They should not be interpreted as population-level BFCL accuracy.

Qwen3-8B — mixed-only best-of-N (142 problems)

linear CoT L26linear + answer L28MLP CoT L26MLP + answer L28LM judgetotal logproboraclerandom

Qwen3-14B — mixed-only best-of-N (120 problems)

linear CoT L40linear + answer L33MLP CoT L28MLP + answer L25LM judgetotal logproboraclerandom

Best-of-N broken down by category. The aggregate curves above mix categories whose composition is far from uniform, and the detection categories dominate the pool. The panels below repeat the identical calculation (exact subset expectation, 10,000 whole-problem bootstrap replicates, seed 0) within each category of the same 400-problem held-out split, for every category with at least 20 test problems. Each panel carries its own random-pick and oracle anchors, so the visible gap is the selection headroom that actually exists in that category.

linear CoTMLP CoTlinear + answertotal logproboraclerandom

irrelevance · 27 problems

live_irrelevance · 97 problems

live_multiple · 115 problems

live_simple · 28 problems

multiple · 22 problems

parallel · 22 problems

parallel_multiple · 22 problems

simple_python · 44 problems

Reading the panels: on irrelevance and live_irrelevance the probes sit on the oracle line — selection is essentially perfect. On the AST categories the random-pick and oracle lines are close together (the pools are already 93–97% correct) and the probe curves are flat or below random, so the aggregate gain of the combined curve is carried almost entirely by the detection categories. Total logprob is at or below random pick nearly everywhere.

Does the probe discriminate, or just prefer abstention?

CoT only — full pool, cross-fit (3,641 problems; the condition-B selector)

ground truth	picked a call	picked no call	correct behaviour
Relevant — a call is required	2465	52	0.979
Irrelevant — no call is correct	100	1024	0.911

CoT + answer — full pool, cross-fit (3,641 problems; the condition-C selector)

ground truth	picked a call	picked no call	correct behaviour
Relevant — a call is required	2454	63	0.975
Irrelevant — no call is correct	89	1035	0.921

A selector that simply favoured “say nothing” would score well on the irrelevance categories for the wrong reason. The tables below separate the two hypotheses on the same 400 held-out problems (top row) and on the full 3,641-problem cross-fit pool scored by the deployed selectors (bottom row): each rollout is labelled by whether its answer emits a decodable function call, and the cells count problems by what the probe’s best-of-100 pick actually did. The call detector agrees with the benchmark’s own labels on 99.4% of the 12,400 irrelevance rollouts (passing an irrelevance entry is defined as emitting no call).

linear probe, CoT only (L26)

ground truth	picked a call	picked no call	correct behaviour
Relevant — a call is required	274	2	0.993
Irrelevant — no call is correct	9	115	0.927

linear probe, CoT + answer (L28)

ground truth	picked a call	picked no call	correct behaviour
Relevant — a call is required	272	4	0.986
Irrelevant — no call is correct	8	116	0.935

Reading: on problems that require a call the probe selects one 99.3% of the time (CoT-only), so it is not applying a blanket “abstain” rule — a rule that would have scored just as well on the irrelevance categories, which is why those numbers alone cannot settle the question. What it does have is a small tilt: across the full 3,641-problem pool the no-call rate among picks is 2.6× its rate in the pools on call-requiring problems (1.9% vs 0.7%), and roughly twice that for the answer-inclusive probe — the same tilt that costs it the small live_relevance control.

Annealing classifier guidance — with replacement. For each N, draw N candidates independently with replacement from each problem's empirical 100-rollout T=1 proposal, then resample one occurrence with weight MLP_score^β. β=0 is ordinary random sampling, β=1 is classifier importance resampling, and β=∞ is exactly with-replacement best-of-N. It is not the without-replacement best-of-N curve above: at N=100, IID sampling from a 100-row pool sees only about 63.4 distinct rows on average, whereas sampling without replacement sees all 100. Finite-β expectations use 20,000 shared proposal draws per problem and N (seed 0); shaded 95% intervals bootstrap whole problems 10,000 times. The upper pair uses all 400 test problems; the lower pair conditions on mixed problems only.

β=0β=0.25β=0.5β=1β=2β=4β=∞

Qwen3-8B — MLP L26 annealing with replacement · all 400

Qwen3-14B — MLP L28 annealing with replacement · all 400

Qwen3-8B — MLP L26 annealing with replacement · 142 mixed

Qwen3-14B — MLP L28 annealing with replacement · 120 mixed

Probe and training details. The linear model is logistic regression. The MLP is Linear(d,512) → GELU → Dropout(0.2) → Linear(512,1), trained with weighted binary cross-entropy and AdamW (learning rate 10⁻³, weight decay 10⁻⁴, batch 4,096, maximum 60 epochs, patience 8). Candidate representations covered both the last token and the mean of token hidden states at embedding layer 0 and ten transformer depths. Runs are logged in W&B.

Is there room to improve? Probably some, but the present curves do not argue for simply training longer: early stopping selected epoch 10 for 8B and 23 for 14B, well before the 60-epoch cap. The MLP improves over linear on 14B (+0.027 average within-problem AUROC), but is worse on 8B (−0.026), so added nonlinearity is not uniformly useful. The clearest next experiments are a small hidden-width/dropout/weight-decay sweep, multi-layer feature combinations, and repeated split seeds; larger claims should wait for those controls.

8 · Beyond BFCL — LiveCodeBench: the probe vs a trained LM judge

Does the activation probe hold up on a different domain, against a purpose-built verifier? Same recipe on LiveCodeBench thinking-rollouts (generators Qwen3-4B-think / 14B-think, 154 held-out problems, by-problem split — unseen-problem generalization, unlike the earlier row-level analyses): our answer-withheld probe (linear + MLP on generator activations over prompt + CoT — the final code never shown, matched training budget) vs Sheppard's LM judge — his published LoRA checkpoints on Qwen3-8B, which read problem + extracted final code only (no CoT) and answer Yes/No. The two see disjoint parts of the rollout. All §8 evaluation metrics and plots use T=1.0 only; no other generation temperatures enter the reported AUROC, ROC, scatter, coverage, or best-of-N results. Problem split: 500 train / 60 validation / 154 test problems—70.0% / 8.4% / 21.6% of the 714 problems, with no problem overlap. Probe weights are fitted on train only. The 60 validation problems choose the activation layer, pooling, regularization and MLP stopping point; after those choices are fixed, the 154 test problems are used once for the reported evaluation. Rows: matched training is capped at 100 rollouts/problem (50,000 intended); after requiring a closed CoT, the actual training matrices contain 46,930 rows for 4B and 48,651 for 14B. The final T=1 test set starts from exactly 15,400 rollouts per generator (100 × 154); 14,970 / 15,070 are jointly scoreable by all methods for 4B / 14B. Sheppard's published judge follows the same 500-problem, ≤100-rollout/problem training cap (≤50,000 rows), although its exact selected training row IDs are not published. Scoring cost, measured: the probe runs off activations already cached from generation at ~0.1–0.23 ms/rollout on 4 CPU threads; the judge needs its own 8B forward pass at ~110 ms/rollout batched on an A100.

LCB probe-training composition	all training rows wrong	mixed training rows	all training rows correct	training problems
Qwen3-4B-think · 46,930 rows	53 · 10.6%	202 · 40.4%	245 · 49.0%	500
Qwen3-14B-think · 48,651 rows	46 · 9.2%	150 · 30.0%	304 · 60.8%	500

Composition is computed from the exact closed-CoT activation matrices used to fit the LCB linear and MLP probes. It describes the training rows, not the 154 held-out T=1 problems. Sheppard's judge uses the same 500-problem cap, but its exact selected training rows are unpublished.

LCB T=1 macro over mixed-label problems from the same 154-problem held-out set	mean within-problem AUROC	mean accuracy@0.5 95% problem-bootstrap CI	mean precision@0.5	mean recall@0.5	mean F1@0.5
Qwen3-4B · linear · 65 mixed problems	0.628	0.727 [0.649, 0.800]	0.760 [0.669, 0.841]	0.700 [0.599, 0.797]	0.674 [0.577, 0.766]
Qwen3-4B · MLP · 65 mixed problems	0.633	0.711 [0.637, 0.786]	0.781 [0.693, 0.862]	0.684 [0.585, 0.778]	0.669 [0.576, 0.759]
Qwen3-4B · LM judge · 65 mixed problems	0.629	0.764 [0.700, 0.827]	0.800 [0.729, 0.864]	0.841 [0.773, 0.902]	0.774 [0.702, 0.841]
Qwen3-4B · MLP + judge · 65 mixed problems	0.653	0.752 [0.683, 0.819]	0.807 [0.734, 0.873]	0.784 [0.699, 0.862]	0.738 [0.656, 0.815]
Qwen3-14B · linear · 47 mixed problems	0.622	0.674 [0.587, 0.755]	0.761 [0.663, 0.852]	0.644 [0.530, 0.750]	0.624 [0.520, 0.724]
Qwen3-14B · MLP · 47 mixed problems	0.657	0.688 [0.605, 0.765]	0.800 [0.711, 0.879]	0.609 [0.499, 0.716]	0.614 [0.512, 0.710]
Qwen3-14B · LM judge · 47 mixed problems	0.638	0.736 [0.652, 0.812]	0.751 [0.656, 0.836]	0.866 [0.784, 0.937]	0.754 [0.660, 0.839]
Qwen3-14B · MLP + judge · 47 mixed problems	0.662	0.792 [0.727, 0.851]	0.800 [0.710, 0.879]	0.769 [0.669, 0.862]	0.738 [0.643, 0.825]

Each metric is computed separately within each mixed-label problem on the shared scoreable pool, then averaged with equal problem weight. Intervals resample problems 10,000 times (seed 0). If a scorer predicts no positives for a problem, its precision and F1 are defined as zero. Pooled @0.5 metrics in the table above weight problems in proportion to their number of scoreable rollouts.

ROC — per-rollout, pooled, T=1.0 only. The simple ensemble is the arithmetic mean of the MLP and LM-judge probabilities on their shared scoreable pool. It reaches pooled AUROC 0.905 / 0.907 for 4B / 14B, above every individual scorer; its mean within-problem AUROC is 0.653 / 0.662. Pooled AUROC uses T=1 rollouts from all 154 test problems, including homogeneous problems. Mean within-problem AUROC asks a different question—whether a scorer ranks correct above wrong rollouts of the same problem: compute AUROC separately for each mixed-label problem, then average those problem AUROCs equally. It is defined only for the 65 / 47 mixed-label T=1 problems in 4B / 14B. A ROC for one all-correct or all-wrong problem is mathematically undefined because one of TPR/FPR has a zero denominator; those problems remain represented in pooled AUROC but cannot contribute a separate within-problem AUROC.

probe — linearprobe — MLPLM judgeMLP + judge averagechance

Qwen3-4B-think — ROC

Qwen3-14B-think — ROC

Calibration at T=1.0. Points compare mean predicted probability with empirical correctness in ten equal-width score bins. ECE and threshold metrics use the same shared scoreable pool as pooled AUROC; unlike AUROC, they depend on the numerical probability scale and the 0.5 threshold.

Qwen3-4B-think — calibration

Qwen3-14B-think — calibration

How complementary are the two scores at T=1.0? Each point is one rollout from the shared T=1 evaluation pool: LM-judge probability on x, MLP probability on y. Green passed execution; red failed. All 14,970 / 15,070 jointly scoreable candidates are drawn for 4B / 14B—no subsampling.

correctwrong

Qwen3-4B-think — judge vs MLP score

Qwen3-14B-think — judge vs MLP score

End-to-end best-of-N at T=1.0. Every curve uses the same 100 original rollouts for each of all 154 problems; malformed or otherwise unscoreable candidates are never removed. When is a score missing? The linear and MLP probes require a closed reasoning segment: if the raw generation has no closing </think>, no activation example or probe score is produced. Their 24,576-token extraction limit is a fail-fast dataset gate—not a row filter—and no reported row was dropped by it. The LM judge requires non-empty extracted_code and a formatted problem-plus-code judge prompt of at most 12,288 tokens; an empty code extraction or an over-limit prompt receives no judge score. Execution failure by itself does not make a score missing. The ensemble requires both the MLP and judge scores. A missing score ranks below every available score; if all N candidates are unscoreable, selection falls back to a uniform pick. Values are exact expectations over sampling without replacement and random tie-breaking—not Monte Carlo estimates. The dotted baseline is the original generator pass rate; the dashed curve is oracle pass@N on those same original rollouts. At N=100, linear / MLP / judge / ensemble selection accuracy is 0.825 / 0.805 / 0.815 / 0.818 for 4B and 0.867 / 0.844 / 0.828 / 0.857 for 14B. Ensemble-minus-judge is +0.003 [−0.032, +0.039] for 4B and +0.029 [−0.006, +0.068] for 14B. T=1 score coverage is 97.8% / 97.8% / 97.2% / 97.2% for linear / MLP / judge / ensemble on 4B and 98.4% / 98.4% / 97.9% / 97.9% on 14B.

Why can accuracy fall as N grows? “Best” means the highest verifier score, not necessarily the correct answer. The maximum predicted score is monotone in N, but the correctness of the item holding that maximum is not. Adding candidates can introduce a confidently wrong rollout whose score exceeds every previously sampled correct rollout; the selector then switches from correct to wrong. With more draws, this extreme-value or “winner's curse” risk grows. Only the oracle curve is guaranteed to be non-decreasing, because it ranks candidates by their true validity. A learned verifier's best-of-N curve measures the competition between finding more correct candidates and finding more high-scoring false positives, so saturation or decline is a real selector failure mode rather than a contradiction.

probe — linearprobe — MLPLM judge (sees code)MLP + judge averageoracle pass@N (dashed)random pick (dotted)

Qwen3-4B-think — best-of-N selection accuracy

Qwen3-14B-think — best-of-N selection accuracy

Mixed-problem view. The same end-to-end calculation restricted to problems whose full set of 100 original T=1 rollouts contains both correct and wrong candidates: 66 problems for 4B and 47 for 14B. This is 66 rather than the 65 shown in the shared-scoreable-pool table because one 4B problem is mixed in the original rollout pool but becomes homogeneous after requiring every method's score. Bands are 95% confidence intervals from 10,000 whole-problem bootstrap replicates.

Qwen3-4B-think — mixed-only best-of-N (66 problems)

Qwen3-14B-think — mixed-only best-of-N (47 problems)

Leading bin (<250 samples, <80 tokens) omitted from chart 1 as noise. Correlation is not causation: chart 3 is the difficulty-controlled read. Cards show the sample nearest each class's median length; error type is BFCL's grader verdict. 1 100k/MLP run: 100k train / same 7,377 eval; both data & capacity gains outside the 95% CIs.

9 · DS-1000 & LiveCodeBench — layerwise probes (interim)

Interim results — campaign in progress. Same generator family, new domains: Qwen3-8B-think rollouts at t=1.0 from the group's thinking campaign (DS-1000: 1,000 problems ×100, random pick 0.501, best-of-100 oracle 0.811, 649 mixed problems; LCB: 714 ×100, random 0.821, oracle 0.906, 261 mixed). Protocol identical to §7: K=3 problem-disjoint cross-fit (seed 0), nested grouped val slices, class-balanced logistic probes on last-token activations at every layer, both spans (CoT-only at </think>; CoT+answer at the final token). Prompt reconstruction validated per-row against the campaign's teacher-forced logprobs (token identity 1.0 on both domains). Curves show test-fold within-problem AUROC by layer.

DS-1000 — linear probe by layer (within-problem AUROC)

computed on 1,000 problems × 100 rollouts; CoT-only span uses the 93,757 rows with a closed CoT

linear CoT onlylinear CoT + answerdashed = MLP-2 (4096→512→256→1), same folds

LiveCodeBench — linear probe by layer (within-problem AUROC)

computed on 714 problems × 100 rollouts; CoT-only span uses the 69,399 rows with a closed CoT

linear CoT onlylinear CoT + answerdashed = MLP-2 (4096→512→256→1), same folds

Linear results. DS-1000: CoT-only peaks at layer 22 with within-problem AUROC 0.584 — far below BFCL's 0.68 — while CoT+answer reaches 0.701 at layer 19, and the all-layer ensembles push to 0.727 (concat) / 0.720 (stacked): on DS-1000, reading the answer matters far more than on BFCL. LCB: 0.614 (CoT-only) / 0.653 (CoT+answer, layer 16) — an independent replication of our earlier LCB numbers through a different pipeline. Pooled AUROCs are higher everywhere (up to 0.93 on LCB) but conflate problem difficulty and are not used for selection. MLP sweep (dashed): input = one layer's last-token vector, d_model = 4096 (Qwen3-8B). MLP-1: 4096→512→1 (ReLU, ≈2.10M params); MLP-2: 4096→512→256→1 (≈2.23M params); class-balanced BCE, AdamW, early stop on grouped-val within-AUROC. MLP-2 at every layer beats linear on all four domain × span cells — most on DS-1000 CoT+answer, 0.744 at layer 21 vs linear's 0.701 (also above concat 0.727 and the stacker 0.720), consistent with capacity paying off at ~60k training rows per fold. LCB: 0.666 vs 0.653 (CoT+answer, both picking layer 16), 0.629 vs 0.614 (CoT-only).

Best-of-N selection (interim: linear probes and ensembles; attention follows). Same estimator as every other selection plot on this page: exact expectation over size-N subsets drawn without replacement from each problem’s 100 rollouts, 95% bands from a 10,000-replicate whole-problem bootstrap (seed 0). Left column: unconditional over all problems. Right column: restricted to mixed problems (at least one correct and one wrong rollout — the population where selection can act; oracle = 1 by construction there). One comparability caveat: the CoT-only curves are computed on each problem’s closed-CoT rollouts only (~94% / ~97% of the pool), so their per-problem candidate pools differ slightly from the other curves.

linear CoT onlylinear CoT + answerstacked ensemble (CoT + answer)concat all layers (CoT + answer)MLP-2 (CoT + answer)MLP-1 (CoT only)total logproboraclerandom

DS-1000 — best-of-N · all problems

computed on … problems × 100 rollouts

DS-1000 — best-of-N · mixed problems only

computed on … problems × 100 rollouts

LiveCodeBench — best-of-N · all problems

computed on … problems × 100 rollouts

LiveCodeBench — best-of-N · mixed problems only

computed on … problems × 100 rollouts

Reading. The answer-inclusive linear probe delivers the only positive selection: DS-1000 +6.3pp over random pick at N=100 (0.564 vs 0.501; +9.6pp on mixed, and the stacked ensemble adds to 0.641 there against a 1.0 oracle); LCB +2.1pp (0.842 vs 0.821; +5.6pp mixed). Total logprob is exactly at random everywhere. Corrected reading (the earlier apparent below-random fall of the CoT-only curves was an estimator artifact: closed-CoT pools hold ~94 rollouts and the subset estimator returned 0 for N above the pool size; truncated-CoT candidates are now scored never-select, which restores 100-candidate pools): the CoT-only probe selects nearly as well as the answer-inclusive one. DS-1000 at N=100: CoT-only 0.557 / CoT+answer 0.564 / stacked ensemble 0.578 vs random 0.501 (mixed-only: 0.607 / 0.619 / 0.641 vs 0.523). LCB: CoT-only and CoT+answer tie at 0.842 vs random 0.821 (mixed-only: both 0.824 vs 0.768). Total logprob sits at random everywhere. The MLPs extend the lead: MLP-2 (CoT+answer) is DS-1000's best selector at 0.609 (+10.8pp over random; 0.689 on mixed vs the stacker's 0.641), and MLP-1 (CoT-only) edges ahead on LCB at 0.846 (mixed 0.835). Its wrong argmax picks are short, confidently concluded CoTs (score–length correlation −0.32), not degenerate outputs.

Attention family at layer 4 — first epochs (preliminary). The attention probes train online over every token position of the span. Values are val within-problem AUROC, the mean over 3 folds of each fold's best epoch (the epoch column gives that epoch per fold), against converged linear probes. Full-layer runs are in flight. The entropy column is the attention-weight Shannon entropy normalized by its maximum ln T, where T is the number of positions the head attends over: the rollout's generated tokens only (prompt excluded), up to and including </think> for the CoT-only span or up to the final token for CoT+answer — H(α)/ln T ∈ [0,1], so 1 = uniform weights (mean pooling) and 0 = a single-token spike, comparable across rollouts of different lengths (multi-head variants: weights averaged over heads first). Mean over 240 rollouts (DS-1000, one per sampled problem): the free-query heads are still effectively mean-pooling, while cross-attention has already learned to concentrate — on the CoT+answer span it puts ~30% of its mass on the final 1% of tokens.

domain	span	probe	val within-AUROC @ layer 4	attention entropy	epoch of best (f0/f1/f2)
DS-1000	CoT only	linear @L4 (control)	0.535	—	—
DS-1000	CoT only	attention (free query)	0.540	0.99 (uniform)	1 / 0 / 1
DS-1000	CoT only	cross-attention (multi-head, H=4)	0.544	0.37	1 / 1 / 0
DS-1000	CoT + answer	linear @L4 (control)	0.719	—	—
DS-1000	CoT + answer	attention (free query)	0.568	0.99 (uniform)	1 / 0 / 1
DS-1000	CoT + answer	cross-attention (multi-head, H=4)	0.582	0.04–0.41	1 / 1 / 0
LCB	CoT only	linear @L4 (control)	0.643	—	—
LCB	CoT only	attention (free query)	0.623	n/m	0 / 0 / 0
LCB	CoT only	cross-attention	0.597	n/m	0 / 0 / 0
LCB	CoT + answer	linear @L4 (control)	0.613	—	—
LCB	CoT + answer	attention (free query)	0.640	n/m	0 / 0 / 0
LCB	CoT + answer	cross-attention	0.628	n/m	0 / 0 / 0

n/m = not yet measured (the attention-map autopsy has run on DS-1000 only). Interim by design: the relaunched chains train all top-5 layers per span for up to 6 epochs with early stopping; best-of-N selection curves against oracle/random follow with the final export.