Generation logprobs as a quality signal

Per-completion log-likelihoods of the thinking-campaign rollouts under the distribution the sampler actually drew from (t=0.6, top-p 0.95, top-k 20), reconstructed exactly and certified against fp32 full-vocabulary ground truth (mean deviation 0.007 nats/token, zero violations). 20 cells, 1,507,400 completions; the raw model measure ships in the same tables.

Likelihood ranks difficulty across problems, not correctness within them

Across problems the signal is strong and the calibration below is monotone: completions the model scores higher do pass more often. But that is difficulty (easy problems draw both high likelihood and high pass rates), not a verifier, and it does not survive conditioning on the problem. Computed per cell, never pooling arms or sizes whose per-token scales differ several-fold, the within-instance AUROC of the sampling log-probability ranges only 0.41 to 0.53 across all twelve cells, straddling and mostly sitting below the 0.5 no-signal line (first figure). Below 0.5 means the higher-likelihood sample of a problem is if anything the more likely to be wrong: within one problem the highest per-token likelihood goes to the lowest-entropy, most repetitive completions, which for the think arm are often degenerate and incorrect (the same repetition effect documented on the reasoning-traces page). So best-of-n by likelihood does not plateau at random, it drifts down as n grows for most cells (second figure): on LiveCodeBench 8B-think single-sample accuracy 0.81 drops to 0.75 at n=50 (oracle 0.90), and on DS-1000 8B-think it stays flat, 0.49 to 0.49, against a 0.78 oracle. A perfect verifier (the dotted oracle) would gain tens of points; likelihood gives them back. Mean sampling logprob is a difficulty meter for routing and abstention, not a reranker for samples of the same problem.

Sampler likelihood is near-monotonically calibrated

Empirical pass rate by decile of mean sampling logprob, per cell (all four cells per domain). The risk curve rises across nearly the whole range for every cell, with at most a single one-decile reversal, so the score ranks completions usefully far from the decision boundary and supports thresholding. This is the across-problem view of the same difficulty signal that, as the selection panel showed, does not survive conditioning on one problem.

Length explains the think-arm signal, not the nothink one

Shorter completions pass more often and per-token normalization already favours short outputs, so we ask how much of the pooled signal is just brevity. The two arms split sharply. For the nothink cells, shortness alone gives AUROC 0.60–0.87 and log-probability residualized on length still separates pass from fail (adjusted AUROC 0.63–0.83), so likelihood carries information beyond brevity. For the think cells the opposite holds: shortness alone is a very strong signal (AUROC 0.68–0.92, short traces are the ones that did not spiral) and once length is removed the log-probability collapses to chance or below (adjusted AUROC 0.42–0.53). The think-arm pooled signal is almost entirely a length proxy, consistent with its within-problem anti-signal in the selection panel.

Sampler confidence per cell

Thinking-arm completions run about 4–10× lower in per-token sampling likelihood than nothink (widest on DS-1000, narrowest on the LiveCodeBench windows): reasoning traces genuinely explore, while nothink decoding nearly retraces a deterministic path. This is the sampling-side view of the coverage/mode-collapse contrast from the pass@k analyses.

The warp gap: sampling vs raw measure

Per-token difference between the two measures on the same completions. Positive: truncation and sharpening concentrate mass on the tokens taken. The gap is a property of the sampling configuration, not model confidence, which is why both measures are published side by side.

Min-K%: is the original window memorised?

Shi et al. 2023, Detecting Pretraining Data (Min-K% Prob) test whether a piece of text was in a model's training set by averaging the K% lowest token log-probabilities in it: genuinely memorised text has few very-surprising tokens, so its Min-K% sits higher (less negative) than unseen text. The same low tail is the group/tail confidence that DeepConf (Fu et al. 2025) filters reasoning traces with. Here we score Min-20% on the model's own correct solutions and ask whether original-window solutions look more familiar than fresh-window ones. The paired bars are nearly equal within every cell, and the annotated per-cell AUROC for telling the two windows apart is only 0.52–0.58 (strongest for the think arm, essentially chance for nothink): at most a faint familiarity signal, consistent with the freshness drop reflecting harder unseen problem solving rather than lost recitation. (Note this scores the model's own generations, not the benchmark reference text, so it is a familiarity probe rather than the canonical benchmark-contamination test.)

Token fragility: is the single worst token better than the mean?

Shi et al. 2026, Code Is More Than Text argue that code has token fragility: one wrong token can break an entire program, so a program's uncertainty is concentrated in a few positions rather than spread evenly. As a minimal probe we take the single lowest chosen-token log-probability per completion, its most surprising token, and race it against the obvious baselines on the same completions: the mean log-probability, and Min-20% (the mean of the lowest fifth). The single worst token is a strong pooled pass/fail signal (AUROC 0.77–0.85) and it beats the mean-logprob baseline in six of the eight cells, most clearly on the fresh window and the nothink arm (up to +0.07 AUROC): failing programs reliably carry one catastrophic low-probability token that the mean washes out. Two caveats keep it honest. Min-20% barely differs from the mean, so it is specifically the single extreme token that adds signal, not the soft tail. And for the strongest cells (14B-think) the mean edges ahead, so those failures are more diffuse than a single bad token.

None of the three is a same-problem verifier, though. With the problem held fixed, the within-instance AUROC of all three collapses to the 0.5 line below, and for the 8B-think cells the mean logprob dips below 0.5 (the repetition/degeneration effect, looping high-probability failures, documented on the reasoning-traces page). Pooled, the tail is a difficulty meter; it does not rank samples of the same problem.

Top-K token entropy: strong pooled, weak within-problem

Shi et al. 2026, Code Is More Than Text propose Top-K token entropy as a cheap, single-pass code-uncertainty signal: at each position take the entropy of the top-K next-token distribution, then average over the completion, with no resampling. We recompute it on a subsample (top-32 head, shards 0–1 for nothink and 0–2 for think, roughly a fifth of the samples). Pooled it separates pass from fail strongly (blue, AUROC up to 0.85 on 14B-think), echoing their finding that a single-pass entropy can rival multi-pass uncertainty baselines. But within a fixed problem it too falls to the chance line (orange): entropy reads problem difficulty, not which sample is correct.

Self-certainty: the signal that survives conditioning

Kang et al. 2025 do reward-model-free best-of-N selection with self-certainty: the KL divergence of each step's next-token distribution from uniform (how peaked the model is), averaged over the completion. On the same subsample, picking the single highest-self-certainty sample per problem (blue) beats a random pick (grey) in every cell and recovers part of the gap to the oracle (green, any-correct upper bound). Crucially, unlike sequence likelihood or entropy, whose within-instance AUROC sat at ~0.5, self-certainty carries genuine within-problem signal (AUROC 0.54–0.68, strongest for the think cells): it is the one token-distribution statistic here that ranks different attempts at the same problem above chance.

Which failures does the model see coming? (per cell, never pooled)

A confidence signal is only useful for selective prediction (Kadavath et al. 2022, do models know what they know?) if the model is less confident when it is about to be wrong. Per-token confidence differs several-fold between the think and nothink arms, so we never pool them: each panel is one cell, and every bar is a failure theme's mean confidence relative to that same cell's correct-solution baseline (0 = exactly as confident as when correct, i.e. confidently wrong). Two things stand out. In the nothink cells every theme sits within about 0.02 nats/token of the baseline: nothink is roughly as confident on its wrong answers as its right ones whatever the error, so likelihood cannot flag any of them. In the think cells the gaps are larger and graded (up to ~0.06), so confidence does dip on failures, but which theme dips most is domain-specific, truncation and syntax lowest on DS-1000, logic errors lowest on LiveCodeBench. No single error type is universally the confidently-wrong one.

DS-1000: confidence does not track library difficulty (per cell)

DS-1000 spans seven Python libraries of very different hardness, computed per cell (pooling cells here mixes base rates and manufactures Simpson-style artefacts, an earlier pooled version of this panel showed a spurious Tensorflow inversion that vanishes once split by cell). Within every cell pass@1 swings enormously by library, from Matplotlib (0.63–0.73 across the four cells) down to Tensorflow (0.00–0.32; the nothink cells never solve it). If log-probability measured difficulty the verifier AUROC would fall with pass rate. Instead it is lowest on the easy Matplotlib (0.54–0.69) and among the highest on the harder Pandas (0.69–0.74), fairly consistently across all four cells: confidence separates pass from fail better where the library is harder, not where the model does better. Ankner et al. 2024 use perplexity as a difficulty proxy; within one benchmark, per cell, it does not behave like one.

OCaml: low-resource, low-confidence, but the model knows it

How does confidence behave on a language the model barely knows? We compare qwen3-14b-nothink on Python (DS-1000, LiveCodeBench) against OCaml (the multilingual LiveCodeBench port) using the raw logprob measure, since the OCaml cells have no sampling-measure capture. On OCaml the model solves only 8% (vs 43% and 52% on the Python benchmarks) and its per-token confidence is markedly lower (-0.114 vs -0.03 and -0.05, left panel): it is visibly less fluent. Yet confidence is a stronger pass/fail separator on OCaml (pooled AUROC 0.82 vs 0.66 on DS-1000, right panel): when the model is out of its depth it emits diffusely low-probability tokens, so the rare successes stand out sharply from the failures. Unfamiliarity hurts accuracy but sharpens the confidence signal. (One model, one temperature, raw measure, so read this as a single clean cross-language slice.)

Data

samuki-hf/thinking-rollouts: logprobs_sampling/domain=<d>/model=<tag>/temp=0.6/ (both measures per row, warp diagnostics included); logprobs/ holds the raw measure for every cell and temperature.