PCA of the activations the correctness probe reads (Marks & Tegmark, arXiv 2310.06824). Hover for the rollout; drag to zoom.
Switch the colouring to see what the projection does and does not encode, and step through the layers to see where structure appears.
The 3D panel adds PC3 to the same projection and shares the controls above. It needs WebGL; without it (some remote desktops and managed browsers) a note appears here instead, and the 2D view above shows the same data.
Because the deployed probe mean-pools hidden states before its logistic head, its logit decomposes exactly: logit = (1/T) Σt v·ht + const. Each token position therefore carries a definite, additive share of the decision. This is an algebraic identity of the deployed model, not a saliency heuristic: the per-token shares below sum back to the probe's actual output (the reconstruction line above each rollout verifies it). Tokens are tinted by their contribution, positive toward “passes”, negative toward “fails”; sixteen rollouts per model, four from each behaviour class (true and false positives and negatives), so both correct and mistaken decisions can be inspected. Two findings to look for. First, expand the prompt box: roughly half of the probe's evidence mass sits in the prompt tokens, before the model has written anything, which is the token-level view of R151's result that 81 to 91% of pooled AUROC is problem difficulty; the probe partly recognises which problem it is looking at. Second, within the completion the strongest contributions concentrate on I/O idiom and mutability tokens (Scanf/read_line choices, refs and loops), the features R156 identified, rather than on the algorithmic core of the program.
Here the probe's weight vector is treated as a geometric object in activation space. Directions are refit with the deployed recipe (train split, the shipped C) and gated at cos > 0.98 against the shipped weights, so every direction analysed is faithful to what actually runs in the ladder. Three results. Left/middle: cosine similarity between per-layer directions within each model. The late layers share one direction family (high pairwise cosines), so the probe is finding a stable representation that persists across depth, not layer-local noise; early layers do not participate, matching the layer sweep. Right: the same comparison across checkpoints. No pair of directions between the untrained and distilled model exceeds |cos| 0.16, which is the geometric mechanism behind R149: fine-tuning rotates the correctness direction almost completely, so a probe reused on the other checkpoint scores at chance and every new model needs its own labelled rollouts. Below: at every layer the probe is nearly orthogonal to two natural reference directions, the class-mean difference θmm (the “mass-mean” direction connecting the average failing and average passing activation) and the top-10 PC subspace (the directions of dominant variance). Logistic regression under regularisation finds a margin direction, one that separates the classes where they are closest, not the direction along which the classes or the data vary most. That is why the PCA panels above cannot display what the probe reads (cos(probe, PC1) = 0.021 untrained, 0.009 distilled), why mass-mean probing underperforms here (R152), and why interpretability claims about “the correctness feature” cannot be read off this vector alone.
With raw activations, colouring by problem solvable separates; colouring by correctness barely does. Problem identity alone explains R² 0.81 (untrained) and 0.88 (distilled) of the probe's output (R151).
With problem mean subtracted, difficulty is removed and what remains is what a best-of-k selector sees. The usable signal is small: within-problem AUROC 0.66 (untrained) / 0.71 (distilled probe, trained on that checkpoint's own rollouts).
Every probe architecture the campaign has, on the same rollouts and the same split: trained on 401 problems, configuration chosen on 56 validation problems, reported on the 150 held-out problems. Within-problem AUROC exists only on class-mixed problems, 37 of the 150 on the base cell and 52 on the distilled one, and nothing is selected on the reported slice.
| family | pooling | base 8B rollouts | distilled rollouts | ||
|---|---|---|---|---|---|
| val-selected | spread over configs | val-selected | spread | ||
| MLP 512–256 | last | 0.721 | 0.428 – 0.721 | 0.715 | 0.453 – 0.715 |
| MLP 512 | last | 0.665 | 0.424 – 0.681 | 0.691 | 0.427 – 0.709 |
| MLP 512–256 | mean | 0.632 | 0.437 – 0.647 | 0.638 | 0.490 – 0.663 |
| MLP 512 | mean | 0.643 | 0.443 – 0.643 | 0.605 | 0.463 – 0.644 |
| linear logistic | last | 0.551 | 0.504 – 0.726 | 0.666 | 0.493 – 0.716 |
| linear logistic | mean | 0.610 | 0.600 – 0.728 | 0.736 | 0.530 – 0.750 |
| stacker over 37 layers | last | 0.710 | no layer to pick | 0.660 | – |
| stacker over 37 layers | mean | 0.653 | – | 0.667 | – |
| concat, 37 x 4096 features | last | 0.698 | – | 0.656 | – |
| concat, 37 x 4096 features | mean | 0.673 | – | 0.746 | – |
| attention (attn, attn_mh, xattn, xattn_mh) | learned | 0.528 | 0.450 – 0.710 | 0.688 | 0.503 – 0.738 |
Choosing the layer costs more than choosing the architecture. The linear probe's validation pick on the base cell scores 0.551 against its own spread's 0.726, a 0.175 selection loss, larger than the gap between any two families; the whole-depth readouts, which have no layer to choose, reach 0.710 and 0.698 from the same activations. No architecture wins overall, and with every family inside a band about 0.10 wide on 37 and 52 mixed problems, the ordering is below the resolution of this data.
A different population, so a separate table: the k=50 thinking pools over the same 150 problems, 94 of them class-mixed. The probe is cross-fit by problem on the pool it selects over, since probes do not transfer across checkpoints.
| rung | distilled 8B | distilled + GRPO | what it costs |
|---|---|---|---|
| random pick | 0.3256 | 0.3596 | nothing |
| probe alone | 0.4133 | 0.4667 | one forward pass |
| compile gate | 0.4344 | 0.4502 | ocamlc per sample |
| compile + probe | 0.4933 | 0.5200 | both |
| public tests | 0.6018 | 0.6162 | execution + sandbox |
| public + probe | 0.6067 | 0.6200 | both |
| oracle | 0.6733 | 0.6867 | – |
| within-problem AUROC of the cross-fit probe: 0.620 and 0.657, on 94 class-mixed problems | |||
Where there is no sandbox the probe earns its place; where there is one it adds nothing. Alone it beats a random pick by 8 to 11 points and, on the GRPO arm, beats the compile gate outright with no toolchain. On top of the public-test filter it adds +0.0049 (CI [−0.0102, +0.0200]) and +0.0038 (CI [−0.0181, +0.0272]), both indistinguishable from zero, superseding the earlier k=16 estimate of +0.0156 measured on 37 mixed problems.