LiveCodeBench: the fresh window (2410–2504)

The tool-use cells (Qwen3 8B/14B, think vs nothink, run_code tool loop, 100 samples per problem at t=0.6) re-run on 341 problems published between October 2024 and April 2025 — after the models' training data — against the original window's 714 problems (May 2023–September 2024). Same prompts, same decoding, same grading; only problem freshness changes.

pass@k, original vs fresh window

The freshness drop

Every configuration loses 37–43% of its pass@1 relative to the original window. The uniformity across sizes and reasoning modes reads as broad familiarity with the older problems rather than something one configuration exploits; era-related difficulty drift is not controlled here.

What the fresh window contains

The fresh window is hard-skewed (45% hard vs 28% originally) with a similar platform mix (AtCoder + LeetCode) and similar statement lengths, so composition is a real but bounded factor.

Old vs new, within each difficulty tier

The direct contamination test: pass@1 on the original vs fresh window computed separately inside each announced difficulty tier, per cell (never pooling arms or sizes). A contaminated original window would show up as inflated original accuracy on hard problems, since memorization lets a model “solve” hard items it otherwise could not, so the original-minus-fresh gap would concentrate there. It does not. Even the strongest cell solves only 54% of original hard problems, nowhere near the ceiling recitation would produce, and the decline is concentrated in the medium tier (and, for the think arm, medium and hard alike), while easy sits at the ceiling and hard-nothink at the floor. That is the signature of a genuine capability drop on unseen problems, not recited hard ones, and it agrees with the Min-K% no-recitation result on the logprobs page. (Easy tiers near 1.0 and hard-nothink near 0.0 have little room to move, so the mid-range carries the cleanest signal.)

Decomposing the drop: mix vs within-tier

Shift-share decomposition of each cell's pass@1 drop into the harder tier mix vs performance decline inside each tier. The counterfactual makes it concrete: give 8B-think the fresh window's harder mix but keep its original per-tier ability, and it would score 0.693 — the mix explains the move from 0.770 only that far; the observed 0.460 requires the within-tier collapse (medium 0.816→0.503, hard 0.446→0.164). For thinking cells about a quarter of the drop is composition, 3/4 within-tier; for nothink cells the split is roughly even. Never-solved problems (0/100 attempts) double for nothink cells (29→54%) and triple for think cells (10→25–32%).

The decline starts before the window boundary

Monthly solve rate across BOTH windows. The drop does not occur at the window boundary (red): it starts around July 2024 (violet), three months inside the original window, and at constant difficulty mix — April and July 2024 have nearly identical hard shares (0.32) but solve rates of 0.83 vs 0.63. This places the model's effective knowledge boundary around mid-2024: the "original" window is itself part-clean, and the honest familiar-vs-unseen comparison is pre-2024-07 (pass@1 ≈ 0.82) against everything after (≈ 0.45–0.55), a larger gap than the window-vs-window number. Difficulty labels drifting harder over 2024–25 (grey line) compounds but does not explain the step.

Tool iteration across windows

Samples that used more run_code rounds pass less often in both windows (selection: harder problems trigger iteration). The fresh window shifts mass toward more rounds at every level of the 8B-think cell.

No recitation signal in the logprobs

If the original window's solutions were partially memorised, the model should assign its own CORRECT solutions higher likelihood there. It does not: the per-token likelihood of passing final answers is statistically indistinguishable across windows (deltas of a few thousandths of a nat, mixed sign). The freshness drop reflects harder unseen problem solving, not lost recitation; era-difficulty drift remains uncontrolled.

The thinking advantage survives freshness

Paired per-instance bootstrap (shared problems, so common difficulty cancels). Thinking roughly doubles pass@1 on both windows: the lift is not a contamination artifact. On fresh problems, 8B-think matches 8B-nothink's performance on the original window.

How failures change on fresh problems

Execution-grounded failure modes (per-test harness verdicts, dominant class per failing sample). The two arms fail differently: nothink failures are dominated by running-but-wrong code, and freshness amplifies exactly that (wrong answers grow from 32% to 53% of ALL samples). Think failures are dominated by producing no runnable final answer at all — reasoning that never converges to a program — and that failure mode expands from 16% to 39% of all samples on fresh problems, while its crash/timeout modes shrink. The LLM bug-theme panels (the Failure themes section below) add the semantic layer over these mechanical verdicts. The dominant think-arm mode (no runnable answer) is dissected on the reasoning-traces page: 89% of those samples are cut off mid-reasoning by the token budget, on the same problem failing traces think ~10k characters longer than passing ones, and the truncated tail is often a verbatim repetition loop rather than slow reasoning — so budget increases alone recover less than the 9–11 point ceiling that converting these samples implies.

Failure themes: tool vs no-tools

Sonnet-panel classification of failing fresh-window rollouts into the 7-theme LiveCodeBench bug taxonomy (one failing k=1 sample per instance), the tool arm (samuki-hf/tool-use) against the plain no-tools arm (samuki-hf/thinking-rollouts). The two arms fail semantically differently: the plain arm is almost entirely logic_flaw (compiles and runs, wrong answer), while giving the model a run_code tool shifts a substantial slice of failures to surface causes the tool surfaces — output_format and syntax_typo — the errors a quick execution would have caught but the model failed to fix within its three rounds. Within logic_flaw, wrong_algorithm dominates both arms: on fresh problems the model most often reaches for a wrong approach, not a small slip.

Within logic_flaw: the fine-grained wrong-answer cause

The 7-theme pass puts most fresh-window failures in logic_flaw (compiles, runs, wrong answer); the wrong-answer sub-taxonomy resolves what actually went wrong.

Tool behaviour

run_code usage per sample (up to 3 rounds).

Does 14B-think decide not to use the tool? It never deliberates — and the avoidance is LCB-specific

On LiveCodeBench (both windows) 14B-think calls the tool in ~1% of samples while its nothink sibling calls at 0.8+ rounds under the identical prompt. But this is not a blanket thinking-mode prior: the same model calls the tool on 34% of DS-1000 samples and 20% of OCaml samples. The gradient tracks how mentally simulable the domain is — pinned-pandas execution output is hard to reason out (both models test heavily), OCaml compilation is moderately so, and pure algorithmic Python is where 14B trusts its own reasoning entirely while 8B keeps testing. Nowhere is this deliberated in the traces: only ~10–15% of thinking spans mention the tool at all and genuine explicit declines are far below the regex upper bound (most matches are algorithmic phrasing about the problem). The rare tool-aware moment reads like: "But without running it, it's hard to say. However, the code's logic should correctly compute the minimal…" — awareness, then confidence overrides. An implicit, capability- and domain-dependent cost-benefit, expressed as behavior rather than decided in words; on fresh LCB problems it forgoes insurance that tool iteration demonstrably provides. A second truncation effect is visible in the same table: 8B-think's call rate falls from 52% (original) to 38% (fresh) because budget-truncated samples die mid-reasoning before ever reaching a tool call.

Data

Rollouts, per-test execution records, tool transcripts and teacher-forced logprobs for all cells: samuki-hf/tool-use, domains livecodebench and livecodebench-2410.