DS-1000, posterior accuracy ŝ & validity rate Z of the no-error constraint

How accurate is ideal constrained generation? For each problem, the rollouts that run without error are exact samples from the validity-constrained posterior; ŝ is the fraction of them that are correct, averaged one-problem-one-vote over the benchmark (0/0 := 0), together with the fraction of problems with no valid rollout and the validity rate Z. 10 models × 6 temperatures, N = 100 rollouts per problem (1 at t = 0), potential DS1000RuntimeNoErrorPotential, Matplotlib excluded (image-graded). This page previously reported the pooled ratio Ŝ = ∑correct/∑valid as its headline (“skyline”); see Definitions for why the macro estimate is the comparable reference and below for the pooled numbers.
* Qwen2.5-Coder 7B, DeepSeek-Coder: externally ingested (# SOLUTION START/END convention).

Definitions

For each model and temperature (DS-1000, Matplotlib excluded) we report ŝ with a 95% bootstrap CI over problems (2000 resamples), the fraction of problems with no valid rollout, the validity rate Z, and the pooled Ŝ for reference.

posterior accuracy ŝ against temperature

ŝ = mean_x Pr(correct | valid, x), 0/0 := 0 — the accuracy ideal constrained generation reaches on a new problem. It falls with temperature as the target π dilutes.

ŝ by model and temperature

Each cell: ŝ, its 95% bootstrap CI, and how many problems had no valid rollout among the N sampled (these enter ŝ as 0). Greedy t=0 has N=1, so its zero-valid count is simply the number of erroring greedy answers. Browse the underlying rollouts in the explorer.

problems with no valid rollout

Fraction of problems where none of the N rollouts passed the constraint. These are the problems rejection sampling (and any sampler that only re-weights p₀) cannot solve at this budget; they contribute 0 to ŝ and bound its downward bias.

posterior vs prior: what conditioning on validity buys

One dot per model at the selected temperature (optionally within one library). The diagonal is P(V) = 1; a dot’s height above it is what enforcing the constraint buys.

Conditioning on validity: the completion runs without a runtime or syntax error (an AssertionError still counts as valid) — the potential DS1000RuntimeNoErrorPotential

x is the prior pass@1: the accuracy as sampled. y is the macro posterior accuracy ŝ = mean_x Pr(correct | valid, x) (0/0 := 0): the accuracy of a sampler that draws exactly from the posterior the validity notion above defines, one problem one vote. Height above the diagonal is the gain; error bars are 95% bootstrap CIs (2000 resamples of the problem set) on both axes. Hovering a dot also shows two reference lines: the ray from the origin (its slope is the dot’s ŝ/pass@1 ratio) and the line through (1,1) (slope = the surviving-failure fraction).

Z, the validity rate (normalizing constant)

Z is the proportion of rollouts that obey the constraint (run without error), Z = meanx Z(x) — the potential’s normalizing mass under the sampling distribution. Z collapses toward t=1.0 as more rollouts error out.

pooled Ŝ (secondary)

Ŝ = ∑ correct / ∑ valid = pass@1 / Z, the page’s former headline. It estimates 𝔼[Z s]/𝔼[Z], not 𝔼[s]; the gap to ŝ above is Cov(Z, s)/𝔼[Z] plus the zero-valid problems. Shown for continuity and because it factorises pass@1 exactly.

Qwen3-Coder-30B (BF16) and prompt engineering

Four new columns for Qwen/Qwen3-Coder-30B-A3B-Instruct (MoE, 30.5B total / 3.3B active): the official FP8 release under the raw-completion sweep protocol and under its model-card serving (chat template, t=0.7, top_p 0.8, top_k 20, repetition penalty 1.05), and the unquantized BF16 model under model-card serving with the bare prompt and with a few-shot prompt. 100 samples × 1000 problems each; graded by the standard pipeline (official DS-1000 harness + no-runtime-error potential; chat answers extracted as the last fenced block). Hub: model=qwen3-coder-30b-a3b-*. All numbers below are non-Matplotlib macro.

FP8 is measurement-free. The FP8 and BF16 bare-prompt cells agree on every metric (ŝ 0.294 vs 0.305; pass@1 0.213 vs 0.209) — the official FP8 quantization does not move DS-1000 results.

Format, not sampling. Model-card serving changes two things at once — the answer format (chat template) and the sampler (t=0.7, top_p 0.8, top_k 20, repetition penalty 1.05). The third row separates them: keeping raw completion and switching only the sampler moves pass@1 by +0.008 (0.372 → 0.381), while switching only the format at that same sampler moves it by -0.167 (0.381 → 0.213). The card cell’s deficit is answer-format, essentially none of it sampling. The card sampler does visibly concentrate, though, at unchanged format: pass@100 0.725 → 0.638 — the same coverage–concentration trade the few-shot prompt shows below, here from truncation (top_p 0.8 ∩ top_k 20) alone.

Raw completion vs model-card serving

The two protocols put the same official problem text in front of the model in different ways, and the model answers in correspondingly different shapes:

Raw completion (the sweep protocol): the problem is fed as plain text to continue — no chat template — and generation stops at </code> / # SOLUTION END. The official prompt already ends inside the solution slot, so the model emits just the missing code, in DS-1000's native format. A typical complete answer:

result = df.iloc[List]

Model-card serving: the same problem is the user message of the chat template, sampled with the card's settings (t=0.7, top_p 0.8, top_k 20, repetition penalty 1.05). The instruct model answers conversationally; the graded answer must then be extracted (last fenced block). A typical answer to the same problem (abridged, real rollout):

Looking at this problem, I need to reorder the DataFrame rows based on a given list of indices. [...]

BEGIN SOLUTION
```python
result = df.iloc[List]
```

Wait, let me re-read the problem more carefully. [...]

Alternative approaches that would also work:

1. Using `np.nditer()`:
```python
...

That output style is where the bare-prompt cells lose their accuracy: the model often gives the right answer first and then helpfully appends alternatives or a self-correction, so a fixed block-picking rule sometimes grades a snippet that runs (valid) but is not the committed answer — deflating P(correct | valid) at identical validity. Z barely differs between the two modes (0.716 vs 0.700); ŝ is where they diverge (0.480 vs 0.294).

The prompt variants screened

Each variant wraps the official problem before the chat template is applied (expert is the exception: raw completion with a header). Verbatim templates — {prompt} stands for the official DS-1000 problem:

You are completing Python data-science code. The following is a DS-1000 problem: the code after BEGIN SOLUTION must replace the missing part.

{prompt}

Output ONLY the missing completion code in a single ```python block. No explanation, no imports already present in the problem, no surrounding code — exactly what belongs at the BEGIN SOLUTION slot.

Example problem:
Problem:
Given a pandas DataFrame df with a column 'x', add a column 'y' equal to twice 'x'.
A:
<code>
import pandas as pd
df = pd.DataFrame({'x': [1, 2]})
</code>
BEGIN SOLUTION
<code>
Example answer:
```python
df['y'] = 2 * df['x']
```

Now the real problem.

You are completing Python data-science code. The following is a DS-1000 problem: the code after BEGIN SOLUTION must replace the missing part.

{prompt}

Output ONLY the missing completion code in a single ```python block. No explanation, no imports already present in the problem, no surrounding code — exactly what belongs at the BEGIN SOLUTION slot.

{prompt}

First state in at most two sentences what the completion must do, then give the final completion code in a single ```python block.

# Expert data-science reference solution (correct, minimal, idiomatic)
# Passes all hidden tests.
{prompt}

The few-shot prompt

A screen over four prompt variants (150 stratified problems × 25 samples, FP8) selected a prompt with one worked format example plus an instruction to answer in a single fenced block. The example is synthetic (not a DS-1000 problem, so nothing evaluated leaks into the prompt) and trivial by design — it demonstrates the answer shape, not task knowledge:

Example problem:
Problem:
Given a pandas DataFrame df with a column 'x', add a column 'y' equal to twice 'x'.
A:
<code>
import pandas as pd
df = pd.DataFrame({'x': [1, 2]})
</code>
BEGIN SOLUTION
<code>
Example answer:
```python
df['y'] = 2 * df['x']
```

Now the real problem.

You are completing Python data-science code. The following is a DS-1000 problem: the code after BEGIN SOLUTION must replace the missing part.

<official DS-1000 problem>

Output ONLY the missing completion code in a single ```python block. No explanation, no imports already present in the problem, no surrounding code — exactly what belongs at the BEGIN SOLUTION slot.

Screen results (a plan-then-code variant hurts; the raw-completion 'expert header' variant ties but is not model-card serving):

How it changes performance

pass@1 more than doubles (0.209 → 0.490, 2.3×) and ŝ rises 0.305 → 0.516. Two mechanisms: (1) extraction — bare chat answers bury the solution among prose, self-corrections and alternative snippets, so any fixed fenced-block heuristic sometimes grades the wrong block (visible as correct>valid disagreements in the bare-prompt cells); the single-block instruction makes extraction unambiguous. (2) commitment — answers shrink to a median ~50 tokens and concentrate on the model's best guess.

The concentration trade-off. pass@100 does not improve (0.587 → 0.592, vs 0.725 under the diverse raw-completion protocol), and the number of problems with no valid sample in 100 grows (33 → 131 of 845): the engineered prompt raises the floor without extending coverage — it nearly pre-collapses the sample distribution onto one answer. Selection-by-validity (best-of-K, rejection sampling) therefore has far less headroom on a well-prompted model (ŝ − pass@1 ≈ +0.03) than on a bare one (≈ +0.10), and the two serving styles bracket the coverage–concentration spectrum the posterior-conditioning story lives on.

By library (BF16)

Generation: smc_learning/scripts/analysis/gen_ds1000_rollouts.py (sweep-identical official prompts; --chat card serving; --wrap fewshot); scoring: the canonical re-score pipeline (rescore_ds1000_cell.py --postprocess chat) in the DS-1000 environment.yml env.

by library, choose a model

ŝ and Z broken down by library for a single model. Use the selector to switch models; the two plots and the table below update together. Shaded bands are 95% bootstrap CIs (2000 resamples of that library’s problems).

ŝ (macro), zero-valid fraction, Z and pooled Ŝ by library and temperature (selected model):