LiveCodeBench, posterior accuracy ŝ & validity rate Z of two constraints
How accurate is ideal constrained generation? For each problem, the rollouts a potential accepts
are exact samples from the constrained posterior; ŝ is the fraction of them that are correct, averaged
one-problem-one-vote over the benchmark (0/0 := 0), with the fraction of problems with no accepted rollout and
the validity rate Z. Two potentials: no-runtime-error (LCBRuntimeNoErrorPotential) and public-test
(LCBPublicTestPotential), 12 models × 6 temperatures (71 cells, ~4.2M rollouts). This page
previously headlined the pooled ratio Ŝ = ∑correct/∑valid (“skyline”); see
Definitions.
Two potentials, two constraints. Each rollout y for problem x defines a validity indicator φ(y); rejection sampling targets the posterior π(·|x) ∝ p0(·|x)·φ(·) — sample from the model, keep the valid ones. Correct = the LCB evaluator’s verdict over all tests (public + private).
Posterior accuracy over the benchmark (headline). Treating the benchmark as a sample of problems x ~ D, the number we want is the expected posterior accuracy of a new problem, s̄ := 𝔼x~D[ s(x) ], estimated by the macro estimator — one problem, one vote — with 0/0 := 0 for problems none of whose N rollouts is valid:
ŝ is consistent for s̄. Each per-problem ratio is unbiased given at least one valid sample, so at finite N the only bias is the 0/0 := 0 convention, which pulls ŝ down by at most the fraction of zero-valid problems, reported next to every ŝ. (Dropping the zero-valid problems instead is biased too — a ratio of two unbiased estimators — and changes the population to “problems this model can make valid”; we show it only in the tables as ŝ+.) A fine-tuned or approximate sampler is scored by its macro pass@1, so ŝ is the comparable reference.
No-error potential (LCBRuntimeNoErrorPotential): φ = the code runs on the public inputs with no runtime error / timeout. A program that runs but answers wrongly is still valid.
Public-test potential (LCBPublicTestPotential): the strict constraint is “passes all public example tests”. Because the potential is soft (finite penalty per failed public test), every public-test section is indexed by k = the max number of failing public tests allowed: φk=1[n_failed≤k]. k=0 is the strict potential; raising k relaxes it, and at k=max#public-tests every rollout is accepted (Z=1, ŝ=pass@1). A correct solution passes every public test, so correct ⇒ φk for every k.
a validity-weighted average: every valid rollout gets one vote, so problems the model already handles well (high Z) dominate, and problems with Z(x) = 0 have no weight at all. Because Z and s are positively correlated (problems that error more are also wrong more), Ŝ sits above s̄ — not through finite-sample bias but because it estimates the wrong thing. Ŝ is kept below as a secondary number because pass@1 = Z × Ŝ holds exactly on pooled estimates.
Why report this? ŝ is the accuracy of ideal constrained generation: if you could sample exactly from π (enforce the constraint and nothing else), this is how often the result would be correct — a reference for judging approximate inference methods (importance sampling, SMC, fine-tuning) that target the same π. Formerly called the “skyline” on this site. Shaded bands are 95% bootstrap CIs (2000 resamples of the problem set).
LiveCodeBench, 12 models × 6 temperatures (71 cells, ~4.2M rollouts, N = 100 rollouts per problem, 1 at t = 0); both potentials run literally (forkserver, 6 s per-test timeout). The instance with no public tests is excluded. Two cells (Llama 3.2 1B Instruct, t = 0.8 and 1.0) have pooled numbers only.
Posterior vs prior — what conditioning on validity buys
prior pass@1 vs macro posterior accuracy ŝ, per model
Conditioning on validity: the sample's code runs on the public inputs with no runtime error or timeout (wrong answers still count as valid)— the potential LCBRuntimeNoErrorPotential
x is the prior pass@1: the accuracy as sampled. y is the macro posterior accuracy ŝ = meanx Pr(correct | valid, x) (0/0 := 0): the accuracy of a sampler that draws exactly from the posterior the validity notion above defines, one problem one vote. Height above the diagonal is the gain; error bars are 95% bootstrap CIs (2000 resamples of the problem set) on both axes. Hovering a dot also shows two reference lines: the ray from the origin (its slope is the dot’s ŝ/pass@1 ratio) and the line through (1,1) (slope = the surviving-failure fraction).
Part 1 — no-error potential
posterior accuracy ŝ against temperature
ŝ = meanx Pr(correct | runs without error, x), 0/0 := 0. Wrong-but-running answers dilute it.
ŝ by model and temperature
Each cell: ŝ, its 95% bootstrap CI, and how many problems had no valid rollout among the N sampled (these enter ŝ as 0).
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108 [0.086, 0.132] 151/713 zero-valid
0.110 [0.090, 0.131] 12/713 zero-valid
0.099 [0.081, 0.118] 2/713 zero-valid
0.087 [0.071, 0.103] 1/713 zero-valid
0.066 [0.054, 0.079] 0/713 zero-valid
0.043 [0.034, 0.053] 1/713 zero-valid
Llama-3 8B Instruct
0.164 [0.139, 0.192] 129/713 zero-valid
0.166 [0.141, 0.194] 27/713 zero-valid
0.164 [0.140, 0.190] 13/713 zero-valid
0.159 [0.137, 0.185] 6/713 zero-valid
0.151 [0.130, 0.175] 2/713 zero-valid
0.142 [0.121, 0.164] 1/713 zero-valid
Llama-3.1 8B
0.102 [0.081, 0.126] 159/713 zero-valid
0.106 [0.086, 0.128] 17/713 zero-valid
0.099 [0.081, 0.118] 2/713 zero-valid
0.087 [0.071, 0.104] 0/713 zero-valid
0.068 [0.055, 0.082] 0/713 zero-valid
0.045 [0.036, 0.056] 0/713 zero-valid
Llama-3.1 8B Instruct
0.191 [0.163, 0.222] 114/713 zero-valid
0.197 [0.172, 0.225] 5/713 zero-valid
0.196 [0.172, 0.221] 1/713 zero-valid
0.187 [0.165, 0.212] 3/713 zero-valid
0.172 [0.150, 0.195] 1/713 zero-valid
0.149 [0.129, 0.170] 1/713 zero-valid
Llama-3.2 1B
0.015 [0.007, 0.025] 313/713 zero-valid
0.015 [0.008, 0.023] 33/713 zero-valid
0.014 [0.009, 0.020] 4/713 zero-valid
0.010 [0.006, 0.014] 0/713 zero-valid
0.007 [0.004, 0.010] 0/713 zero-valid
0.003 [0.002, 0.005] 0/713 zero-valid
Llama-3.2 1B Instruct
0.062 [0.045, 0.080] 259/713 zero-valid
0.069 [0.054, 0.085] 13/713 zero-valid
0.064 [0.051, 0.079] 5/713 zero-valid
–
–
–
Qwen2.5-Coder 3B
0.220 [0.191, 0.251] 131/713 zero-valid
0.228 [0.200, 0.257] 14/713 zero-valid
0.224 [0.198, 0.250] 3/713 zero-valid
0.212 [0.188, 0.237] 0/713 zero-valid
0.182 [0.162, 0.204] 0/713 zero-valid
0.134 [0.117, 0.152] 0/713 zero-valid
Qwen2.5-Coder 3B Instruct
0.238 [0.206, 0.272] 137/713 zero-valid
0.250 [0.222, 0.280] 9/713 zero-valid
0.251 [0.225, 0.280] 4/713 zero-valid
0.242 [0.216, 0.269] 1/713 zero-valid
0.231 [0.207, 0.257] 1/713 zero-valid
0.210 [0.186, 0.235] 0/713 zero-valid
Qwen2.5-Coder 7B
0.279 [0.245, 0.313] 93/713 zero-valid
0.287 [0.256, 0.318] 5/713 zero-valid
0.283 [0.255, 0.312] 0/713 zero-valid
0.263 [0.236, 0.289] 0/713 zero-valid
0.228 [0.205, 0.252] 0/713 zero-valid
0.172 [0.153, 0.191] 0/713 zero-valid
Qwen2.5-Coder 7B Instruct
0.349 [0.314, 0.386] 76/713 zero-valid
0.354 [0.321, 0.387] 13/713 zero-valid
0.350 [0.318, 0.382] 6/713 zero-valid
0.346 [0.315, 0.377] 3/713 zero-valid
0.337 [0.308, 0.367] 2/713 zero-valid
0.327 [0.298, 0.356] 0/713 zero-valid
DeepSeek-Coder 6.7B
0.163 [0.136, 0.191] 86/713 zero-valid
0.168 [0.143, 0.194] 15/713 zero-valid
0.163 [0.139, 0.187] 2/713 zero-valid
0.153 [0.131, 0.176] 1/713 zero-valid
0.135 [0.116, 0.156] 1/713 zero-valid
0.106 [0.090, 0.123] 2/713 zero-valid
DeepSeek-Coder 6.7B Instruct
0.201 [0.171, 0.231] 110/713 zero-valid
0.207 [0.179, 0.234] 9/713 zero-valid
0.201 [0.175, 0.227] 5/713 zero-valid
0.199 [0.174, 0.225] 2/713 zero-valid
0.194 [0.169, 0.219] 0/713 zero-valid
0.184 [0.161, 0.209] 0/713 zero-valid
problems with no valid rollout
Fraction of problems where none of the N rollouts ran without error. They contribute 0 to ŝ and bound its downward bias.
Z, the validity rate (normalizing constant)
Z = proportion of rollouts that run without error; falls toward t=1.0.
pooled Ŝ (secondary)
Ŝ = ∑correct / ∑valid = pass@1 / Z, the page’s former headline; estimates 𝔼[Z s]/𝔼[Z], not 𝔼[s].
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.138
0.131
0.119
0.106
0.085
0.063
Llama-3 8B Instruct
0.202
0.199
0.195
0.189
0.183
0.180
Llama-3.1 8B
0.132
0.127
0.119
0.106
0.088
0.068
Llama-3.1 8B Instruct
0.229
0.224
0.222
0.216
0.206
0.203
Llama-3.2 1B
0.027
0.022
0.018
0.013
0.009
0.005
Llama-3.2 1B Instruct
0.098
0.095
0.088
–
0.067
0.054
Qwen2.5-Coder 3B
0.273
0.273
0.269
0.257
0.227
0.190
Qwen2.5-Coder 3B Instruct
0.297
0.300
0.298
0.289
0.282
0.273
Qwen2.5-Coder 7B
0.321
0.319
0.315
0.299
0.271
0.240
Qwen2.5-Coder 7B Instruct
0.399
0.394
0.390
0.387
0.381
0.377
DeepSeek-Coder 6.7B
0.185
0.187
0.182
0.173
0.159
0.139
DeepSeek-Coder 6.7B Instruct
0.242
0.234
0.227
0.227
0.224
0.220
by difficulty, choose a model
No-error ŝ and Z split by problem difficulty for a single model. Shaded bands are 95% bootstrap CIs over that difficulty’s problems.
ŝ, zero-valid fraction, Z and pooled Ŝ by difficulty and temperature (selected model):
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.298
0.301
0.274
0.242
0.185
0.122
zero-valid frac.
0.122
0.008
0.000
0.000
0.000
0.000
Z
0.878
0.875
0.863
0.811
0.721
0.494
Ŝ (pooled)
0.340
0.323
0.294
0.260
0.199
0.136
medium
ŝ (macro)
0.022
0.024
0.019
0.015
0.010
0.006
zero-valid frac.
0.224
0.022
0.004
0.000
0.000
0.004
Z
0.773
0.786
0.770
0.718
0.612
0.372
Ŝ (pooled)
0.028
0.029
0.022
0.018
0.012
0.007
hard
ŝ (macro)
0.000
0.000
0.001
0.000
0.000
0.000
zero-valid frac.
0.303
0.020
0.005
0.005
0.000
0.000
Z
0.682
0.681
0.651
0.571
0.440
0.232
Ŝ (pooled)
0.000
0.000
0.001
0.000
0.001
0.000
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.429
0.443
0.438
0.427
0.406
0.382
zero-valid frac.
0.097
0.025
0.008
0.000
0.000
0.000
Z
0.899
0.892
0.879
0.855
0.821
0.769
Ŝ (pooled)
0.477
0.483
0.474
0.460
0.441
0.417
medium
ŝ (macro)
0.047
0.044
0.043
0.042
0.039
0.035
zero-valid frac.
0.155
0.029
0.018
0.007
0.004
0.000
Z
0.845
0.823
0.806
0.781
0.740
0.651
Ŝ (pooled)
0.056
0.049
0.049
0.047
0.044
0.041
hard
ŝ (macro)
0.010
0.003
0.002
0.002
0.002
0.002
zero-valid frac.
0.318
0.066
0.030
0.020
0.005
0.005
Z
0.667
0.676
0.676
0.659
0.614
0.523
Ŝ (pooled)
0.015
0.005
0.002
0.002
0.002
0.003
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.286
0.303
0.283
0.247
0.195
0.131
zero-valid frac.
0.134
0.013
0.000
0.000
0.000
0.000
Z
0.866
0.875
0.872
0.833
0.743
0.527
Ŝ (pooled)
0.330
0.328
0.303
0.264
0.211
0.148
medium
ŝ (macro)
0.018
0.014
0.013
0.011
0.008
0.004
zero-valid frac.
0.220
0.014
0.004
0.000
0.000
0.000
Z
0.780
0.790
0.785
0.739
0.630
0.385
Ŝ (pooled)
0.023
0.014
0.014
0.013
0.009
0.005
hard
ŝ (macro)
0.000
0.000
0.000
0.000
0.000
0.000
zero-valid frac.
0.333
0.051
0.005
0.000
0.000
0.000
Z
0.662
0.685
0.657
0.596
0.463
0.244
Ŝ (pooled)
0.000
0.000
0.000
0.000
0.000
0.000
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.475
0.498
0.503
0.484
0.448
0.396
zero-valid frac.
0.063
0.008
0.004
0.004
0.004
0.000
Z
0.933
0.929
0.926
0.914
0.892
0.830
Ŝ (pooled)
0.509
0.517
0.519
0.501
0.468
0.424
medium
ŝ (macro)
0.072
0.071
0.063
0.060
0.051
0.038
zero-valid frac.
0.134
0.004
0.000
0.004
0.000
0.000
Z
0.863
0.855
0.844
0.820
0.775
0.638
Ŝ (pooled)
0.084
0.076
0.069
0.066
0.055
0.046
hard
ŝ (macro)
0.015
0.013
0.011
0.009
0.008
0.005
zero-valid frac.
0.313
0.010
0.000
0.005
0.000
0.005
Z
0.677
0.724
0.717
0.689
0.606
0.422
Ŝ (pooled)
0.022
0.016
0.014
0.011
0.011
0.010
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.046
0.045
0.042
0.029
0.020
0.010
zero-valid frac.
0.412
0.034
0.008
0.000
0.000
0.000
Z
0.592
0.608
0.588
0.542
0.462
0.297
Ŝ (pooled)
0.078
0.060
0.050
0.033
0.024
0.012
medium
ŝ (macro)
0.000
0.000
0.000
0.001
0.000
0.000
zero-valid frac.
0.394
0.043
0.000
0.000
0.000
0.000
Z
0.607
0.593
0.571
0.532
0.456
0.285
Ŝ (pooled)
0.000
0.000
0.000
0.001
0.000
0.000
hard
ŝ (macro)
0.000
0.000
0.000
0.000
0.000
0.000
zero-valid frac.
0.535
0.066
0.010
0.000
0.000
0.000
Z
0.475
0.437
0.402
0.363
0.295
0.168
Ŝ (pooled)
0.000
0.000
0.000
0.000
0.000
0.000
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.168
0.186
0.175
–
–
–
zero-valid frac.
0.218
0.013
0.000
–
–
–
Z
0.773
0.788
0.765
–
0.635
0.455
Ŝ (pooled)
0.217
0.216
0.203
–
0.149
0.112
medium
ŝ (macro)
0.011
0.015
0.012
–
–
–
zero-valid frac.
0.347
0.011
0.007
–
–
–
Z
0.653
0.674
0.648
–
0.502
0.327
Ŝ (pooled)
0.017
0.018
0.015
–
0.009
0.005
hard
ŝ (macro)
0.005
0.004
0.003
–
–
–
zero-valid frac.
0.561
0.035
0.015
–
–
–
Z
0.434
0.487
0.476
–
0.355
0.195
Ŝ (pooled)
0.012
0.007
0.006
–
0.004
0.003
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.513
0.528
0.520
0.494
0.432
0.334
zero-valid frac.
0.071
0.004
0.004
0.000
0.000
0.000
Z
0.908
0.912
0.907
0.889
0.838
0.679
Ŝ (pooled)
0.565
0.565
0.553
0.525
0.459
0.367
medium
ŝ (macro)
0.119
0.125
0.122
0.115
0.093
0.056
zero-valid frac.
0.184
0.014
0.004
0.000
0.000
0.000
Z
0.816
0.820
0.813
0.780
0.711
0.504
Ŝ (pooled)
0.146
0.145
0.141
0.134
0.110
0.072
hard
ŝ (macro)
0.010
0.011
0.010
0.009
0.008
0.004
zero-valid frac.
0.318
0.045
0.005
0.000
0.000
0.000
Z
0.677
0.673
0.653
0.627
0.548
0.335
Ŝ (pooled)
0.015
0.016
0.016
0.013
0.012
0.006
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.538
0.560
0.564
0.550
0.532
0.498
zero-valid frac.
0.055
0.004
0.000
0.000
0.000
0.000
Z
0.945
0.924
0.922
0.915
0.900
0.866
Ŝ (pooled)
0.569
0.592
0.592
0.578
0.560
0.530
medium
ŝ (macro)
0.148
0.155
0.154
0.144
0.132
0.109
zero-valid frac.
0.177
0.011
0.004
0.000
0.004
0.000
Z
0.816
0.830
0.828
0.812
0.781
0.705
Ŝ (pooled)
0.181
0.176
0.172
0.162
0.153
0.134
hard
ŝ (macro)
0.005
0.010
0.010
0.009
0.008
0.005
zero-valid frac.
0.379
0.025
0.015
0.005
0.000
0.000
Z
0.611
0.627
0.633
0.625
0.591
0.495
Ŝ (pooled)
0.008
0.015
0.015
0.013
0.012
0.009
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.618
0.624
0.612
0.575
0.511
0.398
zero-valid frac.
0.046
0.008
0.000
0.000
0.000
0.000
Z
0.954
0.953
0.951
0.937
0.891
0.727
Ŝ (pooled)
0.648
0.644
0.627
0.591
0.528
0.427
medium
ŝ (macro)
0.181
0.189
0.188
0.169
0.139
0.095
zero-valid frac.
0.108
0.011
0.000
0.000
0.000
0.000
Z
0.892
0.906
0.890
0.853
0.777
0.540
Ŝ (pooled)
0.202
0.203
0.202
0.186
0.159
0.124
hard
ŝ (macro)
0.010
0.019
0.020
0.019
0.013
0.007
zero-valid frac.
0.263
0.000
0.000
0.000
0.000
0.000
Z
0.742
0.761
0.748
0.704
0.606
0.334
Ŝ (pooled)
0.014
0.022
0.024
0.023
0.018
0.014
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.723
0.714
0.706
0.699
0.688
0.676
zero-valid frac.
0.034
0.000
0.000
0.000
0.000
0.000
Z
0.954
0.970
0.969
0.964
0.959
0.945
Ŝ (pooled)
0.758
0.729
0.719
0.711
0.700
0.689
medium
ŝ (macro)
0.260
0.277
0.275
0.273
0.263
0.246
zero-valid frac.
0.087
0.018
0.011
0.004
0.000
0.000
Z
0.888
0.890
0.879
0.874
0.860
0.828
Ŝ (pooled)
0.293
0.299
0.298
0.297
0.288
0.274
hard
ŝ (macro)
0.025
0.027
0.026
0.023
0.020
0.021
zero-valid frac.
0.222
0.040
0.015
0.010
0.010
0.000
Z
0.763
0.762
0.757
0.742
0.724
0.673
Ŝ (pooled)
0.033
0.035
0.032
0.030
0.026
0.026
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.395
0.399
0.386
0.370
0.335
0.270
zero-valid frac.
0.042
0.013
0.004
0.004
0.004
0.004
Z
0.958
0.946
0.938
0.922
0.875
0.723
Ŝ (pooled)
0.412
0.414
0.400
0.385
0.352
0.296
medium
ŝ (macro)
0.076
0.085
0.083
0.073
0.057
0.040
zero-valid frac.
0.126
0.025
0.004
0.000
0.000
0.004
Z
0.877
0.868
0.859
0.833
0.766
0.577
Ŝ (pooled)
0.086
0.095
0.093
0.081
0.065
0.049
hard
ŝ (macro)
0.005
0.006
0.005
0.003
0.003
0.002
zero-valid frac.
0.207
0.025
0.000
0.000
0.000
0.000
Z
0.783
0.808
0.790
0.754
0.662
0.463
Ŝ (pooled)
0.006
0.007
0.006
0.004
0.004
0.002
difficulty
metric
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
easy
ŝ (macro)
0.458
0.480
0.479
0.480
0.470
0.454
zero-valid frac.
0.055
0.004
0.000
0.000
0.000
0.000
Z
0.941
0.948
0.945
0.935
0.910
0.869
Ŝ (pooled)
0.487
0.487
0.487
0.488
0.481
0.471
medium
ŝ (macro)
0.119
0.116
0.102
0.097
0.091
0.083
zero-valid frac.
0.188
0.011
0.004
0.000
0.000
0.000
Z
0.787
0.824
0.825
0.810
0.774
0.720
Ŝ (pooled)
0.151
0.134
0.116
0.108
0.102
0.093
hard
ŝ (macro)
0.005
0.006
0.006
0.005
0.005
0.003
zero-valid frac.
0.227
0.025
0.020
0.010
0.000
0.000
Z
0.752
0.756
0.749
0.726
0.684
0.612
Ŝ (pooled)
0.007
0.005
0.006
0.006
0.005
0.003
Part 2 — public-test potential
Pick the acceptance threshold k (max public-test failures allowed); every plot and table in this part updates to that k. k=0 is the strict “pass all public tests” potential.
posterior accuracy ŝ against temperature
ŝ = meanx Pr(correct | accepted at threshold k, x), 0/0 := 0. At k=0 this is the strict public-test posterior accuracy — much higher than the no-error one.
ŝ by model and temperature
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.187
0.214
0.228
0.219
0.184
Llama-3 8B Instruct
0.164
0.235
0.271
0.293
0.288
0.294
Llama-3.1 8B
0.102
0.188
0.221
0.235
0.223
0.191
Llama-3.1 8B Instruct
0.191
0.315
0.352
0.354
0.351
0.335
Llama-3.2 1B
0.015
0.043
0.064
0.070
0.058
0.037
Llama-3.2 1B Instruct
0.062
0.161
0.189
–
–
–
Qwen2.5-Coder 3B
0.220
0.341
0.378
0.396
0.383
0.355
Qwen2.5-Coder 3B Instruct
0.238
0.367
0.404
0.409
0.413
0.398
Qwen2.5-Coder 7B
0.279
0.402
0.455
0.454
0.449
0.398
Qwen2.5-Coder 7B Instruct
0.349
0.449
0.476
0.492
0.496
0.498
DeepSeek-Coder 6.7B
0.163
0.269
0.312
0.326
0.335
0.310
DeepSeek-Coder 6.7B Instruct
0.201
0.316
0.359
0.378
0.383
0.384
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.151
0.150
0.148
0.139
0.108
Llama-3 8B Instruct
0.164
0.199
0.214
0.216
0.212
0.207
Llama-3.1 8B
0.102
0.150
0.152
0.154
0.134
0.112
Llama-3.1 8B Instruct
0.191
0.245
0.257
0.257
0.247
0.227
Llama-3.2 1B
0.015
0.033
0.047
0.042
0.035
0.021
Llama-3.2 1B Instruct
0.062
0.113
0.117
–
–
–
Qwen2.5-Coder 3B
0.220
0.273
0.285
0.288
0.270
0.234
Qwen2.5-Coder 3B Instruct
0.238
0.305
0.313
0.312
0.307
0.289
Qwen2.5-Coder 7B
0.279
0.341
0.355
0.346
0.327
0.275
Qwen2.5-Coder 7B Instruct
0.349
0.396
0.407
0.410
0.408
0.397
DeepSeek-Coder 6.7B
0.163
0.214
0.228
0.227
0.221
0.191
DeepSeek-Coder 6.7B Instruct
0.201
0.257
0.268
0.277
0.275
0.267
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.111
0.102
0.091
0.072
0.048
Llama-3 8B Instruct
0.164
0.169
0.170
0.164
0.154
0.143
Llama-3.1 8B
0.102
0.108
0.103
0.093
0.073
0.052
Llama-3.1 8B Instruct
0.191
0.201
0.202
0.194
0.179
0.155
Llama-3.2 1B
0.015
0.022
0.021
0.018
0.014
0.008
Llama-3.2 1B Instruct
0.062
0.075
0.071
–
–
–
Qwen2.5-Coder 3B
0.220
0.234
0.232
0.220
0.188
0.138
Qwen2.5-Coder 3B Instruct
0.238
0.259
0.258
0.247
0.238
0.218
Qwen2.5-Coder 7B
0.279
0.297
0.293
0.275
0.238
0.175
Qwen2.5-Coder 7B Instruct
0.349
0.362
0.359
0.356
0.348
0.335
DeepSeek-Coder 6.7B
0.163
0.175
0.171
0.162
0.144
0.113
DeepSeek-Coder 6.7B Instruct
0.201
0.216
0.212
0.209
0.201
0.189
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.092
0.076
0.052
0.026
Llama-3 8B Instruct
0.164
0.161
0.157
0.149
0.136
0.121
Llama-3.1 8B
0.102
0.100
0.093
0.078
0.056
0.029
Llama-3.1 8B Instruct
0.191
0.189
0.187
0.177
0.159
0.132
Llama-3.2 1B
0.015
0.014
0.012
0.009
0.007
0.003
Llama-3.2 1B Instruct
0.062
0.064
0.058
–
–
–
Qwen2.5-Coder 3B
0.220
0.221
0.215
0.200
0.162
0.101
Qwen2.5-Coder 3B Instruct
0.238
0.243
0.241
0.231
0.218
0.193
Qwen2.5-Coder 7B
0.279
0.282
0.275
0.252
0.209
0.133
Qwen2.5-Coder 7B Instruct
0.349
0.348
0.343
0.337
0.327
0.312
DeepSeek-Coder 6.7B
0.163
0.164
0.158
0.146
0.124
0.085
DeepSeek-Coder 6.7B Instruct
0.201
0.198
0.192
0.188
0.178
0.164
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.091
0.075
0.051
0.024
Llama-3 8B Instruct
0.164
0.160
0.155
0.146
0.134
0.118
Llama-3.1 8B
0.102
0.100
0.093
0.077
0.054
0.027
Llama-3.1 8B Instruct
0.191
0.189
0.186
0.176
0.158
0.130
Llama-3.2 1B
0.015
0.012
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.062
0.062
0.056
–
–
–
Qwen2.5-Coder 3B
0.220
0.221
0.215
0.199
0.161
0.098
Qwen2.5-Coder 3B Instruct
0.238
0.242
0.240
0.230
0.217
0.191
Qwen2.5-Coder 7B
0.279
0.281
0.274
0.251
0.208
0.131
Qwen2.5-Coder 7B Instruct
0.349
0.347
0.341
0.336
0.326
0.310
DeepSeek-Coder 6.7B
0.163
0.164
0.157
0.146
0.123
0.083
DeepSeek-Coder 6.7B Instruct
0.201
0.198
0.192
0.188
0.178
0.163
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.091
0.075
0.051
0.024
Llama-3 8B Instruct
0.164
0.160
0.155
0.146
0.134
0.118
Llama-3.1 8B
0.102
0.100
0.093
0.077
0.054
0.027
Llama-3.1 8B Instruct
0.191
0.189
0.186
0.176
0.158
0.130
Llama-3.2 1B
0.015
0.012
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.062
0.062
0.056
–
–
–
Qwen2.5-Coder 3B
0.220
0.221
0.215
0.199
0.161
0.098
Qwen2.5-Coder 3B Instruct
0.238
0.242
0.240
0.230
0.217
0.191
Qwen2.5-Coder 7B
0.279
0.281
0.274
0.251
0.208
0.131
Qwen2.5-Coder 7B Instruct
0.349
0.347
0.341
0.336
0.326
0.310
DeepSeek-Coder 6.7B
0.163
0.164
0.157
0.146
0.123
0.083
DeepSeek-Coder 6.7B Instruct
0.201
0.198
0.192
0.188
0.178
0.163
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.091
0.075
0.051
0.024
Llama-3 8B Instruct
0.164
0.160
0.155
0.146
0.134
0.118
Llama-3.1 8B
0.102
0.100
0.093
0.077
0.054
0.027
Llama-3.1 8B Instruct
0.191
0.189
0.186
0.176
0.158
0.130
Llama-3.2 1B
0.015
0.012
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.062
0.062
0.056
–
–
–
Qwen2.5-Coder 3B
0.220
0.221
0.215
0.199
0.161
0.098
Qwen2.5-Coder 3B Instruct
0.238
0.242
0.240
0.230
0.217
0.191
Qwen2.5-Coder 7B
0.279
0.281
0.274
0.251
0.208
0.131
Qwen2.5-Coder 7B Instruct
0.349
0.347
0.341
0.336
0.326
0.310
DeepSeek-Coder 6.7B
0.163
0.164
0.157
0.146
0.123
0.083
DeepSeek-Coder 6.7B Instruct
0.201
0.198
0.192
0.188
0.178
0.163
problems with no accepted rollout
Fraction of problems where none of the N rollouts is accepted at threshold k. At k=0 these are the problems for which the model never passes all public tests in N tries; they enter ŝ as 0.
Z, the acceptance rate (normalizing constant)
Z = proportion of rollouts accepted at threshold k (k=0: pass all public tests). Rises toward 1 as k grows.
pooled Ŝ (secondary)
Ŝ = ∑correct / ∑accepted = pass@1 / Zk, the former headline.
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.706
0.694
0.676
0.694
0.686
0.723
Llama-3 8B Instruct
0.684
0.662
0.670
0.675
0.685
0.697
Llama-3.1 8B
0.670
0.662
0.673
0.686
0.707
0.734
Llama-3.1 8B Instruct
0.716
0.729
0.742
0.758
0.770
0.789
Llama-3.2 1B
0.579
0.590
0.563
0.501
0.481
0.412
Llama-3.2 1B Instruct
0.746
0.668
0.669
–
0.682
0.692
Qwen2.5-Coder 3B
0.766
0.772
0.777
0.789
0.793
0.825
Qwen2.5-Coder 3B Instruct
0.783
0.775
0.784
0.788
0.795
0.805
Qwen2.5-Coder 7B
0.783
0.799
0.808
0.816
0.825
0.844
Qwen2.5-Coder 7B Instruct
0.814
0.805
0.805
0.808
0.809
0.815
DeepSeek-Coder 6.7B
0.758
0.735
0.734
0.732
0.743
0.754
DeepSeek-Coder 6.7B Instruct
0.812
0.779
0.775
0.777
0.784
0.796
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.342
0.334
0.308
0.289
0.249
0.195
Llama-3 8B Instruct
0.385
0.380
0.381
0.380
0.377
0.373
Llama-3.1 8B
0.326
0.313
0.306
0.290
0.258
0.207
Llama-3.1 8B Instruct
0.426
0.438
0.442
0.437
0.429
0.416
Llama-3.2 1B
0.120
0.097
0.083
0.056
0.041
0.019
Llama-3.2 1B Instruct
0.251
0.255
0.248
–
0.204
0.159
Qwen2.5-Coder 3B
0.497
0.489
0.488
0.487
0.463
0.425
Qwen2.5-Coder 3B Instruct
0.496
0.508
0.509
0.503
0.499
0.494
Qwen2.5-Coder 7B
0.551
0.554
0.553
0.541
0.519
0.479
Qwen2.5-Coder 7B Instruct
0.589
0.586
0.584
0.582
0.580
0.579
DeepSeek-Coder 6.7B
0.417
0.408
0.405
0.398
0.386
0.355
DeepSeek-Coder 6.7B Instruct
0.480
0.471
0.463
0.463
0.461
0.460
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.152
0.146
0.131
0.113
0.083
0.045
Llama-3 8B Instruct
0.215
0.213
0.208
0.200
0.189
0.174
Llama-3.1 8B
0.143
0.140
0.132
0.115
0.087
0.050
Llama-3.1 8B Instruct
0.245
0.245
0.243
0.233
0.216
0.192
Llama-3.2 1B
0.029
0.023
0.018
0.012
0.008
0.003
Llama-3.2 1B Instruct
0.092
0.094
0.088
–
0.058
0.035
Qwen2.5-Coder 3B
0.291
0.291
0.285
0.270
0.231
0.162
Qwen2.5-Coder 3B Instruct
0.299
0.309
0.308
0.298
0.287
0.266
Qwen2.5-Coder 7B
0.351
0.353
0.348
0.327
0.287
0.210
Qwen2.5-Coder 7B Instruct
0.415
0.411
0.407
0.403
0.396
0.387
DeepSeek-Coder 6.7B
0.216
0.217
0.211
0.199
0.175
0.133
DeepSeek-Coder 6.7B Instruct
0.268
0.261
0.253
0.249
0.241
0.228
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.110
0.105
0.093
0.077
0.053
0.025
Llama-3 8B Instruct
0.167
0.164
0.158
0.149
0.137
0.121
Llama-3.1 8B
0.104
0.102
0.094
0.079
0.056
0.028
Llama-3.1 8B Instruct
0.194
0.191
0.189
0.179
0.161
0.134
Llama-3.2 1B
0.016
0.013
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.063
0.064
0.058
–
0.035
0.019
Qwen2.5-Coder 3B
0.224
0.225
0.219
0.203
0.165
0.102
Qwen2.5-Coder 3B Instruct
0.242
0.246
0.244
0.234
0.221
0.196
Qwen2.5-Coder 7B
0.283
0.285
0.278
0.256
0.213
0.135
Qwen2.5-Coder 7B Instruct
0.353
0.351
0.345
0.340
0.330
0.315
DeepSeek-Coder 6.7B
0.165
0.167
0.160
0.148
0.126
0.085
DeepSeek-Coder 6.7B Instruct
0.203
0.201
0.195
0.191
0.181
0.166
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.091
0.076
0.051
0.024
Llama-3 8B Instruct
0.164
0.161
0.155
0.147
0.134
0.118
Llama-3.1 8B
0.103
0.101
0.093
0.077
0.055
0.027
Llama-3.1 8B Instruct
0.191
0.189
0.186
0.176
0.158
0.131
Llama-3.2 1B
0.015
0.012
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.062
0.063
0.057
–
0.034
0.018
Qwen2.5-Coder 3B
0.221
0.222
0.215
0.199
0.161
0.098
Qwen2.5-Coder 3B Instruct
0.239
0.242
0.241
0.230
0.217
0.192
Qwen2.5-Coder 7B
0.280
0.281
0.274
0.252
0.209
0.131
Qwen2.5-Coder 7B Instruct
0.349
0.347
0.342
0.336
0.326
0.311
DeepSeek-Coder 6.7B
0.163
0.165
0.158
0.146
0.123
0.083
DeepSeek-Coder 6.7B Instruct
0.201
0.199
0.192
0.188
0.178
0.163
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.091
0.075
0.051
0.024
Llama-3 8B Instruct
0.164
0.161
0.155
0.146
0.134
0.118
Llama-3.1 8B
0.102
0.100
0.093
0.077
0.054
0.027
Llama-3.1 8B Instruct
0.191
0.189
0.186
0.176
0.158
0.130
Llama-3.2 1B
0.015
0.012
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.062
0.062
0.057
–
0.034
0.018
Qwen2.5-Coder 3B
0.221
0.221
0.215
0.199
0.161
0.098
Qwen2.5-Coder 3B Instruct
0.239
0.242
0.240
0.230
0.217
0.191
Qwen2.5-Coder 7B
0.279
0.281
0.274
0.251
0.208
0.131
Qwen2.5-Coder 7B Instruct
0.349
0.347
0.341
0.336
0.326
0.310
DeepSeek-Coder 6.7B
0.163
0.164
0.158
0.146
0.123
0.083
DeepSeek-Coder 6.7B Instruct
0.201
0.198
0.192
0.188
0.178
0.163
model
t=0
t=0.2
t=0.4
t=0.6
t=0.8
t=1.0
Llama-3 8B
0.108
0.103
0.091
0.075
0.051
0.024
Llama-3 8B Instruct
0.164
0.160
0.155
0.146
0.134
0.118
Llama-3.1 8B
0.102
0.100
0.093
0.077
0.054
0.027
Llama-3.1 8B Instruct
0.191
0.189
0.186
0.176
0.158
0.130
Llama-3.2 1B
0.015
0.012
0.010
0.006
0.004
0.001
Llama-3.2 1B Instruct
0.062
0.062
0.057
–
0.034
0.018
Qwen2.5-Coder 3B
0.220
0.221
0.215
0.199
0.161
0.098
Qwen2.5-Coder 3B Instruct
0.238
0.242
0.240
0.230
0.217
0.191
Qwen2.5-Coder 7B
0.279
0.281
0.274
0.251
0.208
0.131
Qwen2.5-Coder 7B Instruct
0.349
0.347
0.341
0.336
0.326
0.310
DeepSeek-Coder 6.7B
0.163
0.164
0.158
0.146
0.123
0.083
DeepSeek-Coder 6.7B Instruct
0.201
0.198
0.192
0.188
0.178
0.163
by difficulty, choose a model
Public-test ŝ and Z split by difficulty for a single model, at the selected k. Use both selectors; the plots and table update together.
ŝ, zero-valid fraction, Z and pooled Ŝ by difficulty and temperature (selected model & k):