LiveCodeBench Skyline

Acceptance rate (Z) and accuracy ceiling (skyline) of two potentials — the no-runtime-error potential (LCBRuntimeNoErrorPotential) and the public-test potential (LCBPublicTestPotential) — under exact posterior sampling, across 12 models and 6 temperatures (71 cells, ~4.2M rollouts).

Definitions

Two potentials, two constraints. Each rollout y for question x defines a validity indicator φ(y); rejection sampling targets π(·|x) ∝ p₀(·|x)·φ(·) — sample from the model, keep the valid ones.

Z(x)=Pr(y valid|x) skyline(x)=Pr(y correct|y valid,x) Pr(y correct|x)=Z(x)·skyline(x)

Pooled over all rollouts, Z=(∑valid)/(∑N), skyline=(∑correct)/(∑valid)=pass@1/Z, so pass@1 = Z × skyline holds on the plotted numbers. Correct = the LCB evaluator’s verdict over all tests (public + private).

No-error potential (LCBRuntimeNoErrorPotential): φ = the code runs on the public inputs with no runtime error / timeout. A program that runs but answers wrongly is still valid.

Public-test potential (LCBPublicTestPotential): the strict constraint is “passes all public example tests”. Because the potential is soft (finite penalty per failed public test), every public-test section is indexed by k = the max number of failing public tests allowed: φ_k=1[n_failed≤k]. k=0 is the strict potential; raising k relaxes it, and at k=max#public-tests every rollout is accepted (Z=1, skyline=pass@1). A correct solution passes every public test, so correct ⇒ φ_k and pass@1 = Z_k × skyline_k at every k.

Nesting: correct ⇒ passes-all-public (k=0) ⇒ runs-no-error, hence pass@1 ≤ Z_public ≤ Z_no-error.

Why “skyline”? It is the accuracy of ideal constrained generation: if you could sample exactly from π (enforce the constraint and nothing else), this is how often the result would be correct — a reference for judging approximate inference methods (importance sampling, SMC, fine-tuning) that target the same π. Reported pooled over problems so pass@1 = Z × skyline holds; shaded bands are 95% bootstrap CIs (2000 resamples of the problem set).

LiveCodeBench, 12 models × 6 temperatures (71 cells, ~4.2M rollouts); both potentials run literally (forkserver, 6 s per-test timeout). Instances with no public tests excluded from the public-test analysis.

Part 1 — no-error potential

skyline against temperature

skyline = (correct)/(runs-without-error), pooled. Wrong-but-running answers dilute it.

skyline by model and temperature

Z, the acceptance rate (normalizing constant)

by difficulty, choose a model

No-error skyline and Z split by problem difficulty for a single model. Shaded bands are 95% bootstrap CIs over that difficulty’s problems.

Part 2 — public-test potential

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.138	0.131	0.119	0.106	0.085	0.063
Llama-3 8B Instruct	0.202	0.199	0.195	0.189	0.183	0.180
Llama-3.1 8B	0.132	0.127	0.119	0.106	0.088	0.068
Llama-3.1 8B Instruct	0.229	0.224	0.222	0.216	0.206	0.203
Llama-3.2 1B	0.027	0.022	0.018	0.013	0.009	0.005
Llama-3.2 1B Instruct	0.098	0.095	0.088	–	0.067	0.054
Qwen2.5-Coder 3B	0.273	0.273	0.269	0.257	0.227	0.190
Qwen2.5-Coder 3B Instruct	0.297	0.300	0.298	0.289	0.282	0.273
Qwen2.5-Coder 7B	0.321	0.319	0.315	0.299	0.271	0.240
Qwen2.5-Coder 7B Instruct	0.399	0.394	0.390	0.387	0.381	0.377
DeepSeek-Coder 6.7B	0.185	0.187	0.182	0.173	0.159	0.139
DeepSeek-Coder 6.7B Instruct	0.242	0.234	0.227	0.227	0.224	0.220

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.340	0.323	0.294	0.260	0.199	0.136
Z	0.878	0.875	0.863	0.811	0.721	0.494
medium	skyline	0.028	0.029	0.022	0.018	0.012	0.007
Z	0.773	0.786	0.770	0.718	0.612	0.372
hard	skyline	0.000	0.000	0.001	0.000	0.001	0.000
Z	0.682	0.681	0.651	0.571	0.440	0.232

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.477	0.483	0.474	0.460	0.441	0.417
Z	0.899	0.892	0.879	0.855	0.821	0.769
medium	skyline	0.056	0.049	0.049	0.047	0.044	0.041
Z	0.845	0.823	0.806	0.781	0.740	0.651
hard	skyline	0.015	0.005	0.002	0.002	0.002	0.003
Z	0.667	0.676	0.676	0.659	0.614	0.523

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.330	0.328	0.303	0.264	0.211	0.148
Z	0.866	0.875	0.872	0.833	0.743	0.527
medium	skyline	0.023	0.014	0.014	0.013	0.009	0.005
Z	0.780	0.790	0.785	0.739	0.630	0.385
hard	skyline	0.000	0.000	0.000	0.000	0.000	0.000
Z	0.662	0.685	0.657	0.596	0.463	0.244

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.509	0.517	0.519	0.501	0.468	0.424
Z	0.933	0.929	0.926	0.914	0.892	0.830
medium	skyline	0.084	0.076	0.069	0.066	0.055	0.046
Z	0.863	0.855	0.844	0.820	0.775	0.638
hard	skyline	0.022	0.016	0.014	0.011	0.011	0.010
Z	0.677	0.724	0.717	0.689	0.606	0.422

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.078	0.060	0.050	0.033	0.024	0.012
Z	0.592	0.608	0.588	0.542	0.462	0.297
medium	skyline	0.000	0.000	0.000	0.001	0.000	0.000
Z	0.607	0.593	0.571	0.532	0.456	0.285
hard	skyline	0.000	0.000	0.000	0.000	0.000	0.000
Z	0.475	0.437	0.402	0.363	0.295	0.168

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.217	0.216	0.203	–	0.149	0.112
Z	0.773	0.788	0.765	–	0.635	0.455
medium	skyline	0.017	0.018	0.015	–	0.009	0.005
Z	0.653	0.674	0.648	–	0.502	0.327
hard	skyline	0.012	0.007	0.006	–	0.004	0.003
Z	0.434	0.487	0.476	–	0.355	0.195

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.565	0.565	0.553	0.525	0.459	0.367
Z	0.908	0.912	0.907	0.889	0.838	0.679
medium	skyline	0.146	0.145	0.141	0.134	0.110	0.072
Z	0.816	0.820	0.813	0.780	0.711	0.504
hard	skyline	0.015	0.016	0.016	0.013	0.012	0.006
Z	0.677	0.673	0.653	0.627	0.548	0.335

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.569	0.592	0.592	0.578	0.560	0.530
Z	0.945	0.924	0.922	0.915	0.900	0.866
medium	skyline	0.181	0.176	0.172	0.162	0.153	0.134
Z	0.816	0.830	0.828	0.812	0.781	0.705
hard	skyline	0.008	0.015	0.015	0.013	0.012	0.009
Z	0.611	0.627	0.633	0.625	0.591	0.495

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.648	0.644	0.627	0.591	0.528	0.427
Z	0.954	0.953	0.951	0.937	0.891	0.727
medium	skyline	0.202	0.203	0.202	0.186	0.159	0.124
Z	0.892	0.906	0.890	0.853	0.777	0.540
hard	skyline	0.014	0.022	0.024	0.023	0.018	0.014
Z	0.742	0.761	0.748	0.704	0.606	0.334

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.758	0.729	0.719	0.711	0.700	0.689
Z	0.954	0.970	0.969	0.964	0.959	0.945
medium	skyline	0.293	0.299	0.298	0.297	0.288	0.274
Z	0.888	0.890	0.879	0.874	0.860	0.828
hard	skyline	0.033	0.035	0.032	0.030	0.026	0.026
Z	0.763	0.762	0.757	0.742	0.724	0.673

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.412	0.414	0.400	0.385	0.352	0.296
Z	0.958	0.946	0.938	0.922	0.875	0.723
medium	skyline	0.086	0.095	0.093	0.081	0.065	0.049
Z	0.877	0.868	0.859	0.833	0.766	0.577
hard	skyline	0.006	0.007	0.006	0.004	0.004	0.002
Z	0.783	0.808	0.790	0.754	0.662	0.463

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.487	0.487	0.487	0.488	0.481	0.471
Z	0.941	0.948	0.945	0.935	0.910	0.869
medium	skyline	0.151	0.134	0.116	0.108	0.102	0.093
Z	0.787	0.824	0.825	0.810	0.774	0.720
hard	skyline	0.007	0.005	0.006	0.006	0.005	0.003
Z	0.752	0.756	0.749	0.726	0.684	0.612

Pick the acceptance threshold k (max public-test failures allowed); every plot and table in this part updates to that k. k=0 is the strict “pass all public tests” potential.

skyline against temperature

skyline = (correct)/(accepted at threshold k), pooled. At k=0 this is the strict public-test skyline — much higher than the no-error skyline, and it tends to rise with temperature.

skyline by model and temperature

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.706	0.694	0.676	0.694	0.686	0.723
Llama-3 8B Instruct	0.684	0.662	0.670	0.675	0.685	0.697
Llama-3.1 8B	0.670	0.662	0.673	0.686	0.707	0.734
Llama-3.1 8B Instruct	0.716	0.729	0.742	0.758	0.770	0.789
Llama-3.2 1B	0.579	0.590	0.563	0.501	0.481	0.412
Llama-3.2 1B Instruct	0.746	0.668	0.669	–	0.682	0.692
Qwen2.5-Coder 3B	0.766	0.772	0.777	0.789	0.793	0.825
Qwen2.5-Coder 3B Instruct	0.783	0.775	0.784	0.788	0.795	0.805
Qwen2.5-Coder 7B	0.783	0.799	0.808	0.816	0.825	0.844
Qwen2.5-Coder 7B Instruct	0.814	0.805	0.805	0.808	0.809	0.815
DeepSeek-Coder 6.7B	0.758	0.735	0.734	0.732	0.743	0.754
DeepSeek-Coder 6.7B Instruct	0.812	0.779	0.775	0.777	0.784	0.796

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.342	0.334	0.308	0.289	0.249	0.195
Llama-3 8B Instruct	0.385	0.380	0.381	0.380	0.377	0.373
Llama-3.1 8B	0.326	0.313	0.306	0.290	0.258	0.207
Llama-3.1 8B Instruct	0.426	0.438	0.442	0.437	0.429	0.416
Llama-3.2 1B	0.120	0.097	0.083	0.056	0.041	0.019
Llama-3.2 1B Instruct	0.251	0.255	0.248	–	0.204	0.159
Qwen2.5-Coder 3B	0.497	0.489	0.488	0.487	0.463	0.425
Qwen2.5-Coder 3B Instruct	0.496	0.508	0.509	0.503	0.499	0.494
Qwen2.5-Coder 7B	0.551	0.554	0.553	0.541	0.519	0.479
Qwen2.5-Coder 7B Instruct	0.589	0.586	0.584	0.582	0.580	0.579
DeepSeek-Coder 6.7B	0.417	0.408	0.405	0.398	0.386	0.355
DeepSeek-Coder 6.7B Instruct	0.480	0.471	0.463	0.463	0.461	0.460

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.152	0.146	0.131	0.113	0.083	0.045
Llama-3 8B Instruct	0.215	0.213	0.208	0.200	0.189	0.174
Llama-3.1 8B	0.143	0.140	0.132	0.115	0.087	0.050
Llama-3.1 8B Instruct	0.245	0.245	0.243	0.233	0.216	0.192
Llama-3.2 1B	0.029	0.023	0.018	0.012	0.008	0.003
Llama-3.2 1B Instruct	0.092	0.094	0.088	–	0.058	0.035
Qwen2.5-Coder 3B	0.291	0.291	0.285	0.270	0.231	0.162
Qwen2.5-Coder 3B Instruct	0.299	0.309	0.308	0.298	0.287	0.266
Qwen2.5-Coder 7B	0.351	0.353	0.348	0.327	0.287	0.210
Qwen2.5-Coder 7B Instruct	0.415	0.411	0.407	0.403	0.396	0.387
DeepSeek-Coder 6.7B	0.216	0.217	0.211	0.199	0.175	0.133
DeepSeek-Coder 6.7B Instruct	0.268	0.261	0.253	0.249	0.241	0.228

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.110	0.105	0.093	0.077	0.053	0.025
Llama-3 8B Instruct	0.167	0.164	0.158	0.149	0.137	0.121
Llama-3.1 8B	0.104	0.102	0.094	0.079	0.056	0.028
Llama-3.1 8B Instruct	0.194	0.191	0.189	0.179	0.161	0.134
Llama-3.2 1B	0.016	0.013	0.010	0.006	0.004	0.001
Llama-3.2 1B Instruct	0.063	0.064	0.058	–	0.035	0.019
Qwen2.5-Coder 3B	0.224	0.225	0.219	0.203	0.165	0.102
Qwen2.5-Coder 3B Instruct	0.242	0.246	0.244	0.234	0.221	0.196
Qwen2.5-Coder 7B	0.283	0.285	0.278	0.256	0.213	0.135
Qwen2.5-Coder 7B Instruct	0.353	0.351	0.345	0.340	0.330	0.315
DeepSeek-Coder 6.7B	0.165	0.167	0.160	0.148	0.126	0.085
DeepSeek-Coder 6.7B Instruct	0.203	0.201	0.195	0.191	0.181	0.166

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.108	0.103	0.091	0.076	0.051	0.024
Llama-3 8B Instruct	0.164	0.161	0.155	0.147	0.134	0.118
Llama-3.1 8B	0.103	0.101	0.093	0.077	0.055	0.027
Llama-3.1 8B Instruct	0.191	0.189	0.186	0.176	0.158	0.131
Llama-3.2 1B	0.015	0.012	0.010	0.006	0.004	0.001
Llama-3.2 1B Instruct	0.062	0.063	0.057	–	0.034	0.018
Qwen2.5-Coder 3B	0.221	0.222	0.215	0.199	0.161	0.098
Qwen2.5-Coder 3B Instruct	0.239	0.242	0.241	0.230	0.217	0.192
Qwen2.5-Coder 7B	0.280	0.281	0.274	0.252	0.209	0.131
Qwen2.5-Coder 7B Instruct	0.349	0.347	0.342	0.336	0.326	0.311
DeepSeek-Coder 6.7B	0.163	0.165	0.158	0.146	0.123	0.083
DeepSeek-Coder 6.7B Instruct	0.201	0.199	0.192	0.188	0.178	0.163

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.108	0.103	0.091	0.075	0.051	0.024
Llama-3 8B Instruct	0.164	0.161	0.155	0.146	0.134	0.118
Llama-3.1 8B	0.102	0.100	0.093	0.077	0.054	0.027
Llama-3.1 8B Instruct	0.191	0.189	0.186	0.176	0.158	0.130
Llama-3.2 1B	0.015	0.012	0.010	0.006	0.004	0.001
Llama-3.2 1B Instruct	0.062	0.062	0.057	–	0.034	0.018
Qwen2.5-Coder 3B	0.221	0.221	0.215	0.199	0.161	0.098
Qwen2.5-Coder 3B Instruct	0.239	0.242	0.240	0.230	0.217	0.191
Qwen2.5-Coder 7B	0.279	0.281	0.274	0.251	0.208	0.131
Qwen2.5-Coder 7B Instruct	0.349	0.347	0.341	0.336	0.326	0.310
DeepSeek-Coder 6.7B	0.163	0.164	0.158	0.146	0.123	0.083
DeepSeek-Coder 6.7B Instruct	0.201	0.198	0.192	0.188	0.178	0.163

model	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
Llama-3 8B	0.108	0.103	0.091	0.075	0.051	0.024
Llama-3 8B Instruct	0.164	0.160	0.155	0.146	0.134	0.118
Llama-3.1 8B	0.102	0.100	0.093	0.077	0.054	0.027
Llama-3.1 8B Instruct	0.191	0.189	0.186	0.176	0.158	0.130
Llama-3.2 1B	0.015	0.012	0.010	0.006	0.004	0.001
Llama-3.2 1B Instruct	0.062	0.062	0.057	–	0.034	0.018
Qwen2.5-Coder 3B	0.220	0.221	0.215	0.199	0.161	0.098
Qwen2.5-Coder 3B Instruct	0.238	0.242	0.240	0.230	0.217	0.191
Qwen2.5-Coder 7B	0.279	0.281	0.274	0.251	0.208	0.131
Qwen2.5-Coder 7B Instruct	0.349	0.347	0.341	0.336	0.326	0.310
DeepSeek-Coder 6.7B	0.163	0.164	0.158	0.146	0.123	0.083
DeepSeek-Coder 6.7B Instruct	0.201	0.198	0.192	0.188	0.178	0.163

Z, the acceptance rate (normalizing constant)

Z = pooled proportion of rollouts accepted at threshold k (k=0: pass all public tests). Rises toward 1 as k grows.

by difficulty, choose a model

Public-test skyline and Z split by difficulty for a single model, at the selected k. Use both selectors; the plots and table update together.

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.934	0.905	0.881	0.889	0.851	0.835
easy	Z	0.319	0.312	0.288	0.237	0.169	0.081
medium	skyline	0.222	0.256	0.230	0.231	0.220	0.239
medium	Z	0.098	0.088	0.073	0.055	0.034	0.012
hard	skyline	0.000	0.008	0.013	0.007	0.015	0.024
hard	Z	0.030	0.037	0.038	0.029	0.017	0.004

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.617	0.584	0.542	0.509	0.442	0.358
easy	Z	0.483	0.484	0.467	0.414	0.325	0.188
medium	skyline	0.081	0.088	0.070	0.060	0.045	0.029
medium	Z	0.267	0.255	0.242	0.212	0.165	0.096
hard	skyline	0.000	0.002	0.003	0.001	0.002	0.001
hard	Z	0.182	0.174	0.168	0.146	0.115	0.075

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.364	0.339	0.309	0.271	0.202	0.114
easy	Z	0.819	0.833	0.821	0.779	0.712	0.592
medium	skyline	0.033	0.034	0.026	0.020	0.013	0.005
medium	Z	0.664	0.667	0.659	0.634	0.591	0.513
hard	skyline	0.000	0.001	0.001	0.000	0.001	0.000
hard	Z	0.636	0.611	0.597	0.575	0.529	0.468

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.864	0.894	0.902	0.903	0.901	0.898
easy	Z	0.496	0.482	0.461	0.436	0.402	0.357
medium	skyline	0.382	0.292	0.300	0.303	0.309	0.313
medium	Z	0.123	0.138	0.132	0.122	0.106	0.086
hard	skyline	0.105	0.031	0.017	0.018	0.019	0.025
hard	Z	0.096	0.100	0.093	0.085	0.073	0.059

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.650	0.667	0.665	0.660	0.650	0.633
easy	Z	0.660	0.647	0.626	0.597	0.558	0.506
medium	skyline	0.141	0.119	0.122	0.120	0.115	0.110
medium	Z	0.332	0.337	0.323	0.309	0.285	0.244
hard	skyline	0.036	0.011	0.006	0.007	0.006	0.008
hard	Z	0.278	0.271	0.257	0.237	0.213	0.187

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.493	0.503	0.491	0.473	0.448	0.411
easy	Z	0.870	0.857	0.848	0.832	0.809	0.780
medium	skyline	0.062	0.054	0.054	0.052	0.047	0.041
medium	Z	0.751	0.738	0.725	0.714	0.694	0.653
hard	skyline	0.016	0.005	0.002	0.003	0.002	0.003
hard	Z	0.646	0.651	0.647	0.633	0.615	0.583

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.432	0.435	0.421	0.399	0.367	0.326
easy	Z	0.992	0.990	0.989	0.988	0.986	0.982
medium	skyline	0.048	0.041	0.040	0.038	0.034	0.028
medium	Z	0.975	0.977	0.976	0.975	0.975	0.971
hard	skyline	0.010	0.003	0.002	0.002	0.001	0.002
hard	Z	0.975	0.973	0.971	0.970	0.965	0.959

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.931	0.908	0.894	0.881	0.869	0.839
easy	Z	0.307	0.316	0.295	0.250	0.180	0.093
medium	skyline	0.200	0.145	0.165	0.181	0.181	0.174
medium	Z	0.090	0.079	0.069	0.053	0.031	0.010
hard	skyline	0.000	0.000	0.003	0.008	0.000	0.020
hard	Z	0.056	0.056	0.044	0.031	0.018	0.005

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.602	0.588	0.556	0.516	0.461	0.375
easy	Z	0.475	0.488	0.475	0.426	0.340	0.208
medium	skyline	0.069	0.043	0.046	0.045	0.033	0.018
medium	Z	0.260	0.262	0.244	0.214	0.168	0.098
hard	skyline	0.000	0.000	0.001	0.002	0.000	0.001
hard	Z	0.197	0.199	0.176	0.149	0.117	0.079

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.286	0.288	0.265	0.222	0.159	0.081
easy	Z	1.000	0.997	0.996	0.992	0.984	0.967
medium	skyline	0.018	0.012	0.012	0.010	0.006	0.002
medium	Z	0.978	0.980	0.979	0.974	0.969	0.959
hard	skyline	0.000	0.000	0.000	0.000	0.000	0.000
hard	Z	0.965	0.974	0.970	0.965	0.957	0.943

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.911	0.901	0.911	0.916	0.913	0.906
easy	Z	0.521	0.533	0.528	0.500	0.457	0.388
medium	skyline	0.408	0.429	0.411	0.428	0.417	0.415
medium	Z	0.177	0.151	0.142	0.126	0.103	0.070
hard	skyline	0.176	0.146	0.141	0.130	0.144	0.148
hard	Z	0.086	0.079	0.070	0.059	0.045	0.028

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.846	0.798	0.750	0.696	0.655	0.577
easy	Z	0.055	0.046	0.039	0.026	0.017	0.006
medium	skyline	0.000	0.008	0.021	0.048	0.040	0.054
medium	Z	0.014	0.009	0.009	0.007	0.004	0.002
hard	skyline	0.000	0.007	0.024	0.014	0.056	0.000
hard	Z	0.010	0.007	0.004	0.004	0.002	0.001

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.082	0.063	0.051	0.033	0.022	0.008
easy	Z	0.567	0.580	0.568	0.544	0.517	0.477
medium	skyline	0.000	0.000	0.000	0.001	0.000	0.000
medium	Z	0.541	0.538	0.534	0.526	0.510	0.477
hard	skyline	0.000	0.000	0.000	0.000	0.000	0.000
hard	Z	0.490	0.495	0.492	0.484	0.471	0.450

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.430	0.449	0.443	–	0.377	0.304
easy	Z	0.391	0.379	0.351	–	0.252	0.168
medium	skyline	0.059	0.063	0.054	–	0.033	0.020
medium	Z	0.184	0.191	0.179	–	0.130	0.086
hard	skyline	0.032	0.022	0.018	–	0.012	0.007
hard	Z	0.157	0.159	0.149	–	0.109	0.081

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.171	0.173	0.158	–	0.098	0.053
easy	Z	0.983	0.984	0.982	–	0.972	0.961
medium	skyline	0.011	0.012	0.010	–	0.004	0.002
medium	Z	0.982	0.974	0.972	–	0.964	0.956
hard	skyline	0.005	0.004	0.003	–	0.001	0.001
hard	Z	0.950	0.955	0.956	–	0.950	0.940

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.731	0.732	0.732	0.722	0.678	0.621
easy	Z	0.702	0.703	0.685	0.647	0.568	0.401
medium	skyline	0.306	0.295	0.295	0.296	0.270	0.204
medium	Z	0.390	0.403	0.388	0.354	0.290	0.177
hard	skyline	0.049	0.049	0.047	0.042	0.040	0.021
hard	Z	0.207	0.219	0.216	0.196	0.162	0.098

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.576	0.581	0.568	0.540	0.470	0.348
easy	Z	0.891	0.886	0.883	0.865	0.820	0.716
medium	skyline	0.166	0.162	0.156	0.146	0.116	0.062
medium	Z	0.718	0.738	0.734	0.716	0.676	0.580
hard	skyline	0.015	0.017	0.016	0.013	0.011	0.004
hard	Z	0.651	0.638	0.630	0.613	0.578	0.504

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.587	0.604	0.605	0.590	0.572	0.536
easy	Z	0.916	0.906	0.902	0.895	0.882	0.857
medium	skyline	0.188	0.191	0.187	0.175	0.162	0.136
medium	Z	0.787	0.765	0.762	0.753	0.734	0.693
hard	skyline	0.008	0.014	0.015	0.012	0.011	0.007
hard	Z	0.667	0.661	0.657	0.649	0.631	0.586

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.620	0.617	0.600	0.557	0.474	0.317
easy	Z	0.996	0.995	0.994	0.994	0.992	0.979
medium	skyline	0.182	0.186	0.182	0.162	0.127	0.069
medium	Z	0.993	0.987	0.986	0.983	0.978	0.966
hard	skyline	0.010	0.017	0.018	0.017	0.011	0.005
hard	Z	0.970	0.972	0.972	0.969	0.966	0.950

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.618	0.614	0.597	0.554	0.471	0.311
easy	Z	1.000	1.000	1.000	1.000	0.999	0.996
medium	skyline	0.181	0.184	0.181	0.160	0.124	0.067
medium	Z	0.996	0.996	0.996	0.995	0.994	0.993
hard	skyline	0.010	0.017	0.018	0.017	0.011	0.005
hard	Z	1.000	1.000	1.000	1.000	1.000	1.000

LiveCodeBench, rejection-sampling skyline & Z

Definitions

Part 1 — no-error potential

skyline against temperature

skyline by model and temperature

Z, the acceptance rate (normalizing constant)

by difficulty, choose a model

Part 2 — public-test potential

skyline against temperature

skyline by model and temperature

Z, the acceptance rate (normalizing constant)

by difficulty, choose a model

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.843	0.826	0.820	0.814	0.807	0.804
easy	Z	0.857	0.856	0.850	0.842	0.832	0.809
medium	skyline	0.465	0.476	0.479	0.478	0.473	0.458
medium	Z	0.560	0.559	0.548	0.543	0.524	0.496
hard	skyline	0.078	0.082	0.077	0.073	0.065	0.066
hard	Z	0.323	0.320	0.318	0.305	0.290	0.261

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.771	0.751	0.740	0.730	0.720	0.707
easy	Z	0.937	0.942	0.942	0.939	0.933	0.920
medium	skyline	0.310	0.316	0.314	0.311	0.303	0.285
medium	Z	0.838	0.845	0.836	0.833	0.818	0.796
hard	skyline	0.035	0.036	0.034	0.032	0.027	0.026
hard	Z	0.732	0.725	0.719	0.708	0.697	0.668

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.454	0.450	0.435	0.416	0.375	0.292
easy	Z	0.870	0.869	0.864	0.853	0.822	0.732
medium	skyline	0.105	0.113	0.111	0.096	0.074	0.048
medium	Z	0.722	0.730	0.720	0.707	0.675	0.596
hard	skyline	0.008	0.009	0.007	0.005	0.004	0.002
hard	Z	0.657	0.654	0.643	0.630	0.597	0.528

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.395	0.392	0.377	0.357	0.311	0.218
easy	Z	1.000	0.999	0.997	0.994	0.991	0.981
medium	skyline	0.077	0.084	0.081	0.069	0.051	0.029
medium	Z	0.986	0.984	0.982	0.981	0.978	0.968
hard	skyline	0.005	0.006	0.005	0.003	0.002	0.001
hard	Z	0.970	0.973	0.974	0.973	0.969	0.956

difficulty	metric	t=0	t=0.2	t=0.4	t=0.6	t=0.8	t=1.0
easy	skyline	0.512	0.521	0.520	0.517	0.505	0.482
easy	Z	0.895	0.886	0.885	0.883	0.867	0.847
medium	skyline	0.168	0.149	0.130	0.121	0.111	0.097
medium	Z	0.708	0.740	0.736	0.728	0.711	0.688
hard	skyline	0.008	0.006	0.007	0.006	0.005	0.003
hard	Z	0.626	0.638	0.637	0.635	0.618	0.593