Millions of generations across code and structured-generation domains, each individually scored by a deterministic evaluator and browsable down to the single completion with its prompt, verdict, and error. Every curve carries a 95% bootstrap confidence interval.
The 1000-problem sweep in full: temperature curves, significance tests, and the complete error analysis, including Qwen2.5-Coder and DeepSeek-Coder.
Competitive programming (714 problems, 2023-05 to 2024-09): temperature curves with CIs, failure categories, and bug themes. 6 models complete; more being added.
The same temperature sweep on three non-code domains (molecular synthesis, goal inference, text-to-SQL) shown next to DS-1000, for contrast with the code benchmarks above.
The raw material behind every figure, browsable in the browser with no download.
The same tasks decoded under sequential Monte Carlo with grammar constraints, compared against raw sampling.
The sweep methodology applied to competitive-programming problems.
For each domain, each model draws 100 completions per instance at every temperature with plain vLLM sampling (no constrained decoding, no SMC). Every completion is scored individually; pass@k uses the unbiased estimator, averaged over instances, with confidence intervals from 2000 bootstrap resamples of the instance set and paired tests for peak temperatures and temperature steps.
| Domain | Task | Instances | Scored by |
|---|---|---|---|
| DS-1000 | data-science code completion, 7 libraries | 1000 | official test execution + source check, fresh subprocess, deterministic |
| Molecular synthesis | generate drug-like molecules (SMILES) | 100 | RDKit validity + QED |
| Goal inference | infer PDDL goals (blocksworld) | 100 | planetarium goal equivalence |
| Text-to-SQL | Spider | 100 | SQLite execution match, sequential scoring |
Everything lives in a public Hugging Face dataset,
samuki-hf/temperature-sweep-data:
hive-partitioned Parquet, one partition per (domain, model, temperature). Each generation
carries its verdict (passed, plus score/valid where
applicable) and joins to the prompt, failure, and bug-theme tables by
(instance_id, sample).
DuckDB queries it in place, so a question costs seconds, not a download,
and the dataset is public: no token needed. The repo's CLAUDE.md documents
every schema, so you can also clone the repo, run Claude Code, and ask questions about the
data directly.
import duckdb
con = duckdb.connect()
con.sql("""
SELECT model, temp, avg(passed::INT) AS pass_rate
FROM read_parquet(
'hf://datasets/samuki-hf/temperature-sweep-data'
'/rollouts/**/*.parquet', hive_partitioning=1)
WHERE domain = 'ds1000' AND passed IS NOT NULL
GROUP BY model, temp ORDER BY model, temp
""").df()