Temperature sweep across four domains

Raw base-model decoding (no SMC) on molecular synthesis, goal inference, text-to-SQL and DS-1000 code generation. How does sampling temperature trade off single-shot quality against best-of-many coverage? The legend is interactive. Click a model to show/hide its curves everywhere.

Which temperature works best depends on how many samples you draw. With one sample, decoding near greedy (about t = 0.2) gives the highest pass@1 and mean QED. If you keep the best of ten samples, t = 0.6 to 0.8 works better. Quality drops quickly once temperature goes above 1.0. The 1B models behave the same way but peak at lower temperatures and stay well below the 8B models.

On this page

Peak per model
Failure modes
What the categories look like

Peak per model

Best value@temperature: per-sample quality (mean QED / pass@1) and coverage@10 (valid@10 / pass@10), per domain.

model	Molecules (QED)		Goal inference		Text-to-SQL		DS-1000
model	quality	cov@10	quality	cov@10	quality	cov@10	quality	cov@10
Llama-3 8B	0.54@0.4	1.00@0.4	0.11@0.8*	0.45@0.8	0.45@0.0	0.71@0.4	0.32@0.0	0.53@0.6*
Llama-3 8B Instruct	0.49@0.4	1.00@0.4	0.14@1.0*	0.53@1.0*	0.45@0.2	0.67@0.8	0.29@0.0	0.53@0.8*
Llama-3.1 8B	0.52@0.6	1.00@0.4*	0.52@0.0*	0.76@0.6	0.43@0.0	0.72@0.4*	0.29@0.0*	0.50@0.6*
Llama-3.1 8B Instruct	0.48@0.4*	1.00@0.4	0.12@0.8	0.39@0.8	0.61@0.2	0.86@0.8	0.27@0.2	0.52@0.6
Llama-3.2 1B	0.20@0.2	0.97@0.4*	0.10@0.0*	0.11@0.8	0.12@0.2	0.30@0.6	0.06@0.0	0.14@0.6
Llama-3.2 1B Instruct	0.16@0.2*	0.91@0.4*	0.09@0.0*	0.12@0.8	0.34@0.0	0.50@0.6	0.07@0.2	0.17@0.4

* the peak temperature significantly beats the runner-up temperature (paired bootstrap over the shared instances, 95%; the per-instance pairing cancels shared difficulty, so this is sharper than comparing the marginal bands above). Unstarred peaks are statistical ties with their neighbour.

Failure modes

What the failures actually are, and how that changes with temperature and model size. Browse the raw generations behind every figure in the explorer.

Outcome of every sample, stacked to 1. Green is success (correct, or a valid molecule); amber is coherent but wrong (a goal that is mostly right, or SQL that runs but returns the wrong rows); red is malformed (unparseable SMILES, garbage PDDL, or SQL that will not execute). Categories were validated against the real evaluators (goal 98.9%, spider 96.1% agreement).

At low temperature the 8B models fail by being coherent but wrong (amber): a goal with one wrong predicate, or SQL that runs but returns the wrong rows. As temperature rises the failures turn malformed (red): unparseable SMILES, garbage PDDL, SQL that will not run.
The 1B models barely have an amber band. They produce malformed output even when cold. The 1B is only about 25 to 30% valid on molecules at greedy, and its goals are mostly garbage at every temperature.
For molecules, the quality of the valid ones (QED) is roughly flat across temperature. The whole battle is producing a parseable molecule at all, which is exactly the slice a grammar or constrained decoder removes.

What the categories look like

Real model outputs, with the gold answer for comparison.

Goal inference (PDDL goal predicates)

correct Your goal is to have the following: Your arm should be empty. b1 should be clear. b1 should be on the table.

gold (arm-empty) (clear b1) (on-table b1)
model (arm-empty) (clear b1) (on-table b1)

coherent but wrong Your goal is to have the following: Your arm should be empty. b2 should be on b1. b3 should be on b2. b3 shoul

gold (arm-empty) (clear b3) (on b2 b1) (on b3 b2) (on-table b1)
model (clear b3) (empty b1) (on b2 b1) (on b3 b2) (on-table b1)

malformed Your goal is to have the following: Your arm should be empty. b1 should be clear. b1 should be on the table.

gold (arm-empty) (clear b1) (on-table b1)
model (arm-empty) (stacked b1)

Molecules (SMILES)

valid molecule CC1CN(C)C2CCC(O)C(O)C2C1=O QED 0.577

malformed (unparseable) CC(=O)N1C=CC(=O)C1(C2C(O)C(C1O)C2=O)C(=O)O

Text-to-SQL (Spider)

correct Find the id of the pet owned by student whose last name is ‘Smith’.

gold SELECT T2.petid FROM student AS T1 JOIN has_pet AS T2 ON T1.stuid = T2.stuid WHERE T1.Lname = 'Smith'
model SELECT petid FROM Has_Pet AS T1 JOIN Student AS T2 ON T1.Stuid = T2.StuID WHERE T2.Lname = 'Smith'

coherent but wrong (wrong rows) What are the names and release years for all the songs of the youngest singer?

gold SELECT song_name , song_release_year FROM singer ORDER BY age LIMIT 1
model SELECT name , song_release_year FROM singer WHERE age = (SELECT min(age) FROM singer)

malformed (will not execute) What are the different countries with singers above age 20?

gold SELECT DISTINCT country FROM singer WHERE age > 20
model SELECT DISTINCT country FROM singer WHERE age 20

Method: raw vLLM sampling, official genlm-eval prompts & evaluators. Scoring. molecules: RDKit QED (validity = parseable SMILES); goal: PDDL equivalence (planetarium); spider: SQL execution match; DS-1000: official test harness (test_execution + test_string). pass@k = unbiased estimator. n = 100 instances for the new domains, 1000 for DS-1000; 100 samples each (1 for greedy).