Spider 2.0-Snow: what makes an instance hard?

Interim findings from an all-pairs execution study of the vxef/spider2-snow-temperature-sweep rollouts — Qwen3 {1.7B, 4B, 8B} × {thinking, no-thinking}, 100 samples per instance per cell, 523 grounded instances. Every distinct generated SQL was quoting-normalized (identifiers quoted, case preserved), compiled with Snowflake EXPLAIN, and every compile survivor was executed and scored against gold (Spider 2's own comparison); the raw as-is SQL got its own EXPLAIN sweep. All records are in the dataset PR and browsable row-by-row in the explorer (quoted-verdict filter + expected-vs-got diffs). July 16, 2026.

1 · pass@k by model and reasoning arm

The topline: mean pass@1 across instances, per cell (each cell = one model size × one reasoning arm, 100 quoting-normalized samples per instance; error bars are 95% bootstrap CIs over instances). Thinking roughly doubles-to-triples accuracy at every size — 4B-think (4.3%) beats 8B-no-think (2.8%), so the reasoning arm is worth more than a size doubling here — and the best cell, 8B-think, still sits at 6.9%. The right chart shows what a sampling budget buys: the expected share of instances solved at k samples (mean unbiased pass@k); at k = 100 the best cell reaches ~22% of instances, and the curves are still rising — sampling is nowhere near exhausted on the solvable set.

2 · The funnel: two walls, not one

Reading each cell's samples left to right: finish → parse (sqlglot) → static grounding → compile (Snowflake EXPLAIN) → ran → matched gold (≡ pass@1). The compile stage is not another static check: EXPLAIN is a dry run through Snowflake's actual query compiler against the live account catalog — parsing in the real dialect, resolving every table/column name under the engine's quoting and case-folding rules, type-checking expressions and function signatures, checking privileges — everything short of dispatching the plan for execution (it runs in the cloud-services layer, no warehouse needed). That makes it strictly stronger than client-side static analysis, and weaker than execution only on runtime, data-dependent failures (division by zero, casts on bad values, timeouts), which the “ran” stage then catches. The “compiles (as-is)” group is the raw SQL's compile rate — formatting alone (unquoted case-sensitive identifiers) kills 97–99% of every cell's raw queries before anything else can matter; the adjacent “compiles (quoted)” group shows the same queries after quoting normalization. What remains of the compile wall is semantic. The bar chart below is a failure-mode census of the 142k distinct failing (instance, SQL) pairs — the union across cells, deduped because identical SQL compiles once; it characterizes the corpus of errors, not any one model: 43% reference a nonexistent column and 29% a nonexistent table — schema-linking failures, not dialect (5%) or syntax (3%). Browse compile failures →

3 · Per-instance pass rates — a consistent frontier

Each cell's instances ranked by pass@1 (that cell's 100 samples; only instances with ≥1 pass drawn — the counts at zero are in the hover). Each point is an instance; click to open it in the explorer. (We deliberately don't plot pass@1 against pass@k for the same pool — both are deterministic functions of the same pass count, so that “relationship” is a parametric curve with no information beyond this ranking.) The solvable sets are nested: 90–100% of every weaker cell's solvable instances are contained in 8B-think's 114 (per-instance pass@1 Spearman 0.83 between 4B-think and 8B-think) — capability growth widens one frontier, it does not unlock different problems. Per-instance whiskers are 95% bootstrap CIs from resampling that instance's own 100 samples with replacement (equivalently c* ~ Binomial(100, c/100)); lower bounds of 0 are truncated at the axis floor, and CIs this near c = 0 are rough by nature.

4 · What predicts hardness (and what doesn't)

Mean pass@k across instances per covariate bin, per cell; error bars are 95% bootstrap CIs resampling instances within bin. The cell's own compile rate is the strongest predictor (schema-linking again; solvable-vs-not AUC 0.86 in the 8B-think cell). Self-inconsistency (distinct-SQL rate) is the best behavioral tell. Question length acts as a threshold, not a slope. Static column-grounding — the flag the dataset ships — is ~uninformative (AUC 0.51, 8B-think). Thinking-token volume predicts nothing for 4B/8B; explicit date/window phrasing drops match-given-compiled from 0.23 to 0.11 (8B-think; same sign in 4B-think). Beyond compiling, misses are far off, not near-misses — a census of the ~30k distinct ran-but-wrong executions (union across cells): 31% have exactly the gold's shape with wrong values, and unit/rounding confusions are absent (1 case of ×100 in 3,355 scalar comparisons).

5 · The tantalizing stratum: compiles often, never matches

177 instances (8B-think) have ≥10 compiled samples and zero matches — the analytic-composition wall. They read like multi-step aggregation with precise windows and exact output contracts. This is the stratum where execution-feedback methods have signal to amplify; the never-compiles stratum needs grounding (constrained decoding over schema identifiers) instead.