Looking at the 16 languages exposed by livecodebench_multilingual in genlm-eval, profiled across the design axes that matter for picking a diverse set, every claim sourced.
drag a column edge to resize it · the header row and language column stay pinned while you scroll · tier dots: ● Multi-LCB · ● Agnostics low-resource · ● Agnostics high-risk ?
| Language | Paradigm | Typing | Type inference | Discipline strength · style | Garbage collection | Scope | Macros / metaprog. | Exec | Training data GB · files · v2-dedup ↗ | Popularity TIOBE · SO'24 | Distinctive |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Tier 1 — Multi-LCB (11) | |||||||||||
| Multi-paradigmOOP · procedural · functional · reflective | dynamic | None hints are external (mypy), not runtime | Strong · ducknominal for user classes | refcount + cycle collector | Lexical (LEGB)no block scope | nonedecorators · metaclasses | VM | HUGE 233.3 GB · 56.9M |
#1 18.96% SO 51% |
Duck typing + significant indentation | |
| Multi-paradigmprocedural · OOP · generic · functional | static | Local auto / decltype (C++11) |
Strong · nominalpervasive implicit conversions → "loosely type-safe" | no GCmanual · RAII · smart ptrs | Lexical · blockstorage duration = RAII | yespreprocessor + templates (TMP) + constexpr | compiled | HUGE 353.9 GB · 43.2M |
#3 8.03% SO 23% |
RAII — deterministic scope-bound resources | |
| Class-based OOP+ generic · functional (lambdas/streams) | static | Local var (10), diamond <> |
Strong · nominalno duck typing | tracing, generational (G1/ZGC) | Lexical · blocklambdas capture effectively-final | noneannotations + processors (JSR 269) | VM/JIT | HUGE 548.0 GB · 154.3M |
#4 7.90% SO 30.3% |
Write-once-run-anywhere bytecode | |
| Multi-paradigm OOPfunctional · generic · component · event | staticopt-in dynamic type |
Local var, target-typed new |
Strong · nominaldynamic = duck-typing escape hatch |
tracing, generational (gen 0/1/2) | Lexical · blocklambdas capture variables | noneattributes · Source Generators (Roslyn) | VM/JIT | HUGE 239.9 GB · 51.2M |
#5 4.85% SO 27.1% |
LINQ — type-checked integrated queries | |
| Imperative + concurrentno classes/inheritance | static | Local :=; generic call-site inference |
Strong · structural interfacesnominal named types · no implicit numeric coercion | concurrent tracing mark-sweep | Lexical · blockshadowing | nonego:generate (convention) · generics |
compiled | LARGE 54.6 GB · 9.3M |
#13 1.20% SO 13.5% |
Goroutines + channels (CSP) | |
| Multi-paradigmevent-driven · functional · prototype-OOP | dynamic | None no static type inference | Weak · duckimplicit coercion (==, +) · prototype model |
tracing mark-sweep (gen/incremental) | Lexicalvar function-scope vs let/const block · closures |
noneeval · transpilers (Babel) |
VM/JIT | HUGE 1,115 GB* · 108.9M |
#6 3.04% SO 62.3% |
Prototypal inheritance + single-thread event loop | |
| Multi-paradigmstrict superset of JS + static types | staticerased at compile | Extensivelocal + contextual (best-common-type) | Strong · structuralgradual; any escape hatch |
via JS engine (V8 gen. mark-sweep) | Lexical (JS)let/const block · closures |
nonedecorators · type-level programming (erased) | →JS | LARGE 61.0 GB · 23.9M |
#37 0.43% SO 38.5% |
Gradual structural typing over JS, fully erased | |
| Multi-paradigmfunctional-influenced · OOP via traits | static | Extensive (local)HM-based; signatures annotated | Strong · nominaltraits · no implicit numeric coercion | no GCownership + borrow checker · RAII · opt. Rc/Arc | Lexical · blockdeterministic drop | yeshygienic macro_rules! + procedural macros |
compiled | MEDIUM 15.6 GB · 2.2M |
#12 1.26% SO 12.6% |
Ownership + borrow checker = safety w/o GC | |
| Pure OOPeverything is an object · functional (blocks) | dynamic | None RBS / Sorbet are add-ons | Strong · duckno silent cross-type coercion | mark-sweep · generational + incremental | Lexicaldef/class/module = scope gates · blocks close over |
noneruntime metaprog. (method_missing) |
VM | MEDIUM 31.7 GB · 17.8M |
#33 0.52% SO 5.2% |
Blocks/closures on every method + uniform object model | |
| Imperative + OOPprocedural web-scripting origins | dynamicopt. type declarations · strict_types |
None no static type inference | Weak · duck-ishtype juggling ("0"==0) · === strict |
refcount + cyclic collector (5.3+) | Function-localno auto global access (need global) · superglobals |
noneruntime reflection · magic methods · eval |
VM | HUGE 224.6 GB · 46.0M |
#14 1.17% SO 18.2% |
HTML-embedded per-request server scripting | |
| Multi-paradigmOOP + functional · imperative · generic | static | Local + function-signature, bidirectional (not global HM) | Strong · nominalnull-safety in the type system | JVM collector · Native CMS | Lexical · blockclosures | noneinline + reified generics · KSP |
VM/JIT | SMALL 5.8 GB† · 2.2M |
#23 0.71% SO 9.4% |
Null safety enforced at compile time | |
| Tier 2–3 — Agnostics low-resource (5) | |||||||||||
| Multi-paradigm scriptingprocedural · OOP via metatables · functional | dynamic | None no static type inference | Strong-ish · duckmetatables · narrow coercions (string↔number) | incremental mark-sweep + generational (5.4) | Lexicalclosures with upvalues | nonemetatables / metamethods (runtime) | VM | MEDIUM 33.9 GB · 2.4M |
#31 0.53% SO 6.2% |
Metatables as the single extensibility mechanism | |
| Multiple dispatch+ functional · array · metaprogramming | dynamicoptional annotations | Extensivefor JIT specialization (not a static guarantee) | Strong · nominal + parametricdeep dispatch on argument types | concurrent, parallel, generational mark-sweep | Lexical | yestrue Lisp-style homoiconic macros (@, AST-level) |
JIT | SMALL 6.1 GB · 0.45M |
#26 0.57% SO 1.1% |
Multiple dispatch + parametric types + Lisp macros | |
| Functional + arraymultiple OOP systems (S3/S4/R5/R6) | dynamic | None pervasive runtime coercion | Implicit coercionatomic vectors: char→double→int→logical | generational (3 levels) + write barrier | Lexicaladopted from Scheme (departure from S) | nonecomputing on the language: quote/eval/substitute · lazy promises |
interp. | MEDIUM 22.4 GB · 5.2Mv1-dedup only 0.30 GB |
#9 1.69% SO 4.3% |
Lazy evaluation via promises → pervasive NSE | |
| Functional-first+ imperative + OOP | static | Full Hindley-Milnerprincipal types; params need no annotation | Strong · structuralobjects/poly-variants structural; records/variants nominal | generational · incremental · precise | Lexicalmodule system + functors | none built-inPPX AST preprocessors (4.02+) | compiled | TINY 1.1 GB† · 0.16M |
#43 0.29% SO 0.8% |
Full type inference + ADTs + functor module system | |
| Imperative + arrayOOP since F2003 · numeric/scientific | staticstrong · manifest | None implicit typing legacy (I–N = INTEGER); IMPLICIT NONE modern |
Strong · manifest nominalsome implicit numeric conversions | no GCmanual; ALLOCATABLE auto-freed at scope (not GC) |
Lexical by unithost / use association · legacy COMMON |
none nativeexternal C preprocessor (cpp/fpp) for .F90 |
compiled | SMALL 4.7 GB · 0.27Mv1-dedup only 0.17 GB |
#17 0.95% SO 1.1% |
Array/numeric focus + implicit-typing legacy | |
How the 16 split on each axis lopsided axes show which corners are rare — and worth deliberately including
Choose the following subset to roughly maximize axis coverage with the fewest languages (with data volume to span the spectrum huge → medium → small → tiny).
Native LCB language; the high-resource dynamic baseline.
The rare corner: no GC + a real hygienic macro system + strong nominal static typing.
Functional + full inference + genuinely tiny data — maximally distant from Python.
A paradigm no other harness language has: multiple dispatch + Lisp macros.
The numeric/legacy extreme: no GC, no inference, no macros, low data.
Python → Rust → Julia → OCaml walks the data axis huge → medium → small → tiny while also covering static/dynamic, GC/no-GC, the macro languages (Rust, Julia), full-HM inference (OCaml), structural vs nominal vs duck, and four paradigm families. Cut to three by dropping Julia; add Fortran for the numeric/no-inference corner.
The colored dot on each language marks where it comes from and how the harness groups it.
How each column is sourced — and why the training-data numbers are a defensible estimate of relative pretraining exposure rather than a measurement of any one model's diet.
How well do these models write OCaml? We ran the four instruct/chat models plus the two Qwen3 reasoning (think) cells through livecodebench_multilingual's OCaml track (436 problems, exact-match grading) and compared the pass rates against the same models' Python LiveCodeBench numbers. The short version: OCaml pass@1 sits at a few percent, and the gap to Python is mostly about writing OCaml that compiles, not about reasoning (see the error analysis below).
livecodebench_multilingual's 436 OCaml problems (competitive-programming tasks, stdin→stdout), graded by exact match on the whole output.| Llama-3.1 8B Instruct | llama31-8b-instruct-nothink |
| Qwen2.5-Coder 7B Instruct | qwen25-coder-7b-instruct-nothink |
| Qwen3 8B | qwen3-8b-nothink |
| Qwen3 14B | qwen3-14b-nothink |
The four base rows are the instruct / chat (non-thinking) variants; the two Qwen3 think cells (reasoning mode, generated at t=0.6 only) appear in bold / as diamonds. Sampling temperature t=0.6 unless noted.
Pass@k versus temperature on all 436 OCaml problems; diamonds are the Qwen3 think cells (t=0.6 only). Non-thinking pass@1 sits at roughly 2–9% and is highest at greedy decoding; sampling helps pass@10/50 a little.
OCaml pass@1 over all 436 OCaml problems against Python pass@1 over the 714-problem Python window, at t=0.6. This is a directional language-capability gap, not a controlled per-problem comparison: the two pools are different sets and sizes and use different graders (exact vs functional). Read it as order-of-magnitude.
| Model | OCaml pass@1436 problems · exact | Python LCB pass@1714 problems · functional | Python / OCaml |
|---|---|---|---|
| Llama-3.1 8B Instruct | 2.9% | 17.6% | 6.0× |
| Qwen2.5-Coder 7B Instruct | 5.3% | 33.5% | 6.3× |
| Qwen3 8B | 6.3% | 47.5% | 7.6× |
| Qwen3 14B | 7.7% | 52.0% | 6.7× |
| Qwen3 8B think | 21.0% | 81.4% | 3.9× |
| Qwen3 14B think | 15.6% | 84.7% | 5.4× |
The OCaml campaign drew from a wider LiveCodeBench pool than our Python window, so only 222 of the 436 OCaml problems are also in the Python set. Restricting both sides to those shared ids gives a controlled comparison: same problems, OCaml versus Python, per-problem pass@1 with 95% bootstrap CIs.
| Model | OCaml pass@1222 shared | Python pass@1222 shared | Python pass@1714 full | Python / OCamlshared |
|---|---|---|---|---|
| Llama-3.1 8B Instruct | 2.6% [1.4, 4.1] | 16.7% [12.7, 20.7] | 17.6% | 6.4× |
| Qwen2.5-Coder 7B Instruct | 6.3% [3.9, 9.1] | 23.2% [18.3, 28.2] | 33.5% | 3.7× |
| Qwen3 8B | 7.4% [4.9, 10.5] | 38.1% [32.5, 44.3] | 47.5% | 5.1× |
| Qwen3 14B | 8.1% [5.4, 11.2] | 42.6% [36.8, 48.8] | 52.0% | 5.3× |
| Qwen3 8B think | 24.2% [20.2, 28.5] | 73.7% [68.2, 79.0] | 81.4% | 3.0× |
| Qwen3 14B think | 16.6% [12.8, 20.7] | 75.5% [70.1, 80.7] | 84.7% | 4.5× |
The reasoning arm (Qwen3, t=0.6). Thinking lifts OCaml pass@1 substantially for both sizes, but with a twist in how the two sizes compare.
| Model | Arm | pass@1 | pass@10 | pass@50 | pass@1 lift |
|---|---|---|---|---|---|
| Qwen3 8B | nothink | 6.3% | 14.8% | 20.8% | +14.7pp (3.3×) |
| think | 21.0% | 42.2% | 52.2% | ||
| Qwen3 14B | nothink | 7.7% | 15.9% | 21.4% | +7.9pp (2.0×) |
| think | 15.6% | 30.1% | 42.5% |
Failure-mode mix for each reasoning cell (share of failing completions, full execution record). The 14B's failures skew to runtime crashes; the 8B's skew to truncation and compile errors.
| Failure mode | Qwen3 8B (think) | Qwen3 14B (think) |
|---|---|---|
| Compile error | 68.8% | 67.2% |
| Runtime crash | 8.2% | 20.9% |
| Wrong answer | 5.9% | 3.3% |
| Empty / truncated | 15.6% | 8.2% |
| Timeout | 1.6% | 0.4% |
To see why OCaml is hard, we classified all 164,683 failing completions across these four models from the full per-test execution record (the compiler / runtime output the pass/fail score throws away). Each failure is bucketed by how far the code got: did it parse, compile, run, finish?
Share of each model's compile errors by the exact message the OCaml toolchain printed. Genuine syntax errors, unbound names, and type errors each take a big slice: the models write plausible-looking OCaml that does not parse or type-check. (Syntax errors include generations cut off at the token limit.)
| Compiler message | Llama-3.1 8B Instruct | Qwen2.5-Coder 7B Instruct | Qwen3 8B | Qwen3 14B |
|---|---|---|---|---|
| Syntax error | 24% | 17% | 43% | 32% |
| Type mismatch (value) | 23% | 34% | 17% | 18% |
| Unbound value | 21% | 24% | 17% | 22% |
| Type mismatch (expression) | 16% | 11% | 8% | 9% |
| Unbound constructor | 3% | 3% | 7% | 10% |
| Other compile error | 6% | 2% | 3% | 4% |
| Type mismatch (constant) | 2% | 1% | 2% | 2% |
| Unbound module | 2% | 2% | 0% | 2% |
| Type mismatch (pattern) | 1% | 2% | 2% | 0% |
| Wrong type expected | 2% | 2% | 0% | 1% |
A fifth of compile errors are Unbound value: the model calls something that is not in OCaml's standard library. The identifiers it reaches for most (count = times seen in the sampled failures) are mostly Python / C habits (continue and break are not OCaml keywords; String.to_list, String.filter, Array.create, Hashtbl.find_default do not exist; printf needs Printf.) and mis-remembered module paths (Scanf.stdin).
Scanf.stdin1305String.to_list1023printf731Array.create720continue637find558String.filter524int64_of_string492Scanf.Scanner453Hashtbl.find_default435break345List.of_string335Scanf.Scanning.fscanf330Core326String.rev325Array.of_string307Char.is_uppercase300List.last255scanf251String.set238Scanf.Scanning.scanf224Scanf.Scanning.bounded_int_list223Scanf.fscanf209List.range193List.sub187String.split180solve177String.count169Share of each model's failing completions that compiled but then crashed. Scanf input-parse crash dominates: the model mis-reads stdin.
| Sub-reason | Llama-3.1 8B Instruct | Qwen2.5-Coder 7B Instruct | Qwen3 8B | Qwen3 14B |
|---|---|---|---|---|
| Scanf input-parse crash | 1.2% | 19.7% | 0.4% | 8.1% |
| End_of_file (read past input) | 0.5% | 2.9% | 1.0% | 2.5% |
| Index out of bounds | 0.3% | 0.7% | 0.7% | 0.4% |
| Invalid_argument | 0.1% | 0.3% | 0.1% | 0.1% |
| Non-exhaustive match | 0.0% | 0.0% | 0.1% | – |
| Not_found | 0.0% | 0.0% | 0.0% | 0.1% |
| Stack overflow | 0.2% | 0.2% | 0.2% | 0.0% |
| Other runtime error | 0.1% | 0.0% | 0.0% | 0.1% |
One real failing completion per failure mode (click to expand the model's OCaml and the exact compiler or runtime message).
(* Read input from stdin *)
let read_int () = Scanf.scanf "%d" (fun x -> x)
let read_int_list () = Scanf.Scanner.scan_string "%d " >> fun s -> List.map int_of_string (Str.split (Str.regexp " ") s) |> List.filter (fun x -> x <> 0)
(* Main function *)
let rec solve n scores =
if n = 1 then scores.(n - 1) else
let sum = List.fold_left (fun acc x -> acc + x) 0 scores in
let avg = sum / (n - 1) in
avg +. (solve (n - 1) scores)
(* Read input and print result *)
let () =
let n = read_int () in
let scores = read_int_list () in
let result = solve n scores in
Printf.printf "%+.0f\n" resultFile "Main.ml", line 3, characters 55-57:
3 | let read_int_list () = Scanf.Scanner.scan_string "%d " >> fun s -> List.map int_of_string (Str.split (Str.regexp " ") s) |> List.filter (fun x -> x <> 0)
^^
Error: Unbound value (>>)(* Problem: Move the K-th 1-block from the beginning in S to immediately after the (K-1)-th 1-block *)
(* Read input from standard input *)
let () =
let _ = Scanf.scanf "%d %d\n" (fun n k -> ()) in
let s = Scanf.scanf "%s\n" (fun s -> s) () in
let n = String.length s in
(* Find all 1-blocks in the string *)
let rec find_1_blocks i acc =
if i = n then acc
else if s.[i] = '1' && (i = 0 || s.[i-1] = '0') then
find_1_blocks (i + 1) ((i, i) :: acc)
else find_1_blocks (i + 1) acc
in
let blocks = find_1_blocks 0 [] in
(* Move the K-th 1-block to immediately after the (K-1)-th 1-block *)
let rec move_block blocks k =
match blocks with
| [] -> []
| _ when k = 1 -> blocks
| (l1, r1) :: (l2, r2) :: tl ->
if k = 2 then (l1, r1) :: (l2, r2 + (r2 - l2) + 1) :: tl
else (l1, r1) :: move_block ((l2, r2) :: tl) (k - 1)
in
let new_blocks = move_block blocks 3 in
(* Create the resulting string *)
let rec create_string i acc =
match new_blocks with
| [] -> acc
| (l, r) :: tl ->
let len = r - l + 1 in
let prefix = String.sub s 0 l in
let block = String.make len '1' in
let suffix = String.sub s (l + len) (n - l - len) in
create_string (l + len) (prefix ^ block ^ suffix ^ acc)
in
let result = create_string 0 "" in
Printf.printf "%s\n" resultFile "Main.ml", line 6, characters 39-40:
6 | let s = Scanf.scanf "%s\n" (fun s -> s) () in
^
Error: The value s has type string but an expression was expected of type(* Problem: Given two lists of integers P and Q of length N,
find the number written on the bib of the person that the person wearing
the bib with number i is staring at. *)
(* Input: N, P, Q *)
(* Output: S where S_i is the number written on the bib of the person
that the person wearing the bib with number i is staring at. *)
(* Read input from standard input *)
let read_int () = Scanf.scanf "%d" (fun x -> x)
let read_int_list n = List.init n read_int
let () = Scanf scanf "%d" read_int_list
let n = read_int ()
let p = read_int_list n
let q = read_int_list n
(* Create a list of pairs (i, P_i) *)
let p_list = List.mapi (fun i x -> (x, i)) p
(* Create a map from P_i to i *)
let p_map = List.fold_left (fun acc (x, i) -> Map.put acc x i) Map.empty p_list
(* Create a list of S_i *)
let s = List.map (fun x -> Map.find x (p_map)) q
(* Print the list of S_i *)
let () = List.iter (fun x -> Printf.printf "%d " x) s;
Printf.printf "\n"File "Main.ml", line 11, characters 21-25:
11 | let () = Scanf scanf "%d" read_int_list
^^^^
Error: Syntax error(* Problem: Find the maximum length of a strictly increasing subsequence of a given sequence
with elements at most X_i, for Q queries. *)
(* Read input from standard input *)
let () =
let n, q = Scanf.scanf "%d %d\n" (fun n q -> n, q) in
let a = Array.init n (fun _ -> Scanf.scanf "%d" (fun x -> x)) in
let queries = Array.init q (fun _ ->
let r, x = Scanf.scanf "%d %d\n" (fun r x -> r, x) in
r - 1, x
) in
(* Process queries *)
let results = Array.init q (fun i ->
let r, x = queries.(i) in
let rec find_max_length r x acc =
if r < 0 then acc
else
let ai = a.(r) in
if ai > x then find_max_length (r - 1) x acc
else
let new_acc = if ai <= acc then acc + 1 else acc in
find_max_length (r - 1) x new_acc
in
find_max_length r x 0
) in
(* Print results *)
List.iter (fun x -> Printf.printf "%d\n" x) (Array.to_list results)Fatal error: exception Stdlib.Scanf.Scan_failure("scanf: bad input at char number 5: character ' ' is not a decimal digit")(* Problem: Check if two strings are equal, if not print the position of the first difference *)
(* Input: Two strings S and T *)
(* Output: If S and T are equal, print 0; otherwise, print the position of the first character where they differ *)
(* Read two strings from the input *)
let () =
let s = read_line () in
let t = read_line () in
(* Compare the strings *)
let rec compare_strings s t pos =
if pos > min (String.length s) (String.length t) then
(* If one string is a prefix of the other, print the position of the first differing character *)
if String.length s < String.length t then
Printf.printf "%d\n" (String.length s + 1)
else if String.length s > String.length t then
Printf.printf "%d\n" (String.length t + 1)
else
Printf.printf "0\n"
else if s.[pos] <> t.[pos] then
(* If the characters at the current position are different, print the position *)
Printf.printf "%d\n" (pos + 1)
else
(* If the characters are the same, continue comparing the rest of the strings *)
compare_strings s t (pos + 1)
in
(* Start comparing the strings from the first position *)
compare_strings s t 0Fatal error: exception Invalid_argument("index out of bounds")(* This program prints all triples of non-negative integers (x,y,z) such that x+y+z <= N in ascending lexicographical order. *)
(* Read an integer from standard input *)
let read_int () =
Scanf.scanf "%d" (fun n -> n)
(* Generate all triples (x, y, z) such that x + y + z <= N *)
let generate_triples n =
let rec generate x y =
if x + y > n then []
else
(x, y, n - x - y) :: generate x (y + 1) @ generate (x + 1) y
in
generate 0 0
(* Print the triples in ascending lexicographical order *)
let print_triples triples =
List.iter (fun (x, y, z) ->
Printf.printf "%d %d %d\n" x y z)
triples
(* Main function *)
let () =
let n = read_int () in
let triples = generate_triples n in
print_triples triples(* Problem: Takahashi's Ticket Gate Records *)
(* Author: [Your Name] *)
(* Date: [Today's Date] *)
(* Function to calculate the minimum number of characters to be inserted *)
let rec min_insertions s =
let rec loop i count =
if i >= String.length s then
count
else if i mod 2 = 0 then
(* If the current character is at an even position, check the next character *)
if i + 1 < String.length s && s.[i] = 'o' && s.[i + 1] = 'i' then
loop (i + 2) count
else
loop (i + 1) (count + 1)
else
(* If the current character is at an odd position, check the previous character *)
if i - 1 >= 0 && s.[i] = 'i' && s.[i - 1] = 'o' then
loop (i - 2) count
else
loop (i - 1) (count + 1)
in
loop 0 0
(* Main function *)
let () =
(* Read the input string from standard input *)
let s = Scanf.scanf "%s" (fun s -> s) in
(* Calculate and print the minimum number of characters to be inserted *)
Printf.printf "%d\n" (min_insertions s)[<class 'subprocess.TimeoutExpired'>][Command '['/scratch/tmp.5294624.skiegeland/tmpj7pwtd53/Main.exe']' timed out after 8 seconds]
empty or unparsed code
Most failing samples never compile, so an obvious lever is to discard malformed code before choosing what to submit. We test two static checks, neither of which looks at the tests: the grammar filter rejects code the OCaml parser cannot parse (a viable-prefix check on the incremental Menhir parser), and the compile filter rejects code that ocamlc refuses to compile (syntax, type, and unbound-name errors). Both are safe by construction here: across all four cells, no passing sample was ever rejected. ~100 samples per problem, all 436 problems.
| Cell (t=0.6) | Filter | Zacceptance rate | skyline@1all samples → accepted only | skyline@10all samples → accepted only | skyline@50all samples → accepted only | gap closed@1, of pass@1 → oracle | oraclecoverage ceiling |
|---|---|---|---|---|---|---|---|
| Qwen3 8B nothink | grammar | 63.6% | 6.3% → 7.3% | 14.8% → 16.3% | 21.0% → 22.0% | 6.1% | 23.4% |
| compile | 20.3% | 6.3% → 12.9% | 14.8% → 21.0% | 21.0% → 23.2% | 38.8% | ||
| Qwen3 8B think | grammar | 64.3% | 21.0% → 23.6% | 42.2% → 45.7% | 52.2% → 54.8% | 7.4% | 56.2% |
| compile | 33.8% | 21.0% → 35.4% | 42.2% → 53.1% | 52.2% → 56.1% | 41.1% | ||
| Qwen3 14B nothink | grammar | 73.3% | 7.8% → 8.3% | 16.2% → 16.9% | 21.8% → 22.6% | 3.1% | 24.3% |
| compile | 24.2% | 7.8% → 12.8% | 16.2% → 21.0% | 21.8% → 24.2% | 30.6% | ||
| Qwen3 14B think | grammar | 73.0% | 15.6% → 16.0% | 30.1% → 31.5% | 42.5% → 44.7% | 1.3% | 48.2% |
| compile | 35.9% | 15.6% → 25.1% | 30.1% → 40.1% | 42.5% → 47.3% | 29.1% |
Z is the share of samples a filter accepts. skyline@k is pass@k among accepted samples only: the score of sampling until k accepted candidates are found, which costs about k/Z generations. The value left of each arrow is the ordinary pass@k over all samples. oracle is coverage, the score of a perfect selector over the same ~100 samples, an upper bound no filter can exceed. Neither filter ever rejects a correct sample, so per problem pass@1 = Z × skyline. Averages are over all 436 problems; problems where a filter accepts nothing count as 0. Same terminology as the DS-1000 and LCB skyline pages.
Keeping only code that compiles roughly doubles pass@1; a grammar check alone adds little. On the 8B thinking cell, the compile filter lifts pass@1 from 21.0% to 35.4% (41.1% of the way to the oracle), at a cost of about 3 generations per kept sample (Z = 33.8%). The grammar filter is nearly free but gains at most +2.6pp: correct samples are already grammatical, and the dominant failure type, type and unbound-name errors (60–71% of the parseable failures), is invisible to a parser but caught by ocamlc without running anything. Under 1% of syntax errors slip past the grammar check.
Filtering selects among the samples you already have, so no filter can beat the oracle column, and its advantage shrinks as the budget grows: with one sample there is nothing to choose between, and by 50 accepted samples the compile filter is within 0.1pp of the oracle. The failures that survive compilation, wrong answers and runtime crashes (30.5% of the 14B thinking cell's residual, mostly Scanf input parsing), can only be caught by running the code.
Caveats: OCaml grading is exact-match on a 436-problem subset; Python LCB grading is the functional harness on the 714-problem window, so the full-set ratio mixes set, size and grader differences (the matched 222-problem comparison removes the set/size difference). The mLCB evaluation was produced by the deprecated eval_mlcb.py; the numbers are usable here but are provisional, to be re-confirmed by the canonical genlm-eval grader.
The second mLCB source (clementepasti/mlcb-rollouts): the same LCB problems in Fortran and COBOL — 150 of the 436 mLCB problems, Qwen3 4B/8B/14B thinking cells only (no non-thinking arm exists for these languages), t=0.6, 100 samples per problem. All numbers below are recomputed from the published per-test executions with the same pass@k estimator as the rest of the site; nothing is read from precomputed metric files.
| Language | Model | pass@1150 problems · n=100 | pass@10150 problems · n=100 | pass@50150 problems · n=100 |
|---|---|---|---|---|
| Fortran | Qwen3 4B think | 11.4% [7.9, 15.2] | 26.4% [20.2, 33.0] | 33.4% [26.2, 40.4] |
| Qwen3 8B think | 18.2% [13.3, 23.2] | 32.6% [25.8, 39.5] | 42.1% [34.8, 49.7] | |
| Qwen3 14B think | 29.6% [23.4, 36.2] | 46.4% [39.2, 53.7] | 59.2% [51.9, 66.5] | |
| COBOL | Qwen3 4B think | 0.2% [0.1, 0.4] | 1.8% [0.6, 3.2] | 4.8% [2.0, 7.9] |
| Qwen3 8B think | 4.3% [2.4, 6.5] | 13.5% [8.9, 18.6] | 19.7% [13.9, 26.0] | |
| Qwen3 14B think | 4.8% [2.8, 7.2] | 16.3% [11.3, 21.6] | 25.0% [18.3, 31.8] |
The same 150 problems exist in all four language pools, which gives a genuinely matched cross-language comparison (Python from the thinking-rollouts LCB cells; OCaml recomputed from its executions on the subset — the 4B has no OCaml cell):
| Model | Python pass@1150 shared | OCaml pass@1150 shared | Fortran pass@1150 shared | COBOL pass@1150 shared |
|---|---|---|---|---|
| Qwen3 4B think | 80.0% [74.0, 85.3] | – | 11.4% [7.9, 15.2] | 0.2% [0.1, 0.4] |
| Qwen3 8B think | 81.6% [75.7, 87.0] | 25.9% [21.0, 31.1] | 18.2% [13.3, 23.2] | 4.3% [2.4, 6.5] |
| Qwen3 14B think | 83.4% [77.8, 88.8] | 18.1% [13.5, 22.9] | 29.6% [23.4, 36.2] | 4.8% [2.8, 7.2] |
Share of all samples per outcome, from the per-test error codes (same categories as the explorer's failure filter). Compile errors dominate both languages, COBOL overwhelmingly so; the last column is the slice of wrong answers whose every failing test matches the expected output after normalizing whitespace, integer leading zeros, and explicit plus signs.
| Language | Model | Pass | Compile error | Runtime crash | Timeout | Wrong answer | Harness | of which formatting-onlyws / leading zeros / + sign |
|---|---|---|---|---|---|---|---|---|
| Fortran | Qwen3 4B think | 11.4% | 72.5% | 3.5% | 0.2% | 12.0% | 0.5% | 3.0% |
| Qwen3 8B think | 18.2% | 60.3% | 3.8% | 0.4% | 16.9% | 0.5% | 5.1% | |
| Qwen3 14B think | 29.6% | 43.9% | 5.2% | 1.6% | 18.9% | 0.7% | 5.7% | |
| COBOL | Qwen3 4B think | 0.2% | 99.3% | 0.0% | 0.0% | 0.5% | 0.0% | 0.0% |
| Qwen3 8B think | 4.3% | 90.8% | 0.2% | 0.1% | 4.6% | 0.1% | 0.7% | |
| Qwen3 14B think | 4.8% | 90.2% | 0.3% | 0.2% | 4.5% | 0.1% | 1.4% |
print *, is list-directed output: the compiler right-justifies each integer in a ~12-character field and emits its own record structure, so a correct 3 prints as eleven spaces and a 3 — exact-match grading fails every test. COBOL's default DISPLAY of an unedited PIC 9(18) prints 0 as eighteen zeros, and signed pictures add an explicit +. A formatting-tolerant grader lifts pass@1 (micro over the three cells) for Fortran from 19.7% to 24.3% (+4.6pp, +23% relative); COBOL from 3.1% to 3.8% (+0.7pp, +22% relative). COBOL keeps a further failure mode no normalization can rescue: declaring too-small PIC widths truncates large values, corrupting the digits themselves. The in-language fixes — write(*,'(I0)'), edited picture clauses — exist in the models' vocabulary but are rarely reached for: nothing in real-world Fortran/COBOL corpora punishes default formatting.The OCaml section's strongest lever was rejecting code that does not compile. Replaying it here (compile only — no grammar checker exists for these languages): Z is the share of samples the filter accepts, skyline@k is pass@k among accepted samples only, oracle is the perfect-selector ceiling over the same ~100 samples. Passing samples always compile, so the filter is sound by construction and per problem pass@1 = Z × skyline@1.
| Language | Cell (t=0.6) | Zacceptance rate | skyline@1all samples → compiled only | skyline@10all samples → compiled only | skyline@50all samples → compiled only | gap closed@1, of pass@1 → oracle | oraclecoverage ceiling |
|---|---|---|---|---|---|---|---|
| Fortran | Qwen3 4B think | 27.5% | 11.4% → 22.3% | 26.4% → 32.7% | 33.4% → 35.2% | 45.7% | 35.3% |
| Qwen3 8B think | 39.7% | 18.2% → 29.1% | 32.6% → 41.3% | 42.1% → 46.1% | 38.4% | 46.7% | |
| Qwen3 14B think | 56.1% | 29.6% → 38.7% | 46.4% → 58.7% | 59.2% → 64.6% | 25.8% | 64.7% | |
| COBOL | Qwen3 4B think | 0.7% | 0.2% → 4.4% | 1.8% → 6.7% | 4.8% → 6.7% | 64.3% | 6.7% |
| Qwen3 8B think | 9.2% | 4.3% → 11.2% | 13.5% → 19.4% | 19.7% → 22.0% | 39.0% | 22.0% | |
| Qwen3 14B think | 9.8% | 4.8% → 13.5% | 16.3% → 26.1% | 25.0% → 27.3% | 38.5% | 27.3% |
Provenance: recomputed from executions/domain=mlcb in clementepasti/mlcb-rollouts (Fortran, COBOL), samuki-hf/temperature-sweep-data (OCaml reference), and rollouts/domain=livecodebench in samuki-hf/thinking-rollouts (Python reference), all at t=0.6. The formatting-only column reads the locally built categories table (see the repo's local/ tooling); grading caveats from the OCaml section apply unchanged.