Comparing languages for rollouts

Looking at the 16 languages exposed by livecodebench_multilingual in genlm-eval, profiled across the design axes that matter for picking a diverse set, every claim sourced.

On this page
  1. Recommended focus set
  2. Language tiers
  3. Sourcing & sources
  4. OCaml results
  5. Thinking vs non-thinking
  6. When and why the models fail
  7. Can filtering fix it? A rejection-sampling skyline
  8. Fortran & COBOL results
  9. Why Fortran and COBOL fail
  10. The compile filter, replayed on Fortran and COBOL

drag a column edge to resize it · the header row and language column stay pinned while you scroll · tier dots: Multi-LCB · Agnostics low-resource · Agnostics high-risk ?

Language Paradigm Typing Type inference Discipline strength · style Garbage collection Scope Macros / metaprog. Exec Training data GB · files · v2-dedup ↗ Popularity TIOBE · SO'24 Distinctive
Tier 1 — Multi-LCB (11)
interpreted
Multi-paradigmOOP · procedural · functional · reflective dynamic None hints are external (mypy), not runtime Strong · ducknominal for user classes refcount + cycle collector Lexical (LEGB)no block scope nonedecorators · metaclasses VM HUGE
233.3 GB · 56.9M
#1 18.96%
SO 51%
Duck typing + significant indentation
compiled · native
Multi-paradigmprocedural · OOP · generic · functional static Local auto / decltype (C++11) Strong · nominalpervasive implicit conversions → "loosely type-safe" no GCmanual · RAII · smart ptrs Lexical · blockstorage duration = RAII yespreprocessor + templates (TMP) + constexpr compiled HUGE
353.9 GB · 43.2M
#3 8.03%
SO 23%
RAII — deterministic scope-bound resources
JVM bytecode · JIT
Class-based OOP+ generic · functional (lambdas/streams) static Local var (10), diamond <> Strong · nominalno duck typing tracing, generational (G1/ZGC) Lexical · blocklambdas capture effectively-final noneannotations + processors (JSR 269) VM/JIT HUGE
548.0 GB · 154.3M
#4 7.90%
SO 30.3%
Write-once-run-anywhere bytecode
CIL · JIT
Multi-paradigm OOPfunctional · generic · component · event staticopt-in dynamic type Local var, target-typed new Strong · nominaldynamic = duck-typing escape hatch tracing, generational (gen 0/1/2) Lexical · blocklambdas capture variables noneattributes · Source Generators (Roslyn) VM/JIT HUGE
239.9 GB · 51.2M
#5 4.85%
SO 27.1%
LINQ — type-checked integrated queries
compiled · native
Imperative + concurrentno classes/inheritance static Local :=; generic call-site inference Strong · structural interfacesnominal named types · no implicit numeric coercion concurrent tracing mark-sweep Lexical · blockshadowing nonego:generate (convention) · generics compiled LARGE
54.6 GB · 9.3M
#13 1.20%
SO 13.5%
Goroutines + channels (CSP)
interpreted · JIT
Multi-paradigmevent-driven · functional · prototype-OOP dynamic None no static type inference Weak · duckimplicit coercion (==, +) · prototype model tracing mark-sweep (gen/incremental) Lexicalvar function-scope vs let/const block · closures noneeval · transpilers (Babel) VM/JIT HUGE
1,115 GB* · 108.9M
#6 3.04%
SO 62.3%
Prototypal inheritance + single-thread event loop
transpiled → JS
Multi-paradigmstrict superset of JS + static types staticerased at compile Extensivelocal + contextual (best-common-type) Strong · structuralgradual; any escape hatch via JS engine (V8 gen. mark-sweep) Lexical (JS)let/const block · closures nonedecorators · type-level programming (erased) →JS LARGE
61.0 GB · 23.9M
#37 0.43%
SO 38.5%
Gradual structural typing over JS, fully erased
compiled · native (LLVM)
Multi-paradigmfunctional-influenced · OOP via traits static Extensive (local)HM-based; signatures annotated Strong · nominaltraits · no implicit numeric coercion no GCownership + borrow checker · RAII · opt. Rc/Arc Lexical · blockdeterministic drop yeshygienic macro_rules! + procedural macros compiled MEDIUM
15.6 GB · 2.2M
#12 1.26%
SO 12.6%
Ownership + borrow checker = safety w/o GC
interpreted · VM
Pure OOPeverything is an object · functional (blocks) dynamic None RBS / Sorbet are add-ons Strong · duckno silent cross-type coercion mark-sweep · generational + incremental Lexicaldef/class/module = scope gates · blocks close over noneruntime metaprog. (method_missing) VM MEDIUM
31.7 GB · 17.8M
#33 0.52%
SO 5.2%
Blocks/closures on every method + uniform object model
interpreted · VM
Imperative + OOPprocedural web-scripting origins dynamicopt. type declarations · strict_types None no static type inference Weak · duck-ishtype juggling ("0"==0) · === strict refcount + cyclic collector (5.3+) Function-localno auto global access (need global) · superglobals noneruntime reflection · magic methods · eval VM HUGE
224.6 GB · 46.0M
#14 1.17%
SO 18.2%
HTML-embedded per-request server scripting
JVM bytecode (+JS/native)
Multi-paradigmOOP + functional · imperative · generic static Local + function-signature, bidirectional (not global HM) Strong · nominalnull-safety in the type system JVM collector · Native CMS Lexical · blockclosures noneinline + reified generics · KSP VM/JIT SMALL
5.8 GB · 2.2M
#23 0.71%
SO 9.4%
Null safety enforced at compile time
Tier 2–3 — Agnostics low-resource (5)
interpreted · LuaJIT
Multi-paradigm scriptingprocedural · OOP via metatables · functional dynamic None no static type inference Strong-ish · duckmetatables · narrow coercions (string↔number) incremental mark-sweep + generational (5.4) Lexicalclosures with upvalues nonemetatables / metamethods (runtime) VM MEDIUM
33.9 GB · 2.4M
#31 0.53%
SO 6.2%
Metatables as the single extensibility mechanism
JIT (LLVM)
Multiple dispatch+ functional · array · metaprogramming dynamicoptional annotations Extensivefor JIT specialization (not a static guarantee) Strong · nominal + parametricdeep dispatch on argument types concurrent, parallel, generational mark-sweep Lexical yestrue Lisp-style homoiconic macros (@, AST-level) JIT SMALL
6.1 GB · 0.45M
#26 0.57%
SO 1.1%
Multiple dispatch + parametric types + Lisp macros
interpreted
Functional + arraymultiple OOP systems (S3/S4/R5/R6) dynamic None pervasive runtime coercion Implicit coercionatomic vectors: char→double→int→logical generational (3 levels) + write barrier Lexicaladopted from Scheme (departure from S) nonecomputing on the language: quote/eval/substitute · lazy promises interp. MEDIUM
22.4 GB · 5.2Mv1-dedup only 0.30 GB
#9 1.69%
SO 4.3%
Lazy evaluation via promises → pervasive NSE
compiled (native/bytecode)
Functional-first+ imperative + OOP static Full Hindley-Milnerprincipal types; params need no annotation Strong · structuralobjects/poly-variants structural; records/variants nominal generational · incremental · precise Lexicalmodule system + functors none built-inPPX AST preprocessors (4.02+) compiled TINY
1.1 GB · 0.16M
#43 0.29%
SO 0.8%
Full type inference + ADTs + functor module system
compiled · native
Imperative + arrayOOP since F2003 · numeric/scientific staticstrong · manifest None implicit typing legacy (I–N = INTEGER); IMPLICIT NONE modern Strong · manifest nominalsome implicit numeric conversions no GCmanual; ALLOCATABLE auto-freed at scope (not GC) Lexical by unithost / use association · legacy COMMON none nativeexternal C preprocessor (cpp/fpp) for .F90 compiled SMALL
4.7 GB · 0.27Mv1-dedup only 0.17 GB
#17 0.95%
SO 1.1%
Array/numeric focus + implicit-typing legacy
static dynamic no GC (rare) has macros (rare) volume: tiny small medium large huge v2 gated → v1 figure * inflated by minified JS
Reading "Training data" The deduplicated volume of source code (GB) and its file count in The Stack v2 — e.g. Python is 233.3 GB across 56.9M files. It measures raw code mass, not usage, so Java (548 GB) > Python (233 GB) simply because Java code is voluminous and boilerplate-heavy (154M vs 57M files) — not more used. What's decision-relevant is the bucket (order of magnitude): the huge → tiny spread, e.g. OCaml (~1 GB) is roughly 200× smaller than Python. Differences within one order of magnitude (Python vs Java) are confounded by verbosity — don't over-read them; the popularity column is the usage cross-check.

How the 16 split on each axis lopsided axes show which corners are rare — and worth deliberately including

Typing 16

static9
dynamic7

Type inference 16

none7
local5
extensive3
full HM1

Memory 16

garbage coll.13
no GC3

True macros 16

none13
yes3

Nominal · structural · duck 16

nominal8
duck5
structural3

Training-data volume 16

huge6
large2
medium4
small3
tiny1

Recommended focus set

Choose the following subset to roughly maximize axis coverage with the fewest languages (with data volume to span the spectrum huge → medium → small → tiny).

Minimal 2-language contrast: Python + OCaml flips nearly every axis at once — dynamic↔static, duck↔full-Hindley-Milner, OOP/procedural↔functional-first, huge↔tiny data — the cheapest test of whether a method generalizes off the high-resource dynamic baseline.
Pythonanchor

Native LCB language; the high-resource dynamic baseline.

  • dynamic · duck · GC
  • data: huge (233 GB)
  • the control condition
Rustcontrast

The rare corner: no GC + a real hygienic macro system + strong nominal static typing.

  • static · nominal · no GC
  • macros (1 of 3)
  • data: medium (15.6 GB)
OCamllow-resource

Functional + full inference + genuinely tiny data — maximally distant from Python.

  • full Hindley-Milner
  • functional-first · structural
  • data: tiny (~1.1 GB)
Juliaparadigm

A paradigm no other harness language has: multiple dispatch + Lisp macros.

  • multiple dispatch
  • macros + dynamic + JIT
  • data: small (6.1 GB)
Fortranwildcard

The numeric/legacy extreme: no GC, no inference, no macros, low data.

  • array/numeric · no GC
  • no inference · no macros
  • data: small (4.7 GB), legacy

Python → Rust → Julia → OCaml walks the data axis huge → medium → small → tiny while also covering static/dynamic, GC/no-GC, the macro languages (Rust, Julia), full-HM inference (OCaml), structural vs nominal vs duck, and four paradigm families. Cut to three by dropping Julia; add Fortran for the numeric/no-inference corner.

Language tiers

The colored dot on each language marks where it comes from and how the harness groups it.

Sourcing & sources

How each column is sourced — and why the training-data numbers are a defensible estimate of relative pretraining exposure rather than a measurement of any one model's diet.

Training-data volume

  • StarCoder2 & The Stack v2 — arXiv 2402.19173, Table 1: per-language v1/v2-dedup GB + files (14 of 16)
  • StarCoder (v1) — arXiv 2305.06161, Tables 1–2: Kotlin & OCaml after-dedup (absent from SC2)
  • The Stack v2 dataset card — 67.5 TB / 658 langs; Software Heritage; per-lang CSV access-gated (the † cells)
  • Qwen2.5-Coder report — example undisclosed GitHub-derived corpus (5.5T tokens), used to justify the proxy

Popularity cross-checks

Language references — Tier 1

Language references — Tier 2–3

  • Lua — reference manual 5.4
  • Julia — manual
  • R — R Language Definition + R-ints
  • OCaml — manual + docs
  • Fortran — fortran-lang + gfortran

OCaml results

How well do these models write OCaml? We ran the four instruct/chat models plus the two Qwen3 reasoning (think) cells through livecodebench_multilingual's OCaml track (436 problems, exact-match grading) and compared the pass rates against the same models' Python LiveCodeBench numbers. The short version: OCaml pass@1 sits at a few percent, and the gap to Python is mostly about writing OCaml that compiles, not about reasoning (see the error analysis below).

How to read this page

  • OCaml tracklivecodebench_multilingual's 436 OCaml problems (competitive-programming tasks, stdin→stdout), graded by exact match on the whole output.
  • Python LCB — the 714-problem Python LiveCodeBench window the rest of this site uses, graded by the functional harness.
  • 222 shared — the problems present in both pools; the controlled OCaml-vs-Python comparison restricts to these.
  • pass@k — probability that at least one of k sampled completions is correct (n=100 samples per problem; greedy t=0 has only pass@1).
  • Failure categoryNo code = no extractable OCaml in the response; Compile error = the OCaml did not type-check or parse (the compiler stopped before running); Runtime crash = it compiled but raised an exception at run time; Timeout = it ran past the per-test time budget; Wrong answer = it compiled, ran to completion, and printed the wrong output.

Models

Llama-3.1 8B Instructllama31-8b-instruct-nothink
Qwen2.5-Coder 7B Instructqwen25-coder-7b-instruct-nothink
Qwen3 8Bqwen3-8b-nothink
Qwen3 14Bqwen3-14b-nothink

The four base rows are the instruct / chat (non-thinking) variants; the two Qwen3 think cells (reasoning mode, generated at t=0.6 only) appear in bold / as diamonds. Sampling temperature t=0.6 unless noted.

Pass@k by temperature

Pass@k versus temperature on all 436 OCaml problems; diamonds are the Qwen3 think cells (t=0.6 only). Non-thinking pass@1 sits at roughly 2–9% and is highest at greedy decoding; sampling helps pass@10/50 a little.

Full-set gap versus Python (directional)

OCaml pass@1 over all 436 OCaml problems against Python pass@1 over the 714-problem Python window, at t=0.6. This is a directional language-capability gap, not a controlled per-problem comparison: the two pools are different sets and sizes and use different graders (exact vs functional). Read it as order-of-magnitude.

ModelOCaml pass@1436 problems · exactPython LCB pass@1714 problems · functionalPython / OCaml
Llama-3.1 8B Instruct2.9%17.6%6.0×
Qwen2.5-Coder 7B Instruct5.3%33.5%6.3×
Qwen3 8B6.3%47.5%7.6×
Qwen3 14B7.7%52.0%6.7×
Qwen3 8B think21.0%81.4%3.9×
Qwen3 14B think15.6%84.7%5.4×

OCaml vs Python on shared problems

The OCaml campaign drew from a wider LiveCodeBench pool than our Python window, so only 222 of the 436 OCaml problems are also in the Python set. Restricting both sides to those shared ids gives a controlled comparison: same problems, OCaml versus Python, per-problem pass@1 with 95% bootstrap CIs.

ModelOCaml pass@1222 sharedPython pass@1222 sharedPython pass@1714 fullPython / OCamlshared
Llama-3.1 8B Instruct2.6% [1.4, 4.1]16.7% [12.7, 20.7]17.6%6.4×
Qwen2.5-Coder 7B Instruct6.3% [3.9, 9.1]23.2% [18.3, 28.2]33.5%3.7×
Qwen3 8B7.4% [4.9, 10.5]38.1% [32.5, 44.3]47.5%5.1×
Qwen3 14B8.1% [5.4, 11.2]42.6% [36.8, 48.8]52.0%5.3×
Qwen3 8B think24.2% [20.2, 28.5]73.7% [68.2, 79.0]81.4%3.0×
Qwen3 14B think16.6% [12.8, 20.7]75.5% [70.1, 80.7]84.7%4.5×
The shared problems are harder-than-average Python problems. Restricting Python to the 222 shared ids lowers its pass@1 (full→shared: Llama-3.1 8B Instruct 17.6%→16.7%; Qwen2.5-Coder 7B Instruct 33.5%→23.2%; Qwen3 8B 47.5%→38.1%; Qwen3 14B 52.0%→42.6%; Qwen3 8B think 81.4%→73.7%; Qwen3 14B think 84.7%→75.5%), so the honest controlled gap is about 3.7–6.4×, narrower than the directional full-set gap of roughly 5–8×. Models still solve OCaml at a small fraction of the Python rate, but the matched comparison is the fair number.

Thinking vs non-thinking

The reasoning arm (Qwen3, t=0.6). Thinking lifts OCaml pass@1 substantially for both sizes, but with a twist in how the two sizes compare.

ModelArmpass@1pass@10pass@50pass@1 lift
Qwen3 8Bnothink6.3%14.8%20.8%+14.7pp (3.3×)
think21.0%42.2%52.2%
Qwen3 14Bnothink7.7%15.9%21.4%+7.9pp (2.0×)
think15.6%30.1%42.5%
Reasoning flips the size ranking on OCaml. Without thinking the 14B leads (as expected); with thinking the 8B overtakes it (21.0% vs 15.6% pass@1). It is not truncation (the 14B truncates less) and not a logic gap (its wrong-answer rate is lower) — the 14B's completions compile more but crash at runtime far more (20.9% of its failures are runtime crashes vs 8.2% for the 8B): it writes more ambitious OCaml that gets past the compiler and then dies on stdin parsing / exceptions. The 8B fails earlier and cheaper.

Where the thinking models fail

Failure-mode mix for each reasoning cell (share of failing completions, full execution record). The 14B's failures skew to runtime crashes; the 8B's skew to truncation and compile errors.

Failure modeQwen3 8B (think)Qwen3 14B (think)
Compile error68.8%67.2%
Runtime crash8.2%20.9%
Wrong answer5.9%3.3%
Empty / truncated15.6%8.2%
Timeout1.6%0.4%

When and why the models fail

To see why OCaml is hard, we classified all 164,683 failing completions across these four models from the full per-test execution record (the compiler / runtime output the pass/fail score throws away). Each failure is bucketed by how far the code got: did it parse, compile, run, finish?

Most OCaml answers never reach a wrong-answer verdict. Across the analysed models, about 80% of failures never compile (no code or a type/syntax error), another 12% compile but crash or time out, and only about 8% compile, run, and simply print the wrong answer. So the gap to Python is mostly a language-fluency gap (writing OCaml that compiles and reads input correctly), not an algorithmic-reasoning gap.

What the compiler objects to

Share of each model's compile errors by the exact message the OCaml toolchain printed. Genuine syntax errors, unbound names, and type errors each take a big slice: the models write plausible-looking OCaml that does not parse or type-check. (Syntax errors include generations cut off at the token limit.)

Compiler messageLlama-3.1 8B InstructQwen2.5-Coder 7B InstructQwen3 8BQwen3 14B
Syntax error24%17%43%32%
Type mismatch (value)23%34%17%18%
Unbound value21%24%17%22%
Type mismatch (expression)16%11%8%9%
Unbound constructor3%3%7%10%
Other compile error6%2%3%4%
Type mismatch (constant)2%1%2%2%
Unbound module2%2%0%2%
Type mismatch (pattern)1%2%2%0%
Wrong type expected2%2%0%1%

The names the models invent

A fifth of compile errors are Unbound value: the model calls something that is not in OCaml's standard library. The identifiers it reaches for most (count = times seen in the sampled failures) are mostly Python / C habits (continue and break are not OCaml keywords; String.to_list, String.filter, Array.create, Hashtbl.find_default do not exist; printf needs Printf.) and mis-remembered module paths (Scanf.stdin).

Scanf.stdin1305String.to_list1023printf731Array.create720continue637find558String.filter524int64_of_string492Scanf.Scanner453Hashtbl.find_default435break345List.of_string335Scanf.Scanning.fscanf330Core326String.rev325Array.of_string307Char.is_uppercase300List.last255scanf251String.set238Scanf.Scanning.scanf224Scanf.Scanning.bounded_int_list223Scanf.fscanf209List.range193List.sub187String.split180solve177String.count169

Runtime-crash breakdown

Share of each model's failing completions that compiled but then crashed. Scanf input-parse crash dominates: the model mis-reads stdin.

Sub-reasonLlama-3.1 8B InstructQwen2.5-Coder 7B InstructQwen3 8BQwen3 14B
Scanf input-parse crash1.2%19.7%0.4%8.1%
End_of_file (read past input)0.5%2.9%1.0%2.5%
Index out of bounds0.3%0.7%0.7%0.4%
Invalid_argument0.1%0.3%0.1%0.1%
Non-exhaustive match0.0%0.0%0.1%
Not_found0.0%0.0%0.0%0.1%
Stack overflow0.2%0.2%0.2%0.0%
Other runtime error0.1%0.0%0.0%0.1%

Examples: what the failures look like

One real failing completion per failure mode (click to expand the model's OCaml and the exact compiler or runtime message).

abc349_a easy Compile error · Unbound name (invented / wrong API)
(* Read input from stdin *)
let read_int () = Scanf.scanf "%d" (fun x -> x)
let read_int_list () = Scanf.Scanner.scan_string "%d " >> fun s -> List.map int_of_string (Str.split (Str.regexp " ") s) |> List.filter (fun x -> x <> 0)

(* Main function *)
let rec solve n scores =
  if n = 1 then scores.(n - 1) else
    let sum = List.fold_left (fun acc x -> acc + x) 0 scores in
    let avg = sum / (n - 1) in
    avg +. (solve (n - 1) scores)

(* Read input and print result *)
let () =
  let n = read_int () in
  let scores = read_int_list () in
  let result = solve n scores in
  Printf.printf "%+.0f\n" result
File "Main.ml", line 3, characters 55-57:
3 | let read_int_list () = Scanf.Scanner.scan_string "%d " >> fun s -> List.map int_of_string (Str.split (Str.regexp " ") s) |> List.filter (fun x -> x <> 0)
                                                           ^^
Error: Unbound value (>>)
abc380_c medium Compile error · Type mismatch
(* Problem: Move the K-th 1-block from the beginning in S to immediately after the (K-1)-th 1-block *)

(* Read input from standard input *)
let () =
  let _ = Scanf.scanf "%d %d\n" (fun n k -> ()) in
  let s = Scanf.scanf "%s\n" (fun s -> s) () in
  let n = String.length s in

  (* Find all 1-blocks in the string *)
  let rec find_1_blocks i acc =
    if i = n then acc
    else if s.[i] = '1' && (i = 0 || s.[i-1] = '0') then
      find_1_blocks (i + 1) ((i, i) :: acc)
    else find_1_blocks (i + 1) acc
  in

  let blocks = find_1_blocks 0 [] in

  (* Move the K-th 1-block to immediately after the (K-1)-th 1-block *)
  let rec move_block blocks k =
    match blocks with
    | [] -> []
    | _ when k = 1 -> blocks
    | (l1, r1) :: (l2, r2) :: tl ->
       if k = 2 then (l1, r1) :: (l2, r2 + (r2 - l2) + 1) :: tl
       else (l1, r1) :: move_block ((l2, r2) :: tl) (k - 1)
  in

  let new_blocks = move_block blocks 3 in

  (* Create the resulting string *)
  let rec create_string i acc =
    match new_blocks with
    | [] -> acc
    | (l, r) :: tl ->
       let len = r - l + 1 in
       let prefix = String.sub s 0 l in
       let block = String.make len '1' in
       let suffix = String.sub s (l + len) (n - l - len) in
       create_string (l + len) (prefix ^ block ^ suffix ^ acc)
  in

  let result = create_string 0 "" in
  Printf.printf "%s\n" result
File "Main.ml", line 6, characters 39-40:
6 |   let s = Scanf.scanf "%s\n" (fun s -> s) () in
                                           ^
Error: The value s has type string but an expression was expected of type
abc392_c medium Compile error · Syntax error
(* Problem: Given two lists of integers P and Q of length N, 
   find the number written on the bib of the person that the person wearing 
   the bib with number i is staring at. *)
(* Input: N, P, Q *)
(* Output: S where S_i is the number written on the bib of the person 
   that the person wearing the bib with number i is staring at. *)

(* Read input from standard input *)
let read_int () = Scanf.scanf "%d" (fun x -> x)
let read_int_list n = List.init n read_int
let () = Scanf scanf "%d" read_int_list
let n = read_int ()
let p = read_int_list n
let q = read_int_list n

(* Create a list of pairs (i, P_i) *)
let p_list = List.mapi (fun i x -> (x, i)) p

(* Create a map from P_i to i *)
let p_map = List.fold_left (fun acc (x, i) -> Map.put acc x i) Map.empty p_list

(* Create a list of S_i *)
let s = List.map (fun x -> Map.find x (p_map)) q

(* Print the list of S_i *)
let () = List.iter (fun x -> Printf.printf "%d " x) s; 
        Printf.printf "\n"
File "Main.ml", line 11, characters 21-25:
11 | let () = Scanf scanf "%d" read_int_list
                          ^^^^
Error: Syntax error
abc393_f hard Runtime crash · Scanf input-parse crash
(* Problem: Find the maximum length of a strictly increasing subsequence of a given sequence
   with elements at most X_i, for Q queries. *)

(* Read input from standard input *)
let () =
  let n, q = Scanf.scanf "%d %d\n" (fun n q -> n, q) in
  let a = Array.init n (fun _ -> Scanf.scanf "%d" (fun x -> x)) in
  let queries = Array.init q (fun _ ->
      let r, x = Scanf.scanf "%d %d\n" (fun r x -> r, x) in
      r - 1, x
    ) in

  (* Process queries *)
  let results = Array.init q (fun i ->
      let r, x = queries.(i) in
      let rec find_max_length r x acc =
        if r < 0 then acc
        else
          let ai = a.(r) in
          if ai > x then find_max_length (r - 1) x acc
          else
            let new_acc = if ai <= acc then acc + 1 else acc in
            find_max_length (r - 1) x new_acc
      in
      find_max_length r x 0
    ) in

  (* Print results *)
  List.iter (fun x -> Printf.printf "%d\n" x) (Array.to_list results)
Fatal error: exception Stdlib.Scanf.Scan_failure("scanf: bad input at char number 5: character ' ' is not a decimal digit")
abc374_b easy Runtime crash · Index out of bounds
(* Problem: Check if two strings are equal, if not print the position of the first difference *)
(* Input: Two strings S and T *)
(* Output: If S and T are equal, print 0; otherwise, print the position of the first character where they differ *)

(* Read two strings from the input *)
let () =
  let s = read_line () in
  let t = read_line () in

  (* Compare the strings *)
  let rec compare_strings s t pos =
    if pos > min (String.length s) (String.length t) then
      (* If one string is a prefix of the other, print the position of the first differing character *)
      if String.length s < String.length t then
        Printf.printf "%d\n" (String.length s + 1)
      else if String.length s > String.length t then
        Printf.printf "%d\n" (String.length t + 1)
      else
        Printf.printf "0\n"
    else if s.[pos] <> t.[pos] then
      (* If the characters at the current position are different, print the position *)
      Printf.printf "%d\n" (pos + 1)
    else
      (* If the characters are the same, continue comparing the rest of the strings *)
      compare_strings s t (pos + 1)
  in

  (* Start comparing the strings from the first position *)
  compare_strings s t 0
Fatal error: exception Invalid_argument("index out of bounds")
abc335_b easy Wrong answer
(* This program prints all triples of non-negative integers (x,y,z) such that x+y+z <= N in ascending lexicographical order. *)

(* Read an integer from standard input *)
let read_int () =
  Scanf.scanf "%d" (fun n -> n)

(* Generate all triples (x, y, z) such that x + y + z <= N *)
let generate_triples n =
  let rec generate x y =
    if x + y > n then []
    else
      (x, y, n - x - y) :: generate x (y + 1) @ generate (x + 1) y
  in
  generate 0 0

(* Print the triples in ascending lexicographical order *)
let print_triples triples =
  List.iter (fun (x, y, z) ->
    Printf.printf "%d %d %d\n" x y z)
    triples

(* Main function *)
let () =
  let n = read_int () in
  let triples = generate_triples n in
  print_triples triples
abc397_b medium Timeout
(* Problem: Takahashi's Ticket Gate Records *)
(* Author: [Your Name] *)
(* Date: [Today's Date] *)

(* Function to calculate the minimum number of characters to be inserted *)
let rec min_insertions s =
  let rec loop i count =
    if i >= String.length s then
      count
    else if i mod 2 = 0 then
      (* If the current character is at an even position, check the next character *)
      if i + 1 < String.length s && s.[i] = 'o' && s.[i + 1] = 'i' then
        loop (i + 2) count
      else
        loop (i + 1) (count + 1)
    else
      (* If the current character is at an odd position, check the previous character *)
      if i - 1 >= 0 && s.[i] = 'i' && s.[i - 1] = 'o' then
        loop (i - 2) count
      else
        loop (i - 1) (count + 1)
  in
  loop 0 0

(* Main function *)
let () =
  (* Read the input string from standard input *)
  let s = Scanf.scanf "%s" (fun s -> s) in
  (* Calculate and print the minimum number of characters to be inserted *)
  Printf.printf "%d\n" (min_insertions s)
[<class 'subprocess.TimeoutExpired'>][Command '['/scratch/tmp.5294624.skiegeland/tmpj7pwtd53/Main.exe']' timed out after 8 seconds]
abc390_d medium No code
empty or unparsed code

Can filtering fix it? A rejection-sampling skyline

Most failing samples never compile, so an obvious lever is to discard malformed code before choosing what to submit. We test two static checks, neither of which looks at the tests: the grammar filter rejects code the OCaml parser cannot parse (a viable-prefix check on the incremental Menhir parser), and the compile filter rejects code that ocamlc refuses to compile (syntax, type, and unbound-name errors). Both are safe by construction here: across all four cells, no passing sample was ever rejected. ~100 samples per problem, all 436 problems.

Cell (t=0.6)FilterZacceptance rateskyline@1all samples → accepted onlyskyline@10all samples → accepted onlyskyline@50all samples → accepted onlygap closed@1, of pass@1 → oracleoraclecoverage ceiling
Qwen3 8B nothinkgrammar63.6%6.3% → 7.3%14.8% → 16.3%21.0% → 22.0%6.1%23.4%
compile20.3%6.3% → 12.9%14.8% → 21.0%21.0% → 23.2%38.8%
Qwen3 8B thinkgrammar64.3%21.0% → 23.6%42.2% → 45.7%52.2% → 54.8%7.4%56.2%
compile33.8%21.0% → 35.4%42.2% → 53.1%52.2% → 56.1%41.1%
Qwen3 14B nothinkgrammar73.3%7.8% → 8.3%16.2% → 16.9%21.8% → 22.6%3.1%24.3%
compile24.2%7.8% → 12.8%16.2% → 21.0%21.8% → 24.2%30.6%
Qwen3 14B thinkgrammar73.0%15.6% → 16.0%30.1% → 31.5%42.5% → 44.7%1.3%48.2%
compile35.9%15.6% → 25.1%30.1% → 40.1%42.5% → 47.3%29.1%

Z is the share of samples a filter accepts. skyline@k is pass@k among accepted samples only: the score of sampling until k accepted candidates are found, which costs about k/Z generations. The value left of each arrow is the ordinary pass@k over all samples. oracle is coverage, the score of a perfect selector over the same ~100 samples, an upper bound no filter can exceed. Neither filter ever rejects a correct sample, so per problem pass@1 = Z × skyline. Averages are over all 436 problems; problems where a filter accepts nothing count as 0. Same terminology as the DS-1000 and LCB skyline pages.

Keeping only code that compiles roughly doubles pass@1; a grammar check alone adds little. On the 8B thinking cell, the compile filter lifts pass@1 from 21.0% to 35.4% (41.1% of the way to the oracle), at a cost of about 3 generations per kept sample (Z = 33.8%). The grammar filter is nearly free but gains at most +2.6pp: correct samples are already grammatical, and the dominant failure type, type and unbound-name errors (60–71% of the parseable failures), is invisible to a parser but caught by ocamlc without running anything. Under 1% of syntax errors slip past the grammar check.

Filtering selects among the samples you already have, so no filter can beat the oracle column, and its advantage shrinks as the budget grows: with one sample there is nothing to choose between, and by 50 accepted samples the compile filter is within 0.1pp of the oracle. The failures that survive compilation, wrong answers and runtime crashes (30.5% of the 14B thinking cell's residual, mostly Scanf input parsing), can only be caught by running the code.

Caveats: OCaml grading is exact-match on a 436-problem subset; Python LCB grading is the functional harness on the 714-problem window, so the full-set ratio mixes set, size and grader differences (the matched 222-problem comparison removes the set/size difference). The mLCB evaluation was produced by the deprecated eval_mlcb.py; the numbers are usable here but are provisional, to be re-confirmed by the canonical genlm-eval grader.

Fortran & COBOL results

The second mLCB source (clementepasti/mlcb-rollouts): the same LCB problems in Fortran and COBOL — 150 of the 436 mLCB problems, Qwen3 4B/8B/14B thinking cells only (no non-thinking arm exists for these languages), t=0.6, 100 samples per problem. All numbers below are recomputed from the published per-test executions with the same pass@k estimator as the rest of the site; nothing is read from precomputed metric files.

LanguageModelpass@1150 problems · n=100pass@10150 problems · n=100pass@50150 problems · n=100
FortranQwen3 4B think11.4% [7.9, 15.2]26.4% [20.2, 33.0]33.4% [26.2, 40.4]
Qwen3 8B think18.2% [13.3, 23.2]32.6% [25.8, 39.5]42.1% [34.8, 49.7]
Qwen3 14B think29.6% [23.4, 36.2]46.4% [39.2, 53.7]59.2% [51.9, 66.5]
COBOLQwen3 4B think0.2% [0.1, 0.4]1.8% [0.6, 3.2]4.8% [2.0, 7.9]
Qwen3 8B think4.3% [2.4, 6.5]13.5% [8.9, 18.6]19.7% [13.9, 26.0]
Qwen3 14B think4.8% [2.8, 7.2]16.3% [11.3, 21.6]25.0% [18.3, 31.8]

The same 150 problems exist in all four language pools, which gives a genuinely matched cross-language comparison (Python from the thinking-rollouts LCB cells; OCaml recomputed from its executions on the subset — the 4B has no OCaml cell):

ModelPython pass@1150 sharedOCaml pass@1150 sharedFortran pass@1150 sharedCOBOL pass@1150 shared
Qwen3 4B think80.0% [74.0, 85.3]11.4% [7.9, 15.2]0.2% [0.1, 0.4]
Qwen3 8B think81.6% [75.7, 87.0]25.9% [21.0, 31.1]18.2% [13.3, 23.2]4.3% [2.4, 6.5]
Qwen3 14B think83.4% [77.8, 88.8]18.1% [13.5, 22.9]29.6% [23.4, 36.2]4.8% [2.8, 7.2]
The difficulty ordering at 14B is Python ≫ Fortran > OCaml > COBOL. Python is effectively saturated on these problems while COBOL is near floor (the 4B passes 32 of 15,000 samples). Two structural notes: scale behaves normally within Fortran and COBOL (4B < 8B < 14B) — the think-arm size inversion documented above is OCaml-specific — and the per-problem difficulty correlates far more between the two low-resource languages than between either of them and Python, which solves nearly everything and so discriminates little.

Why Fortran and COBOL fail

Share of all samples per outcome, from the per-test error codes (same categories as the explorer's failure filter). Compile errors dominate both languages, COBOL overwhelmingly so; the last column is the slice of wrong answers whose every failing test matches the expected output after normalizing whitespace, integer leading zeros, and explicit plus signs.

LanguageModelPassCompile errorRuntime crashTimeoutWrong answerHarnessof which formatting-onlyws / leading zeros / + sign
FortranQwen3 4B think11.4%72.5%3.5%0.2%12.0%0.5%3.0%
Qwen3 8B think18.2%60.3%3.8%0.4%16.9%0.5%5.1%
Qwen3 14B think29.6%43.9%5.2%1.6%18.9%0.7%5.7%
COBOLQwen3 4B think0.2%99.3%0.0%0.0%0.5%0.0%0.0%
Qwen3 8B think4.3%90.8%0.2%0.1%4.6%0.1%0.7%
Qwen3 14B think4.8%90.2%0.3%0.2%4.5%0.1%1.4%
A measurable slice of the wrong answers is right logic, wrong formatting. Fortran's idiomatic print *, is list-directed output: the compiler right-justifies each integer in a ~12-character field and emits its own record structure, so a correct 3 prints as eleven spaces and a 3 — exact-match grading fails every test. COBOL's default DISPLAY of an unedited PIC 9(18) prints 0 as eighteen zeros, and signed pictures add an explicit +. A formatting-tolerant grader lifts pass@1 (micro over the three cells) for Fortran from 19.7% to 24.3% (+4.6pp, +23% relative); COBOL from 3.1% to 3.8% (+0.7pp, +22% relative). COBOL keeps a further failure mode no normalization can rescue: declaring too-small PIC widths truncates large values, corrupting the digits themselves. The in-language fixes — write(*,'(I0)'), edited picture clauses — exist in the models' vocabulary but are rarely reached for: nothing in real-world Fortran/COBOL corpora punishes default formatting.

The compile filter, replayed on Fortran and COBOL

The OCaml section's strongest lever was rejecting code that does not compile. Replaying it here (compile only — no grammar checker exists for these languages): Z is the share of samples the filter accepts, skyline@k is pass@k among accepted samples only, oracle is the perfect-selector ceiling over the same ~100 samples. Passing samples always compile, so the filter is sound by construction and per problem pass@1 = Z × skyline@1.

LanguageCell (t=0.6)Zacceptance rateskyline@1all samples → compiled onlyskyline@10all samples → compiled onlyskyline@50all samples → compiled onlygap closed@1, of pass@1 → oracleoraclecoverage ceiling
FortranQwen3 4B think27.5%11.4% → 22.3%26.4% → 32.7%33.4% → 35.2%45.7%35.3%
Qwen3 8B think39.7%18.2% → 29.1%32.6% → 41.3%42.1% → 46.1%38.4%46.7%
Qwen3 14B think56.1%29.6% → 38.7%46.4% → 58.7%59.2% → 64.6%25.8%64.7%
COBOLQwen3 4B think0.7%0.2% → 4.4%1.8% → 6.7%4.8% → 6.7%64.3%6.7%
Qwen3 8B think9.2%4.3% → 11.2%13.5% → 19.4%19.7% → 22.0%39.0%22.0%
Qwen3 14B think9.8%4.8% → 13.5%16.3% → 26.1%25.0% → 27.3%38.5%27.3%
The compile filter is worth even more here than on OCaml — because compiling is so much rarer. On COBOL only about 7% of samples compile, so rejection sampling pays ~1/Z generations per kept candidate but multiplies pass@1 several-fold; the curves show the filtered lines approaching the oracle ceiling by a few dozen samples. As with OCaml, what survives compilation fails on semantics (and, for these two languages, on default output formatting) — catchable only by running the code.

Provenance: recomputed from executions/domain=mlcb in clementepasti/mlcb-rollouts (Fortran, COBOL), samuki-hf/temperature-sweep-data (OCaml reference), and rollouts/domain=livecodebench in samuki-hf/thinking-rollouts (Python reference), all at t=0.6. The formatting-only column reads the locally built categories table (see the repo's local/ tooling); grading caveats from the OCaml section apply unchanged.