capture

genlm.eval.domains.livecodebench_multilingual.capture

Full per-test execution capture for the multilingual-LCB (Agnostics) executor.

The graded path short-circuits on the first failing test and keeps only a solved/unsolved verdict. capture_run re-runs the loop without short-circuiting and records every test's stdout, stderr, exit code, and verdict. It reuses the vendored primitives and comparators, so its per-test verdicts and aggregate solved match the official grader (checked for parity); the vendored file is untouched.

For OCaml it compiles once (ocamlopt, ocamlc fallback) and runs the binary per test rather than letting ocaml Main.ml recompile every invocation; a compile failure maps to EXECFAIL for all tests, as the official path does.

capture_run(code, inputs, outputs, language, timeout=10.0, grading='exact', max_completion_seconds=1000000000.0)

Run every test of one solution and return (solved, per_test_records), no short-circuit.

Each record has test_idx, passed, output (untruncated stdout), error_message, error_code (rollouts convention), status, and time_s. solved is all(per-test passed). A per-completion wall cap (max_completion_seconds) records tests past the cap as not-passed, status "capped".

Source code in genlm/eval/domains/livecodebench_multilingual/capture.py

def capture_run(
    code: str,
    inputs: List[str],
    outputs: List[str],
    language: str,
    timeout: float = 10.0,
    grading: str = "exact",
    max_completion_seconds: float = 1e9,
) -> Tuple[bool, List[dict]]:
    """Run every test of one solution and return (solved, per_test_records), no short-circuit.

    Each record has test_idx, passed, output (untruncated stdout), error_message, error_code
    (rollouts convention), status, and time_s. `solved` is all(per-test passed). A per-completion
    wall cap (max_completion_seconds) records tests past the cap as not-passed, status "capped".
    """
    n = len(outputs)
    if not (code or "").strip():
        recs = [
            _rec(
                i, False, "", "Empty string instead of a program", Status.EmptyCode, 0.0
            )
            for i in range(n)
        ]
        return False, recs

    code = patch_prog(code, language)
    ext = eval_scripts[language][1]
    recs: List[dict] = []
    t_start = time.time()

    with tempfile.TemporaryDirectory() as tmp:
        path = Path(tmp, "Main" + ext)
        with open(path, "w", encoding="utf8") as f:
            f.write(code)
            f.flush()
        sconf = SubprocessConfig(plang=language, run_timeout=max(1, int(ceil(timeout))))
        sconf.set_cwd(tmp)

        if language in _COMPILED:
            exe = str(path.with_suffix(".exe"))
            run_args, build_err = _build(language, path, exe, sconf)
            if run_args is None:
                return False, [
                    _rec(i, False, "", build_err, Status.BuildFailed, 0.0)
                    for i in range(n)
                ]
        elif language in _INTERPRETED:
            run_args = [a.format(path=str(path)) for a in _INTERPRETED[language]]
        else:
            raise NotImplementedError(
                f"capture_run has no recipe for language {language!r}"
            )

        for idx, (inp, exp) in enumerate(zip(inputs, outputs)):
            if time.time() - t_start > max_completion_seconds:
                recs.append(
                    _rec(idx, False, "", "completion wall cap reached", "capped", 0.0)
                )
                continue
            t0 = time.time()
            res = run(
                run_args, input_data=inp, timeout_seconds=sconf.run_timeout, sconf=sconf
            )
            dt = time.time() - t0
            out = (res.stdout or "").strip()
            rs = get_run_status(res)
            if rs != Status.Done:
                recs.append(_rec(idx, False, out, res.stderr, rs, dt))
            else:
                passed = _verdict(out, inp if inp is not None else "", exp, grading)
                recs.append(
                    _rec(idx, passed, out, "" if passed else res.stderr, rs, dt)
                )

    solved = bool(recs) and all(r["passed"] for r in recs)
    return solved, recs