Skip to content

errata

genlm.eval.domains.livecodebench_multilingual.errata

Problems that cannot be scored by the harness.

Three categories: the problem accepts several correct answers and the harness compares against one scraped answer, the problem needs an interactive judge, or the shipped test data is wrong.

ERRATA_UPSTREAM holds the ids LiveCodeBench records in its own ERRATA.md. The harness does not read that file, so the problems stay in every loaded dataset.

ERRATA_VERIFIED holds the ids established here by grading a verified reference solution against the official test data.

Each entry is a property of the problem statement and its test data, so it applies to every target language. A given slice contains only part of this list: some upstream ids are LeetCode problems, which the stdin filter removes, and some predate the default date window.

Evidence per problem, and the reference solutions behind ERRATA_VERIFIED: https://huggingface.co/datasets/samuki-hf/ocaml-reference-solutions

drop_errata(rows)

Drop rows whose question_id appears on either list.

Source code in genlm/eval/domains/livecodebench_multilingual/errata.py 
68
69
70
def drop_errata(rows: Iterable[Mapping[str, Any]]) -> List[Mapping[str, Any]]:
    """Drop rows whose ``question_id`` appears on either list."""
    return [r for r in rows if str(r.get("question_id")) not in ERRATA]