Skip to content

evaluator

genlm.eval.domains.livecodebench_multilingual.evaluator

Evaluator for multilingual LiveCodeBench.

Extracts the code block from a generation and grades it against the problem's stdin/stdout tests via a MultilingualCodeExecutor. Strict 0/1, memoized on (instance_id, code). grading="lenient" is Multi-LCB's comparator; "exact" is Agnostics rstrip-equality.