Multilingual LiveCodeBench¶

Evaluate a model on LiveCodeBench competitive-programming problems in many programming languages, not just Python.

• Task: given a problem statement, generate a complete program in the target language that reads from stdin and writes to stdout.
• Data: the standard-I/O (stdin/stdout) subset of LiveCodeBench, reused verbatim across every language. Functional/LeetCode problems are excluded (no conversion), which is the same subset the Agnostics Ag-LiveCodeBench-X benchmark uses.
• Languages (16): python, c++, java, c#, go, javascript, typescript, rust, ruby, php, kotlin (from Multi-LCB) and lua, julia, r, ocaml, fortran (from Agnostics).
• Execution: each candidate is compiled/run locally per language via a vendored copy of Multi-LCB's executor; the same domain works for all languages.

Setup¶

Install the Python dependencies:

pip install -e ".[livecodebench_multilingual]" genlm-control

Generated code runs in real per-language compilers and interpreters. These are system toolchains, not pip packages, so install them from the pinned conda spec:

conda env create -f docs/cookbook/domains/livecodebench_multilingual.environment.yml

Then run genlm-eval with that env's binaries on PATH and as CONDA_PREFIX, keeping your genlm-eval Python as the interpreter (the kotlinc and tsc wrappers resolve their real binary through CONDA_PREFIX):

P=$(conda run -n mlcb-tools printenv CONDA_PREFIX)
PATH="$P/bin:$PATH" CONDA_PREFIX="$P" python your_eval_script.py

Notes:

• Run evaluations only on a dedicated or disposable compute node: generated code executes as local subprocesses with rlimit and process-group SIGKILL only, not a container sandbox.
• A language whose toolchain is missing raises a clear error; check availability first.

In [ ]:

from genlm.eval.domains.livecodebench_multilingual import (
    LANGUAGES,
    is_toolchain_available,
)

print("languages:", sorted(LANGUAGES))
print("available on this machine:", [k for k in LANGUAGES if is_toolchain_available(k)])

from genlm.eval.domains.livecodebench_multilingual import ( LANGUAGES, is_toolchain_available, ) print("languages:", sorted(LANGUAGES)) print("available on this machine:", [k for k in LANGUAGES if is_toolchain_available(k)])

Initialize the dataset and evaluator¶

from_hf loads a LiveCodeBench release and keeps only the stdin/stdout problems (the testtypes filter is forced). Pick the target language; the same problems are reused for every language. The first call needs network access to populate the Hugging Face cache.

In [ ]:

from genlm.eval.domains.livecodebench_multilingual import (
    MultilingualLCBDataset,
    MultilingualLCBEvaluator,
)

LANGUAGE = "c++"

dataset = MultilingualLCBDataset.from_hf(
    language=LANGUAGE,
    release="release_v6",
    start_date="2024-01-01",
    max_instances=4,
)
print("Instances loaded:", len(dataset))

# grading="lenient" (default) is Multi-LCB's comparator; grading="exact" is the stricter
# Agnostics rstrip-equality rule.
evaluator = MultilingualLCBEvaluator(timeout_seconds=6.0, grading="lenient")

from genlm.eval.domains.livecodebench_multilingual import ( MultilingualLCBDataset, MultilingualLCBEvaluator, ) LANGUAGE = "c++" dataset = MultilingualLCBDataset.from_hf( language=LANGUAGE, release="release_v6", start_date="2024-01-01", max_instances=4, ) print("Instances loaded:", len(dataset)) # grading="lenient" (default) is Multi-LCB's comparator; grading="exact" is the stricter # Agnostics rstrip-equality rule. evaluator = MultilingualLCBEvaluator(timeout_seconds=6.0, grading="lenient")

In [ ]:

first = next(iter(dataset))
print("Question ID:", first.question_id)
print("Instance ID:", first.instance_id)
print("Language:", first.language)
print("Test type:", first.testtype)

first = next(iter(dataset)) print("Question ID:", first.question_id) print("Instance ID:", first.instance_id) print("Language:", first.language) print("Test type:", first.testtype)

Model adaptor¶

format_multilingual_prompt builds the per-language prompt (a system message naming the target language plus the stdin/stdout format block, byte-identical to Multi-LCB's for the 12 Multi-LCB languages). Code is extracted from the first fenced block, matching Multi-LCB.

In [ ]:

from genlm.control import PromptedLLM, direct_token_sampler
from genlm.eval import ModelOutput, ModelResponse
from genlm.eval.domains.livecodebench_multilingual import format_multilingual_prompt

LLM = PromptedLLM.from_name("Qwen/Qwen2.5-Coder-1.5B-Instruct", temperature=0.2)


async def model(instance, output_dir, replicate):
    # Build the per-language prompt for this instance (chat template for instruct models).
    LLM.prompt_ids = format_multilingual_prompt(
        LLM.model.tokenizer, instance, use_chat_format=True
    )
    sampler = direct_token_sampler(LLM)
    sequences = await sampler.smc(n_particles=5, ess_threshold=0.5, max_tokens=1024)
    return ModelOutput(
        responses=[
            ModelResponse(response=sequence, weight=prob)
            for sequence, prob in sequences.decoded_posterior.items()
        ],
    )

from genlm.control import PromptedLLM, direct_token_sampler from genlm.eval import ModelOutput, ModelResponse from genlm.eval.domains.livecodebench_multilingual import format_multilingual_prompt LLM = PromptedLLM.from_name("Qwen/Qwen2.5-Coder-1.5B-Instruct", temperature=0.2) async def model(instance, output_dir, replicate): # Build the per-language prompt for this instance (chat template for instruct models). LLM.prompt_ids = format_multilingual_prompt( LLM.model.tokenizer, instance, use_chat_format=True ) sampler = direct_token_sampler(LLM) sequences = await sampler.smc(n_particles=5, ess_threshold=0.5, max_tokens=1024) return ModelOutput( responses=[ ModelResponse(response=sequence, weight=prob) for sequence, prob in sequences.decoded_posterior.items() ], )

Run the evaluation¶

In [ ]:

from genlm.eval import run_evaluation

results = await run_evaluation(
    dataset=dataset,
    model=model,
    evaluator=evaluator,
    max_instances=2,
    n_replicates=1,
    verbosity=1,
)

from genlm.eval import run_evaluation results = await run_evaluation( dataset=dataset, model=model, evaluator=evaluator, max_instances=2, n_replicates=1, verbosity=1, )

Reproducing the source benchmarks¶

Because this domain evaluates only stdin/stdout problems, it matches Agnostics' Ag-LiveCodeBench-X (the 499-problem stdin subset of LiveCodeBench 5.0) rather than the full Multi-LCB mixed set (which also includes converted LeetCode functional problems).

For an Agnostics-style run, use their prompt, the exact comparator, and their dataset:

from genlm.eval.domains.livecodebench_multilingual import (
    agnostics_chat_messages,
    MultilingualLCBEvaluator,
)

evaluator = MultilingualLCBEvaluator(grading="exact")
# Prompt the model with agnostics_chat_messages(instance), and for the exact 499-problem set
# load datasets.load_dataset("nuprl/Ag-LiveCodeBench-X", split="test").

Python comparability. Grading is uniform across languages, so python is graded with the lenient Multi-LCB comparator (matching both source papers), not the exact-Decimal default-LCB grader used by the python-only LiveCodeBench domain and genlm-rollouts. Lenient is more permissive (True/true aliasing, 1e-5 float tolerance), so multilingual-python pass@1 is >= the default-LCB/rollouts python number and is not directly comparable to that leaderboard.

References¶

Naman Jain, King Han, Alex Gu, et al. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv:2403.07974, 2024.

Ivanova et al. Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages. arXiv:2606.20517, 2026. https://github.com/Multi-LCB/Multi-LCB

Boruch-Gruszecki et al. Agnostics: Learning to Synthesize Code in Any Programming Language with a Universal Reinforcement Learning Environment. arXiv:2508.04865, 2025. https://github.com/nuprl/agnostics-framework