LiveCodeBench¶

This example shows how to evaluate a genlm.control model on the LiveCodeBench domain.

• Task: Generate a correct Python program for a competitive-programming problem (stdin/stdout or a function to implement).
• Data: LiveCodeBench code_generation_lite (Jain et al., 2024).

Setup¶

First, install the dependencies for this domain. In the root directory, run:

pip install -e .[livecodebench] genlm-control

Usage¶

Initialize the dataset and evaluator¶

from_hf downloads and decodes a release of code_generation_lite. cumulative=True gives the official version_tag semantics (all problems through that release); start_date restricts the contest window (the default 2024-01-01 is after the Llama-3.x cutoffs). The first call needs network access to populate the Hugging Face cache.

In [ ]:

from genlm.eval.domains.livecodebench import (
    LiveCodeBenchDataset, LiveCodeBenchEvaluator
)

dataset = LiveCodeBenchDataset.from_hf(
    release="release_v6",
    start_date="2024-01-01",
    max_instances=8,
)

print("Instances loaded:", len(dataset))
evaluator = LiveCodeBenchEvaluator(timeout_seconds=6.0)

from genlm.eval.domains.livecodebench import ( LiveCodeBenchDataset, LiveCodeBenchEvaluator ) dataset = LiveCodeBenchDataset.from_hf( release="release_v6", start_date="2024-01-01", max_instances=8, ) print("Instances loaded:", len(dataset)) evaluator = LiveCodeBenchEvaluator(timeout_seconds=6.0)

Inspect dataset¶

In [ ]:

first = next(iter(dataset))
print("Question ID:", first.instance_id)
print("Difficulty:", first.difficulty)
print("Test type:", first.testtype)
print("Prompt preview:\n", (first.question_content[:500] + "...") if len(first.question_content) > 500 else first.question_content)

first = next(iter(dataset)) print("Question ID:", first.instance_id) print("Difficulty:", first.difficulty) print("Test type:", first.testtype) print("Prompt preview:\n", (first.question_content[:500] + "...") if len(first.question_content) > 500 else first.question_content)

Model Adaptor¶

default_prompt_formatter builds the official lcb_runner prompt (chat template for instruct models); we sample unconstrained and apply DEFAULT_STOP (lcb_runner's --stop "###"). The two potentials below (runtime-no-error and public-test feedback) are optional and shown afterwards.

In [ ]:

from genlm.control import PromptedLLM, direct_token_sampler
from genlm.eval import ModelOutput, ModelResponse
from genlm.eval.domains.livecodebench import DEFAULT_STOP, default_prompt_formatter

# Load an instruct LLM (chat template applied automatically below).
LLM = PromptedLLM.from_name("meta-llama/Llama-3.1-8B-Instruct", temperature=0.2)


async def model(instance, output_dir, replicate):
    # Build the official LCB prompt for this instance.
    LLM.prompt_ids = default_prompt_formatter(
        LLM.model.tokenizer, instance, use_chat_format=True
    )

    # Unconstrained sampling from the LLM (no constraint potential).
    sampler = direct_token_sampler(LLM)
    sequences = await sampler.smc(
        n_particles=5,
        ess_threshold=0.5,
        max_tokens=512,
    )

    def truncate(text):
        # Truncate at the official lcb_runner stop sequence (vLLM --stop "###").
        for stop in DEFAULT_STOP:
            text = text.split(stop)[0]
        return text

    return ModelOutput(
        responses=[
            ModelResponse(response=truncate(sequence), weight=prob)
            for sequence, prob in sequences.decoded_posterior.items()
        ],
    )

from genlm.control import PromptedLLM, direct_token_sampler from genlm.eval import ModelOutput, ModelResponse from genlm.eval.domains.livecodebench import DEFAULT_STOP, default_prompt_formatter # Load an instruct LLM (chat template applied automatically below). LLM = PromptedLLM.from_name("meta-llama/Llama-3.1-8B-Instruct", temperature=0.2) async def model(instance, output_dir, replicate): # Build the official LCB prompt for this instance. LLM.prompt_ids = default_prompt_formatter( LLM.model.tokenizer, instance, use_chat_format=True ) # Unconstrained sampling from the LLM (no constraint potential). sampler = direct_token_sampler(LLM) sequences = await sampler.smc( n_particles=5, ess_threshold=0.5, max_tokens=512, ) def truncate(text): # Truncate at the official lcb_runner stop sequence (vLLM --stop "###"). for stop in DEFAULT_STOP: text = text.split(stop)[0] return text return ModelOutput( responses=[ ModelResponse(response=truncate(sequence), weight=prob) for sequence, prob in sequences.decoded_posterior.items() ], )

Run the evaluation¶

In [ ]:

from genlm.eval import run_evaluation

results = await run_evaluation(
    dataset=dataset,
    model=model,
    evaluator=evaluator,
    max_instances=2,
    n_replicates=1,
    verbosity=1,
    #output_dir="livecodebench_results", #optionally save the results to a directory
)

from genlm.eval import run_evaluation results = await run_evaluation( dataset=dataset, model=model, evaluator=evaluator, max_instances=2, n_replicates=1, verbosity=1, #output_dir="livecodebench_results", #optionally save the results to a directory )

Optional: steer with the runtime-no-error potential¶

LCBRuntimeNoErrorPotential scores a generation 0.0 while its code runs without raising and -inf once it provably cannot (an unfixable syntax error, or a runtime error in already-complete code). Wrong answers are tolerated, so a correct-but-unfinished solution is never killed. It executes against the problem's public test inputs only. Plug it into a sampler with AWRS, exactly like the DS-1000 example.

In [ ]:

from genlm.control import AWRS
from genlm.eval.domains.livecodebench import LCBRuntimeNoErrorPotential


async def model_runtime_no_error(instance, output_dir, replicate):
    LLM.prompt_ids = default_prompt_formatter(
        LLM.model.tokenizer, instance, use_chat_format=True
    )

    # 0.0 while the code runs without error on the public inputs, -inf once it
    # provably cannot. Uses only public tests, so no private-test leakage.
    potential = LCBRuntimeNoErrorPotential(
        public_eval_sample=instance.public_eval_sample,
        extraction_style="generic",  # match the evaluator's code extraction
        timeout_seconds=6.0,
    ).coerce(LLM)

    sampler = AWRS(LLM, potential)
    sequences = await sampler.smc(n_particles=5, ess_threshold=0.5, max_tokens=512)

    def truncate(text):
        for stop in DEFAULT_STOP:
            text = text.split(stop)[0]
        return text

    return ModelOutput(responses=[
        ModelResponse(response=truncate(seq), weight=prob)
        for seq, prob in sequences.decoded_posterior.items()
    ])

from genlm.control import AWRS from genlm.eval.domains.livecodebench import LCBRuntimeNoErrorPotential async def model_runtime_no_error(instance, output_dir, replicate): LLM.prompt_ids = default_prompt_formatter( LLM.model.tokenizer, instance, use_chat_format=True ) # 0.0 while the code runs without error on the public inputs, -inf once it # provably cannot. Uses only public tests, so no private-test leakage. potential = LCBRuntimeNoErrorPotential( public_eval_sample=instance.public_eval_sample, extraction_style="generic", # match the evaluator's code extraction timeout_seconds=6.0, ).coerce(LLM) sampler = AWRS(LLM, potential) sequences = await sampler.smc(n_particles=5, ess_threshold=0.5, max_tokens=512) def truncate(text): for stop in DEFAULT_STOP: text = text.split(stop)[0] return text return ModelOutput(responses=[ ModelResponse(response=truncate(seq), weight=prob) for seq, prob in sequences.decoded_posterior.items() ])

Optional: self-repair with public-test feedback¶

LCBPublicTestPotential runs only the public (example) tests. prefix is a no-op (it never kills a partial generation) and complete returns a soft, finite penalty for failing tests (never -inf), so a wrong-but-recoverable solution survives. Its main use is the structured feedback: generate a first turn, and if the public tests fail, build a repair prompt with format_repair_prompt and generate again.

In [ ]:

from genlm.eval.domains.livecodebench import (
    LCBPublicTestPotential, format_repair_prompt
)


async def _generate(prompt_ids):
    LLM.prompt_ids = prompt_ids
    sequences = await direct_token_sampler(LLM).smc(
        n_particles=5, ess_threshold=0.5, max_tokens=512
    )
    # Take the highest-weight sample and truncate at the lcb_runner stop.
    seq, _ = max(sequences.decoded_posterior.items(), key=lambda kv: kv[1])
    for stop in DEFAULT_STOP:
        seq = seq.split(stop)[0]
    return seq


async def model_self_repair(instance, output_dir, replicate, max_turns=2):
    public = LCBPublicTestPotential(
        public_eval_sample=instance.public_eval_sample, extraction_style="generic"
    )

    prompt_ids = default_prompt_formatter(
        LLM.model.tokenizer, instance, use_chat_format=True
    )
    generation = await _generate(prompt_ids)

    # Re-prompt with the public-test feedback until the public tests pass.
    for _ in range(max_turns - 1):
        feedback = public.run_public_tests(generation)
        if feedback.all_passed:
            break
        prompt_ids = format_repair_prompt(
            LLM.model.tokenizer, instance, generation, feedback, use_chat_format=True
        )
        generation = await _generate(prompt_ids)

    return ModelOutput(responses=[ModelResponse(response=generation, weight=1.0)])

from genlm.eval.domains.livecodebench import ( LCBPublicTestPotential, format_repair_prompt ) async def _generate(prompt_ids): LLM.prompt_ids = prompt_ids sequences = await direct_token_sampler(LLM).smc( n_particles=5, ess_threshold=0.5, max_tokens=512 ) # Take the highest-weight sample and truncate at the lcb_runner stop. seq, _ = max(sequences.decoded_posterior.items(), key=lambda kv: kv[1]) for stop in DEFAULT_STOP: seq = seq.split(stop)[0] return seq async def model_self_repair(instance, output_dir, replicate, max_turns=2): public = LCBPublicTestPotential( public_eval_sample=instance.public_eval_sample, extraction_style="generic" ) prompt_ids = default_prompt_formatter( LLM.model.tokenizer, instance, use_chat_format=True ) generation = await _generate(prompt_ids) # Re-prompt with the public-test feedback until the public tests pass. for _ in range(max_turns - 1): feedback = public.run_public_tests(generation) if feedback.all_passed: break prompt_ids = format_repair_prompt( LLM.model.tokenizer, instance, generation, feedback, use_chat_format=True ) generation = await _generate(prompt_ids) return ModelOutput(responses=[ModelResponse(response=generation, weight=1.0)])

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024. URL https://arxiv.org/abs/2403.07974