> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Interactive Public Benchmarks

> Run your agent against popular public benchmarks with full control over the execution process.

{/* TODO: sanity check the code below for typos. */}

<Note>
  **Looking to run benchmarks quickly?** For most use cases, we recommend
  [Orchestrated Benchmarks](/docs/benchmarks/orchestrated-benchmarks) which let
  you run full benchmark suites with a single CLI command. This page describes
  the **interactive** approach, which gives you fine-grained control over each
  scenario run and full access to the devbox at any point during execution.
</Note>

## Interactive Benchmarks Overview

Interactive benchmarks use the Runloop SDK to drive benchmark execution step-by-step. This approach is ideal when you need:

* Full control over the execution flow
* Direct access to the devbox during a run
* Custom logic between scenario steps
* Debugging and iterative development
* Synthetic trajectory generation

Each Benchmark contains a set of Scenarios based on each test in the dataset. The Scenario contains the **problem statement** that your agent
must work through, a pre-built **environment** containing all of the context needed to complete the job, and a built-in **scorer**
to properly evaluate the result for correctness.

When working with benchmarks, keep in mind that benchmark datasets are typically large and are therefore paged. Similarly, execution can take a long time, so you should prefer the `AsyncRunloop` client if you're working with Python.

## Viewing Public Benchmarks

We're constantly adding new supported datasets. To view the up-to-date list of supported public Benchmarks, use the following API call:

<CodeGroup>
  ```python Python theme={null}
  # Query to see the latest list of supported public benchmarks
  benchmarks = await runloop.api.benchmarks.list_public()
  ```

  ```typescript TypeScript theme={null}
  // Query to see the latest list of supported public benchmarks
  const benchmarks = await runloop.api.benchmarks.listPublic();
  ```
</CodeGroup>

<Note>
  Are we missing your favorite open source benchmark? Let us know at
  [support@runloop.ai](mailto:support@runloop.ai)
</Note>

Each Benchmark contains a set of Scenarios that correspond to a test-case in the evaluation dataset.

<CodeGroup>
  ```python Python theme={null}
  # The Benchmark definition contains a list of all scenarios
  # contained in the benchmark
  print(benchmarks[0].scenario_ids)
  ```

  ```typescript TypeScript theme={null}
  // The Benchmark definition contains a list of all scenarios
  // contained in the benchmark
  console.log(benchmarks[0].scenarioIds);
  ```
</CodeGroup>

## Running Scenarios & Benchmarks

Each Scenario can be **run** to evaluate an AI agent's performance. Running a scenario involves:

1. Initiating a scenario run.
2. Launching a development environment (devbox).
3. Running the agent against the problem statement.
4. Scoring the results.
5. Uploading traces for analysis.

### Run a single scenario from a public benchmark

Here's an example of how to run a single scenario from a public benchmark against your own agent.

First, create a **scenario run** to track the status and results of this run:

<CodeGroup>
  ```python Python theme={null}
  # Note: we are using the async client here.
  scenario_id = benchmarks[0].scenario_ids[0]
  scenario_run = await runloop.api.scenarios.start_run(
    scenario_id=scenario_id,
    run_name="marshmallow-code__marshmallow-1359 test run"
  )
  ```

  ```typescript TypeScript theme={null}
  const scenarioId = benchmarks[0].scenarioIds[0];
  const scenarioRun = await runloop.api.scenarios.startRun({
    scenario_id: scenarioId,
    run_name: 'marshmallow-code__marshmallow-1359 test run',
  });
  ```
</CodeGroup>

When starting a run, Runloop will create a Devbox with the *environment*
specified by the test requirements.

Wait for the devbox used by the scenario to become ready:

<CodeGroup>
  ```python Python theme={null}
  # Note the async client is used here.
  devbox = runloop.devbox.from_id(scenario_run.devbox_id)
  await devbox.await_running()
  ```

  ```typescript TypeScript theme={null}
  const devbox = runloop.devbox.fromId(scenarioRun.devbox_id);
  await devbox.awaitRunning();
  ```
</CodeGroup>

Now, run your agent. How and where your agent runs is up to you. Here's an example of an agent that uses the problem statement as the prompt:

<CodeGroup>
  ```python Python theme={null}
  problem_statement = scenario_run.scenario.input_context.problem_statement
  my_agent = MyAgent(prompt=problem_statement)
  ```

  ```typescript TypeScript theme={null}
  const problemStatement = scenarioRun.scenario.input_context.problem_statement
  const myAgent = new MyAgent({prompt: problemStatement});
  ```
</CodeGroup>

Finally, run the scoring function to validate the agent's performance:

<CodeGroup>
  ```python Python theme={null}
  # Run the scoring function. Automatically marks the scenario run as done.
  results = await runloop.api.scenarios.runs.score_and_await(scenario_run.id)
  print(results)
  ```

  ```typescript TypeScript theme={null}
  // Run the scoring function. Automatically marks the scenario run as done.
  const results = await runloop.api.scenarios.runs.scoreAndAwait(scenarioRun.id);
  console.log(results);
  ```
</CodeGroup>

### Perform a full benchmark run of a public benchmark

Once your agent is excelling at an individual scenario, you will want to test
against all Scenarios for a given Benchmark.

Here's an example of how to perform a full benchmark run of a public benchmark.

<CodeGroup>
  ```python Python theme={null}
  # Start a full run of the first public benchmark returned
  benchmark_run = await runloop.api.benchmarks.start_run(
    benchmark_id=benchmarks[0].id,
    run_name="optional run name"
  )

  # Example: iterate scenarios (serialize or parallelize as desired)

  for scenario_id in benchmark_run.pending_scenarios:
      scenario_run = await runloop.api.scenarios.start_run(
          scenario_id=scenario_id,
          benchmark_run_id=benchmark_run.id
      )
      devbox = runloop.devbox.from_id(scenario_run.devbox_id)
      await devbox.await_running()

      # Run your agent here using scenario_run.scenario.input_context.problem_statement
      my_agent = MyAgent(
          prompt=scenario_run.scenario.input_context.problem_statement
      )
      await runloop.api.scenarios.runs.score(scenario_run.id)

  ```

  ```typescript TypeScript theme={null}
  // Start a full run of the first public benchmark returned
  const benchmarkRun = await runloop.api.benchmarks.startRun({
    benchmark_id: benchmarks[0].id,
    run_name: 'optional run name',
  });

  // This shows a serialized scenario by scenario runner but can also run in any
  // level of parallelism
  for (const scenarioId of benchmarkRun.pending_scenarios) {
    // create a scenario run tied to the benchmark run
    const scenarioRun = await runloop.api.scenarios.startRunAndAwaitEnvReady({
      scenario_id: scenarioId,
      benchmark_run_id: benchmarkRun.id,
    });

    const devbox = runloop.devbox.fromId(scenarioRun.devbox_id);
    await devbox.awaitRunning();

    // Run your agent on the problem at hand to see how it does
    // (code will vary by agent implementation).
    const myAgent = new MyAgent({
      prompt: scenarioRun.scenario.input_context.problem_statement,
      // other args
    });

    // Score and complete the run. This will also properly shut down the devbox.
    const validateResults = await runloop.api.scenarios.runs.scoreAndComplete(
      scenarioRun.id
    );
  }

  // Benchmark runs will end automatically when no more pending scenarios but also
  // can optionally just end a benchmark run early
  await runloop.api.benchmarks.runs.complete(benchmarkRun.id);
  ```
</CodeGroup>

Interactive benchmarks make it easy to start evaluating your agent against industry standard coding evals with full control over the execution process.

## Next Steps

* **[Orchestrated Benchmarks](/docs/benchmarks/orchestrated-benchmarks)**: Run full benchmarks at cloud scale with a single CLI command
* **[Custom Benchmarks](/docs/benchmarks/custom-benchmarks)**: Create your own benchmarks with custom scenarios and scorers
* **[Custom Scorers](/docs/benchmarks/custom-scorers)**: Build domain-specific scoring functions