> ## Documentation Index
> Fetch the complete documentation index at: https://docs.runloop.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview of Benchmarks & Scenarios on Runloop

> Make your agent better and more reliable with Runloop's tools for benchmarking.

Benchmarks are frequently cited when comparing model performance, but evaluation is not all you can do with benchmarks. Runloop provides a suite of tools to help you evaluate and improve your agent's performance, detect regressions, and fix common problems.

The central challenge in working with benchmarks is one of scale: running a single benchmark on one machine can take weeks, if not months. Runloop allows you to run benchmarks at scale in a secure environment.

## Orchestrated vs Interactive Benchmarks

Runloop supports two ways to run benchmarks:

<CardGroup cols={2}>
  <Card title="Orchestrated Benchmarks" icon="cloud" href="/docs/benchmarks/orchestrated-benchmarks">
    **Recommended for most users.** Submit a benchmark job via the CLI and let
    Runloop handle everything: provisioning devboxes, running agents, scoring
    results, and aggregating outputs. Compare multiple agents side-by-side with
    a single command.

    Best for:

    * Running a full benchmark suite
    * Comparing multiple agents
    * Reinforcement learning
    * CI/CD integration
  </Card>

  <Card title="Interactive Benchmarks" icon="code" href="/docs/benchmarks/public-benchmarks">
    For users who need fine-grained control. Use the SDK to drive benchmark
    execution step-by-step, with full access to the devbox at any point during
    the run.

    Best for:

    * Debugging agent behavior
    * Customizing execution logic
    * Benchmark development
  </Card>
</CardGroup>

### When to Use Each Mode

| Use Case                                                             | Recommended Mode |
| -------------------------------------------------------------------- | ---------------- |
| Running a full benchmark suite                                       | Orchestrated     |
| Comparing multiple agents                                            | Orchestrated     |
| [Reinforcement learning](/docs/benchmarks/training-using-benchmarks) | Orchestrated     |
| CI/CD integration                                                    | Orchestrated     |
| Iterative development                                                | Orchestrated     |
| Debugging agent behavior                                             | Interactive      |
| Custom execution logic                                               | Interactive      |

## Main Features

Runloop enables you to customize every aspect of benchmark creation and execution:

* **[Orchestrated Benchmarks](/docs/benchmarks/orchestrated-benchmarks):** Run benchmarks at cloud scale using your agent or a public agent with a single CLI command. Runloop handles provisioning, execution, scoring, and teardown automatically.
* **[Public Benchmarks](/docs/benchmarks/public-benchmarks):** Run your agent against well-known open source benchmarks like terminal bench 2, AIME, and more.
* **[Custom Benchmarks](/docs/benchmarks/custom-benchmarks):** Craft your own scenarios and benchmarks to train or evaluate your agent on a private codebase or dataset.
* **[Custom Scorers](/docs/benchmarks/custom-scorers):** Create custom scorers to evaluate agents across multiple dimensions, such as security, cost, performance, and compliance.
* **[Training Using Benchmarks](/docs/benchmarks/training-using-benchmarks):** Learn how benchmark runs and scores can support reinforcement learning workflows and targeted agent improvement.
* **[Reports & Insights](/docs/tools/dashboard):** Identify problems and visualize your agent's performance changes in the Runloop dashboard.

## Key Concepts

Whether you're running orchestrated or interactive benchmarks, you'll work with the following key concepts:

* **[Scenario](/docs/benchmarks/creating-scenarios)**: A scenario is a single, self-contained test case or task where an agent is given a problem and is expected to modify a target environment to solve it.
* **[Benchmark](/docs/benchmarks/custom-benchmarks)**: A set of Scenarios that can be run together to produce an overall performance score. Benchmarks can be made up of any number and combination of Scenarios -- even Scenarios from other Benchmarks.
* **[Scoring Function / Scorer](/docs/benchmarks/custom-scorers)**: A script or function that is invoked to grade the performance of a Scenario from 0.0 to 1.0.

## Getting Started

<Tabs>
  <Tab title="Orchestrated (Recommended)">
    Run a benchmark with a single command:

    ```bash theme={null}
    rli benchmark-job run \
      --agent "claude-code:claude-sonnet-4-6" \
      --benchmark "terminal-bench-2" \
      -n "my-first-terminal-bench-2-run"
    ```

    Learn more in the [Orchestrated Benchmarks
    guide](/docs/benchmarks/orchestrated-benchmarks).
  </Tab>

  <Tab title="Interactive">
    Use the SDK for step-by-step control:

    <CodeGroup>
      ```python Python theme={null}
      benchmarks = await runloop.api.benchmarks.list_public()
      scenario_run = await runloop.api.scenarios.start_run(
          scenario_id=benchmarks[0].scenario_ids[0]
      )
      ```

      ```typescript TypeScript theme={null}
      const benchmarks = await runloop.api.benchmarks.listPublic();
      const scenarioRun = await runloop.api.scenarios.startRun({
        scenario_id: benchmarks[0].scenarioIds[0],
      });
      ```
    </CodeGroup>

    Then take control of each scenario and start, score, and complete
    individual scenario runs. Learn more in the [Public Benchmarks
    guide](/docs/benchmarks/public-benchmarks).
  </Tab>
</Tabs>