Your AI coding agent is capable of numerous tasks such as reading code, writing and preparing patches, and submitting commits to code repositories. A common problem with such agents is ensuring that they perform: without training, monitoring, tuning and optimization, your agent may be prone to making mistakes, experience regressions over time, and generally not deliver the best user experience. Runloop Benchmarking is a a suite of tools to help you address these issues and stay focused on building the best possible agent by training and evaluating your agent.

​
Main Features

Runloop Benchmarking includes several tools to save you time while optimizing your agent:
  • Run Public Benchmarks: Easily run your agent against a matrix of well-known and open source benchmarks, such as SWE-bench.
  • Run Custom Benchmarks: Create your own custom test cases, then evaluate the agent’s performance against them at scale.
  • Reports & Insights: As you run benchmarks over time, you will see how your agent’s performance changes in the Runloop dashboard.

​
Key Concepts

Whether you’re using public or custom benchmarks, you’ll keep the following key concepts in mind:
  • Scenario: A single test case where an agent is given a problem and is expected to modify a target environment to solve it. Scenarios help test AI agents in realistic coding environments. Benchmarks may use other language to describe a scenario, such as Task, TestCase or Example.
  • Scoring Function: A script or function that runs after the agent completes its task to validate whether the solution works. These functions generate a final score between 0 and 1 to indicate performance.
  • Benchmark: A collection of Scenarios designed to evaluate AI agents on a broader set of tasks. Benchmarks help measure agent capabilities systematically.
Next, learn how to run public benchmarks.