List BenchmarkRuns. - Runloop AI Docs

API Reference

Introduction

Devbox

The Devbox Object
Devbox Lifecycle
Devbox File Tools
Devbox Shell Tools
Devbox Network Tools
Devbox Persistence Tools
Devbox Observability Tools
Devbox Add-ons

Blueprint

The Blueprint Object
Blueprint Lifecycle
Blueprint Observability

Repository

The Repository Object
Repository Lifecycle

Code Scenario

Code Scenario Lifecycle
Scenarios Runs
Custom Scenario Scorer
Public Scenarios

Benchmark

GET

benchmarks

runs

import Runloop from '@runloop/api-client';

const client = new Runloop({
  bearerToken: process.env['RUNLOOP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  // Automatically fetches more pages as needed.
  for await (const benchmarkRunView of client.benchmarks.runs.list()) {
    console.log(benchmarkRunView.id);
  }
}

main();

{
  "runs": [
    {
      "id": "<string>",
      "benchmark_id": "<string>",
      "name": "<string>",
      "start_time_ms": 123,
      "duration_ms": 123,
      "state": "running",
      "score": 123,
      "pending_scenarios": [
        "<string>"
      ],
      "scenario_runs": [
        {
          "scenario_id": "<string>",
          "scenarioRunId": "<string>",
          "scoringResult": {
            "score": 123,
            "scoring_function_results": [
              {
                "score": 123,
                "scoring_function_name": "<string>",
                "output": "<string>",
                "state": "unknown"
              }
            ]
          }
        }
      ],
      "metadata": {}
    }
  ],
  "has_more": true,
  "total_count": 123,
  "remaining_count": 123
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

limit

integer

The limit of items to return. Default is 20.

starting_after

string

Load the next page of data starting after the item with the given ID.

benchmark_id

string

The Benchmark ID to filter by.

Response

200 - application/json

runs

object[]

required

List of BenchmarkRuns matching filter.

A BenchmarkRunView represents a run of a complete set of Scenarios, organized under a Benchmark.

runs.id

string

required

The ID of the BenchmarkRun.

runs.benchmark_id

string

required

The ID of the Benchmark.

runs.start_time_ms

integer

required

The time the benchmark run execution started (Unix timestamp milliseconds).

runs.state

enum<string>

required

The state of the BenchmarkRun.

Available options:

running,

completed

runs.pending_scenarios

string[]

required

List of Scenarios that need to be completed before benchmark can be completed.

runs.scenario_runs

object[]

required

List of Scenarios have been completed.

A BenchmarkRunScenariosListView represents a run of a complete set of Scenarios run, organized under a Benchmark.

runs.metadata

object

required

User defined metadata to attach to the benchmark run for organization.

runs.name

string | null

The name of the BenchmarkRun.

runs.duration_ms

integer | null

The duration for the BenchmarkRun to complete.

runs.score

number | null

The final score across the BenchmarkRun, present once completed. Calculated as sum of scenario scores / number of scenario runs.

has_more

boolean

required

total_count

integer

required

remaining_count

integer

required

Was this page helpful?

Update a Benchmark.Get a previously created BenchmarkRun.

import Runloop from '@runloop/api-client';

const client = new Runloop({
  bearerToken: process.env['RUNLOOP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  // Automatically fetches more pages as needed.
  for await (const benchmarkRunView of client.benchmarks.runs.list()) {
    console.log(benchmarkRunView.id);
  }
}

main();

{
  "runs": [
    {
      "id": "<string>",
      "benchmark_id": "<string>",
      "name": "<string>",
      "start_time_ms": 123,
      "duration_ms": 123,
      "state": "running",
      "score": 123,
      "pending_scenarios": [
        "<string>"
      ],
      "scenario_runs": [
        {
          "scenario_id": "<string>",
          "scenarioRunId": "<string>",
          "scoringResult": {
            "score": 123,
            "scoring_function_results": [
              {
                "score": 123,
                "scoring_function_name": "<string>",
                "output": "<string>",
                "state": "unknown"
              }
            ]
          }
        }
      ],
      "metadata": {}
    }
  ],
  "has_more": true,
  "total_count": 123,
  "remaining_count": 123
}