Create a Scenario. - Runloop AI Docs

API Reference

Introduction

Devbox

The Devbox Object
Devbox Lifecycle
Devbox File Tools
Devbox Shell Tools
Devbox Network Tools
Devbox Persistence Tools
Devbox Observability Tools
Devbox Add-ons

Blueprint

The Blueprint Object
Blueprint Lifecycle
Blueprint Observability

Repository

The Repository Object
Repository Lifecycle

Code Scenario

Code Scenario Lifecycle
Scenarios Runs
Custom Scenario Scorer
Public Scenarios

Benchmark

Benchmark Lifecycle
Benchmark Runs
Public Benchmarks

POST

scenarios

import Runloop from '@runloop/api-client';

const client = new Runloop({
  bearerToken: process.env['RUNLOOP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const scenario = await client.scenarios.create({
    input_context: { problem_statement: 'problem_statement' },
    is_public: true,
    name: 'name',
    scoring_contract: {
      scoring_function_parameters: [
        {
          name: 'name',
          scorer: { pattern: 'pattern', search_directory: 'search_directory', type: 'ast_grep_scorer' },
          weight: 0,
        },
      ],
    },
  });

  console.log(scenario.id);
}

main();

{
  "id": "<string>",
  "name": "<string>",
  "environment": {
    "blueprint_id": "<string>",
    "snapshot_id": "<string>",
    "prebuilt_id": "<string>",
    "launch_parameters": {
      "launch_commands": [
        "<string>"
      ],
      "resource_size_request": "X_SMALL",
      "keep_alive_time_seconds": 123,
      "available_ports": [
        123
      ],
      "after_idle": {
        "idle_time_seconds": 123,
        "on_idle": "shutdown"
      },
      "custom_cpu_cores": 123,
      "custom_gb_memory": 123
    },
    "working_directory": "<string>"
  },
  "input_context": {
    "problem_statement": "<string>",
    "additional_context": {}
  },
  "scoring_contract": {
    "scoring_function_parameters": [
      {
        "name": "<string>",
        "scorer": {
          "lang": "<string>",
          "search_directory": "<string>",
          "pattern": "<string>",
          "type": "ast_grep_scorer"
        },
        "weight": 123
      }
    ]
  },
  "metadata": {},
  "reference_output": "<string>",
  "is_public": true
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

name

string

required

Name of the scenario.

input_context

object

required

The input context for the Scenario.

scoring_contract

object

required

The scoring contract for the Scenario.

is_public

boolean

required

Whether this scenario is public.

environment_parameters

object

The Environment in which the Scenario will run.

environment_parameters.blueprint_id

string | null

Use the blueprint with matching ID.

environment_parameters.snapshot_id

string | null

Use the snapshot with matching ID.

environment_parameters.prebuilt_id

string | null

Use the prebuilt with matching ID.

environment_parameters.launch_parameters

object

Optional launch parameters to apply to the devbox environment at launch.

environment_parameters.working_directory

string | null

The working directory where the agent is expected to fulfill the scenario. Scoring functions also run from the working directory.

metadata

object | null

User defined metadata to attach to the scenario for organization.

reference_output

string | null

A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.

Response

200 - application/json

A ScenarioView represents a repeatable AI coding evaluation test, complete with initial environment and scoring contract.

string

required

The ID of the Scenario.

name

string

required

The name of the Scenario.

input_context

object

required

The input context for the Scenario.

scoring_contract

object

required

The scoring contract for the Scenario.

metadata

object

required

User defined metadata to attach to the scenario for organization.

environment

object

The Environment in which the Scenario is run.

reference_output

string | null

A string representation of the reference output to solve the scenario. Commonly can be the result of a git diff or a sequence of command actions to apply to the environment.

is_public

boolean

Whether this scenario is public.

Was this page helpful?

List Scenarios.Get a Scenario.

import Runloop from '@runloop/api-client';

const client = new Runloop({
  bearerToken: process.env['RUNLOOP_API_KEY'], // This is the default and can be omitted
});

async function main() {
  const scenario = await client.scenarios.create({
    input_context: { problem_statement: 'problem_statement' },
    is_public: true,
    name: 'name',
    scoring_contract: {
      scoring_function_parameters: [
        {
          name: 'name',
          scorer: { pattern: 'pattern', search_directory: 'search_directory', type: 'ast_grep_scorer' },
          weight: 0,
        },
      ],
    },
  });

  console.log(scenario.id);
}

main();

{
  "id": "<string>",
  "name": "<string>",
  "environment": {
    "blueprint_id": "<string>",
    "snapshot_id": "<string>",
    "prebuilt_id": "<string>",
    "launch_parameters": {
      "launch_commands": [
        "<string>"
      ],
      "resource_size_request": "X_SMALL",
      "keep_alive_time_seconds": 123,
      "available_ports": [
        123
      ],
      "after_idle": {
        "idle_time_seconds": 123,
        "on_idle": "shutdown"
      },
      "custom_cpu_cores": 123,
      "custom_gb_memory": 123
    },
    "working_directory": "<string>"
  },
  "input_context": {
    "problem_statement": "<string>",
    "additional_context": {}
  },
  "scoring_contract": {
    "scoring_function_parameters": [
      {
        "name": "<string>",
        "scorer": {
          "lang": "<string>",
          "search_directory": "<string>",
          "pattern": "<string>",
          "type": "ast_grep_scorer"
        },
        "weight": 123
      }
    ]
  },
  "metadata": {},
  "reference_output": "<string>",
  "is_public": true
}