API Reference
Devbox
- The Devbox Object
- Devbox Lifecycle
- Devbox File Tools
- Devbox Shell Tools
- Devbox Network Tools
- Devbox Persistence Tools
- Devbox Observability Tools
- Devbox Add-ons
Blueprint
- The Blueprint Object
- Blueprint Lifecycle
- Blueprint Observability
Repository
- The Repository Object
- Repository Lifecycle
Code Scenario
- Code Scenario Lifecycle
- Scenarios Runs
- Custom Scenario Scorer
- Public Scenarios
Benchmark
- Benchmark Lifecycle
- Benchmark Runs
- Public Benchmarks
Validate a custom scenario scorer.
Validate a scenario scorer.
import Runloop from '@runloop/api-client';
const client = new Runloop({
bearerToken: process.env['RUNLOOP_API_KEY'], // This is the default and can be omitted
});
async function main() {
const response = await client.scenarios.scorers.validate('id', { scoring_context: {} });
console.log(response.name);
}
main();{
"name": "<string>",
"environment_parameters": {
"blueprint_id": "<string>",
"snapshot_id": "<string>",
"prebuilt_id": "<string>",
"launch_parameters": {
"launch_commands": [
"<string>"
],
"resource_size_request": "X_SMALL",
"keep_alive_time_seconds": 123,
"available_ports": [
123
],
"after_idle": {
"idle_time_seconds": 123,
"on_idle": "shutdown"
},
"custom_cpu_cores": 123,
"custom_gb_memory": 123
},
"working_directory": "<string>"
},
"scoring_context": {},
"scoring_result": {
"score": 123,
"scoring_function_name": "<string>",
"output": "<string>",
"state": "unknown"
}
}Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Path Parameters
The Scorer ID.
Body
Json context that gets passed to the custom scorer
The Environment in which the Scenario will run.
Use the blueprint with matching ID.
Use the snapshot with matching ID.
Use the prebuilt with matching ID.
Optional launch parameters to apply to the devbox environment at launch.
Set of commands to be run at launch time, before the entrypoint process is run.
Manual resource configuration for Devbox. If not set, defaults will be used.
X_SMALL, SMALL, MEDIUM, LARGE, X_LARGE, XX_LARGE, CUSTOM_SIZE Time in seconds after which Devbox will automatically shutdown. Default is 1 hour.
A list of ports to make available on the Devbox. Only ports made available will be surfaced to create tunnels via the 'createTunnel' API.
Configure Devbox lifecycle based on idle activity. If after_idle is set, Devbox will ignore keep_alive_time_seconds.
custom resource size, number of cpu cores, must be multiple of 2.
custom memory size, number in Gi, must be a multiple of 2.
The working directory where the agent is expected to fulfill the scenario. Scoring functions also run from the working directory.
Response
Name of the custom scorer.
Json context that gets passed to the custom scorer
Result of the scoring function.
Final score for the given scoring function.
Scoring function name that ran.
Log output of the scoring function.
The state of the scoring function application.
unknown, complete, error The Environment in which the Scenario will run.
Use the blueprint with matching ID.
Use the snapshot with matching ID.
Use the prebuilt with matching ID.
Optional launch parameters to apply to the devbox environment at launch.
Set of commands to be run at launch time, before the entrypoint process is run.
Manual resource configuration for Devbox. If not set, defaults will be used.
X_SMALL, SMALL, MEDIUM, LARGE, X_LARGE, XX_LARGE, CUSTOM_SIZE Time in seconds after which Devbox will automatically shutdown. Default is 1 hour.
A list of ports to make available on the Devbox. Only ports made available will be surfaced to create tunnels via the 'createTunnel' API.
Configure Devbox lifecycle based on idle activity. If after_idle is set, Devbox will ignore keep_alive_time_seconds.
custom resource size, number of cpu cores, must be multiple of 2.
custom memory size, number in Gi, must be a multiple of 2.
The working directory where the agent is expected to fulfill the scenario. Scoring functions also run from the working directory.
Was this page helpful?
import Runloop from '@runloop/api-client';
const client = new Runloop({
bearerToken: process.env['RUNLOOP_API_KEY'], // This is the default and can be omitted
});
async function main() {
const response = await client.scenarios.scorers.validate('id', { scoring_context: {} });
console.log(response.name);
}
main();{
"name": "<string>",
"environment_parameters": {
"blueprint_id": "<string>",
"snapshot_id": "<string>",
"prebuilt_id": "<string>",
"launch_parameters": {
"launch_commands": [
"<string>"
],
"resource_size_request": "X_SMALL",
"keep_alive_time_seconds": 123,
"available_ports": [
123
],
"after_idle": {
"idle_time_seconds": 123,
"on_idle": "shutdown"
},
"custom_cpu_cores": 123,
"custom_gb_memory": 123
},
"working_directory": "<string>"
},
"scoring_context": {},
"scoring_result": {
"score": 123,
"scoring_function_name": "<string>",
"output": "<string>",
"state": "unknown"
}
}