Quickstart - Controlling a remote computer in a Runloop Devbox

Start with the Runloop Quickstart to use the examples below.

Introduction

This guide will walk you through using the Runloop SDK to control a remote computer inside a Runloop Devbox. The Runloop API provides a computer-ready Devbox, enabling AI agents to interact with the system programmatically.

Setup Your Environment

Follow the instructions in the Runloop Quickstart to set up your environment to use the Runloop SDK.

Create a Devbox and Start the Computer Tool

Create your Devbox, wait for it to be ready, and retrieve the connection details:

# Create a Devbox with a computer instance                                       
computer = runloop.api.devboxes.computers.create()

# Wait for the Devbox to be fully running
devbox = runloop.devbox.from_id(computer.devbox.id)
await devbox.await_running()

# Retrieve the computer connection details
devbox_id = computer.devbox.id
display_url = computer.live_screen_url

// Create a Devbox with a computer instance
const computer = await runloop.api.devboxes.computers.create();

// Wait for the Devbox to be fully running
const devbox = await runloop.devbox.fromId(computer.devbox.id);
await devbox.awaitRunning();

// Retrieve the computer connection details
const devboxId = computer.devbox.id;
const displayUrl = computer.live_screen_url;

Interacting with the Computer

The computer-ready Devbox offers a suite of Computer Tools for agent interactions. The available actions include:

Keyboard interaction: key, type
Mouse interaction: mouse_move, left_click, left_click_drag, right_click, middle_click, double_click
Screen interaction: screenshot, cursor_position

You can access these tools using the Runloop client as shown below:

# Keyboard usage
runloop.api.devboxes.computers.keyboard_interaction(
  devbox_id,
  action=action,
  text=text
)

# Mouse usage
runloop.api.devboxes.computers.mouse_interaction(
  devbox_id,
  action=action,
  coordinate={"x": int, "y": int}
)    

# Take a screenshot or retrieve current mouse coordinates
runloop.api.devboxes.computers.screen_interaction(
  devbox_id,
  action=action
)

// Keyboard usage
await runloop.api.devboxes.computers.keyboardInteraction(
  devboxId, 
  { action: action, 
    text: text }
);

// Mouse usage
await runloop.api.devboxes.computers.mouseInteraction(
  devboxId, 
  { action: action,
    coordinate: { x: number, y: number }}
);

// Take a screenshot or retrieve current mouse coordinates
await runloop.api.devboxes.computers.screenInteraction(
  devboxId, 
  { action: action }
);

Using API Tools with the Computer Tool

Once you create tools for your agent, you can integrate them with your preferred LLM. Here’s an example of integrating it with Anthropic’s Claude:

import Anthropic

anthropic_client = Anthropic(
  api_key="your-anthropic-api-key"
)

response = anthropic_client.messages.create(
  model="claude-3",
  max_tokens=300,
  messages=messages
  tools=[tool],
)

import Anthropic from '@anthropic-ai/sdk';

const anthropicClient = new Anthropic({
  apiKey: 'your-anthropic-api-key'
});

const response = await anthropicClient.messages.create({
  model: 'claude-3',
  maxTokens: 300,
  messages: messages,
  tools: [tool]
});

Different LLM providers have their own specific formats and requirements for defining and passing tools. Make sure to reference your LLM provider’s documentation for the correct implementation details of tool schemas and function calling.

Properly Freeing Resources

To ensure efficient resource management, always shut down the Devbox when you’re done:

await runloop.api.devboxes.shutdown(computer.devbox.id)

await runloop.api.devboxes.shutdown(computer.devbox.id);

Additional Resources

Runloop GitHub Repository - Explore more examples.
Runloop API Documentation - Official API reference.

Overview

Tools

Components

Benchmarks & Evals

Debugging

Quickstart - Controlling a remote computer in a Runloop Devbox

Introduction

Additional Resources

​Introduction

​Additional Resources

Introduction

Additional Resources