computer/libs/python/agent/agent/proxy/README.md

ComputerAgent Proxy

A proxy server that exposes ComputerAgent functionality over HTTP and P2P (WebRTC) connections. This allows remote clients to interact with ComputerAgent instances through a simple REST-like API.

Installation

The proxy requires additional dependencies:

# For HTTP server
pip install starlette uvicorn

# For P2P server (optional)
pip install peerjs-python aiortc

Usage

Starting the Server

# HTTP server only (default)
python -m agent.proxy.cli

# Custom host/port
python -m agent.proxy.cli --host 0.0.0.0 --port 8080

# P2P server only
python -m agent.proxy.cli --mode p2p

# Both HTTP and P2P
python -m agent.proxy.cli --mode both

API Endpoints

POST /responses

Process a request using ComputerAgent and return the first result.

Request Format:

{
  "model": "anthropic/claude-3-5-sonnet-20241022",
  "input": "Your instruction here",
  # Optional agent configuration, passed to ComputerAgent constructor
  "agent_kwargs": { 
    "save_trajectory": true,
    "verbosity": 20
  },
  # Optional computer configuration, passed to the Computer constructor
  "computer_kwargs": {
    "os_type": "linux",
    "provider_type": "cloud"
  },
  # Optional environment overrides for this request
  "env": {
    "OPENROUTER_API_KEY": "your-openrouter-api-key",
    # etc.
    # ref: https://docs.litellm.ai/docs/proxy/config_settings#environment-variables---reference
  }
}

Multi-modal Request:

{
  "model": "anthropic/claude-3-5-sonnet-20241022",
  "input": [
    {
      "role": "user",
      "content": [
        {"type": "input_text", "text": "what is in this image?"},
        {
          "type": "input_image",
          "image_url": "https://example.com/image.jpg"
        }
      ]
    }
  ]
}

Response Format:

{
  "result": {
    // Agent response data
  },
  "model": "anthropic/claude-3-5-sonnet-20241022"
}

GET /health

Health check endpoint.

cURL Examples

# Simple text request
curl http://localhost:8000/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet-20241022",
    "input": "Tell me a three sentence bedtime story about a unicorn.",
    "env": {"CUA_API_KEY": "override-key-for-this-request"}
  }'

# Multi-modal request
curl http://localhost:8000/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet-20241022",
    "input": [
      {
        "role": "user",
        "content": [
          {"type": "input_text", "text": "what is in this image?"},
          {
            "type": "input_image", 
            "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
          }
        ]
      }
    ]
  }'

P2P Usage

The P2P server allows WebRTC connections for direct peer-to-peer communication:

# Connect to P2P proxy
from peerjs import Peer, PeerOptions
import json

peer = Peer(id="client", peer_options=PeerOptions(host="localhost", port=9000))
await peer.start()

connection = peer.connect("computer-agent-proxy")

# Send request
request = {
    "model": "anthropic/claude-3-5-sonnet-20241022",
    "input": "Hello from P2P!"
}
await connection.send(json.dumps(request))

Configuration

Request Parameters

• model: Model string (required) - e.g., "anthropic/claude-3-5-sonnet-20241022"
• input: String or message array (required)
• agent_kwargs: Optional agent configuration
• computer_kwargs: Optional computer configuration
• env: Object - key/value environment variables to override for this request

Agent Configuration (agent_kwargs)

Common options:

• save_trajectory: Boolean - Save conversation trajectory
• verbosity: Integer - Logging level (10=DEBUG, 20=INFO, etc.)
• max_trajectory_budget: Float - Budget limit for trajectory

Computer Configuration (computer_kwargs)

Common options:

• os_type: String - "linux", "windows", "macos"
• provider_type: String - "cloud", "local", "docker"
• name: String - Instance name
• api_key: String - Provider API key

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   HTTP Client   │    │   P2P Client    │    │  Direct Usage   │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          ▼                      ▼                      ▼
┌─────────────────────────────────────────────────────────────────┐
│                    ProxyServer                                  │
├─────────────────────────────────────────────────────────────────┤
│                 ResponsesHandler                                │
├─────────────────────────────────────────────────────────────────┤
│              ComputerAgent + Computer                           │
└─────────────────────────────────────────────────────────────────┘

Examples

See examples.py for complete usage examples:

# Run HTTP tests
python agent/proxy/examples.py

# Show curl examples
python agent/proxy/examples.py curl

# Test P2P (requires peerjs-python)
python agent/proxy/examples.py p2p

Error Handling

The proxy returns structured error responses:

{
  "success": false,
  "error": "Error description",
  "model": "model-used"
}

Common errors:

• Missing required parameters (model, input)
• Invalid JSON in request body
• Agent execution errors
• Computer setup failures

Limitations

• Returns only the first result from agent.run() (as requested)
• P2P requires peerjs-python and signaling server
• Computer instances are created per request (not pooled)
• No authentication/authorization built-in