computer/notebooks/computer_nb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computer\n",
    "\n",
    "This notebook demonstrates how to use Computer to operate sandbox VMs programmatically, either using Cua Cloud Sandbox or local Lume VMs on Apple Silicon macOS systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If running outside of the monorepo:\n",
    "# %pip install \"cua-computer[all]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 1: Cua Cloud Sandbox\n",
    "\n",
    "Cua Cloud Sandbox provides remote VMs that can be accessed from any system without local setup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "### Prerequisites for Cloud Sandbox\n\nTo use Cua Cloud Sandbox, you need to:\n1. Sign up at https://cua.ai\n2. Create a Cloud Sandbox\n3. Generate an API Key\n\nOnce you have these, you can connect to your Cloud Sandbox using its name."
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get API key and container name from environment or prompt user\n",
    "import os\n",
    "\n",
    "cua_api_key = os.getenv(\"CUA_API_KEY\") or input(\"Enter your Cua API Key: \")\n",
    "container_name = os.getenv(\"CONTAINER_NAME\") or input(\"Enter your Cloud Container name: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choose the OS type for your sandbox (linux or macos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os_type = (\n",
    "    input(\"Enter the OS type of your sandbox (linux/macos) [default: linux]: \").lower() or \"linux\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to your Cloud Sandbox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from computer import Computer, VMProviderType"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Connect to your existing Cua Cloud Sandbox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "computer = Computer(\n",
    "    os_type=os_type,  # Must match the OS type of your Cloud Sandbox\n",
    "    api_key=cua_api_key,\n",
    "    name=container_name,\n",
    "    provider_type=VMProviderType.CLOUD,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a screenshot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "await computer.run()  # Initialize the computer first\n",
    "\n",
    "screenshot = await computer.interface.screenshot()\n",
    "\n",
    "with open(\"screenshot.png\", \"wb\") as f:\n",
    "    f.write(screenshot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 2: KASM Local Docker Containers (cross-platform)\n",
    "\n",
    "### Pull from Docker Hub\n",
    "```bash\n",
    "docker pull --platform=linux/amd64 trycua/cua-ubuntu:latest\n",
    "```\n",
    "\n",
    "### Or build locally\n",
    "```bash\n",
    "cd libs/kasm\n",
    "docker build -t cua-ubuntu:latest .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from computer import Computer\n",
    "\n",
    "computer = Computer(\n",
    "    os_type=\"linux\",\n",
    "    provider_type=\"docker\",\n",
    "    image=\"trycua/cua-ubuntu:latest\",\n",
    "    name=\"my-cua-container\",\n",
    ")\n",
    "\n",
    "await computer.run()  # Launch & connect to Docker container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Option 3: Local VMs (Lume daemon)\n",
    "\n",
    "For Apple Silicon Macs, you can run VMs locally using the Lume daemon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lume daemon setup\n",
    "\n",
    "Refer to [../libs/lume/README.md](../libs/lume/README.md) for more details on the lume cli."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For installing the standalone lume binary, run the following command from a terminal:\n",
    "\n",
    "```bash\n",
    "/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/lume/scripts/install.sh)\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pull the latest pre-built macos-sequoia-cua image. This image, based on macOS Sequoia, contains all dependencies needed to be controlled from the cua-computer interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!lume pull macos-sequoia-cua:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "VMs are stored in `~/.lume`, and locally cached images are stored in `~/.lume/cache`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can always see the list of downloaded VM images with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name                              os      cpu     memory  disk            display     status          storage         shared_dirs                                           ip              vnc                                               \n",
      "macos-sequoia-cua_latest          macOS   4       8.00G   25.9GB/80.0GB   1024x768    stopped         default         -                                                     -               -                                                 \n"
     ]
    }
   ],
   "source": [
    "!lume ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the sandbox\n",
    "\n",
    "Once downloaded, you can run the sandbox to test everything is working:\n",
    "\n",
    "```bash\n",
    "lume run macos-sequoia-cua:latest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add additional software and tools to the sandbox - these changes will be saved in the VM disk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize a Local Computer instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computer allows you to create and control a virtual sandbox instances from your host on Apple Silicon. Here's a basic example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from computer import Computer, VMProviderType"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can either use the async context manager or initialize the Computer instance directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the async context manager:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async with Computer(\n",
    "    # name=\"my_vm\", # optional, in case you want to use any other VM created using lume\n",
    "    display=\"1024x768\",\n",
    "    memory=\"8GB\",\n",
    "    cpu=\"4\",\n",
    "    os_type=\"macos\",\n",
    "    provider_type=VMProviderType.LUME,\n",
    ") as computer:\n",
    "    pass\n",
    "    # ... do something with the computer interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Direct initialization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "computer = Computer(display=\"1024x768\", memory=\"8GB\", cpu=\"4\", os_type=\"macos\")\n",
    "\n",
    "await computer.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computer Interface\n",
    "\n",
    "Both cloud and local computers provide the same interface for interaction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to execute commands targeting the sandbox, the Computer interface communicates through websockets to a Fast API server running on the sandbox."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cursor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Move and click\n",
    "await computer.interface.move_cursor(100, 100)\n",
    "await computer.interface.left_click()\n",
    "await computer.interface.right_click(300, 300)\n",
    "await computer.interface.double_click(400, 400)\n",
    "\n",
    "# Drag operations\n",
    "await computer.interface.drag_to(500, 500, duration=1.0)\n",
    "\n",
    "# Get cursor position\n",
    "cursor_pos = await computer.interface.get_cursor_position()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keyboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type text\n",
    "await computer.interface.type_text(\"Hello, World!\")\n",
    "\n",
    "# Press individual keys\n",
    "await computer.interface.press_key(\"enter\")\n",
    "await computer.interface.press_key(\"escape\")\n",
    "\n",
    "# Use hotkeys\n",
    "await computer.interface.hotkey(\"command\", \"c\")  # Command+C on macOS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Screen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get screen dimensions\n",
    "screen_size = await computer.interface.get_screen_size()\n",
    "\n",
    "# Take basic screenshot\n",
    "screenshot = await computer.interface.screenshot()\n",
    "with open(\"screenshot.png\", \"wb\") as f:\n",
    "    f.write(screenshot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clipboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set clipboard content\n",
    "await computer.interface.set_clipboard(\"Text to copy\")\n",
    "\n",
    "# Get clipboard content\n",
    "clipboard_content = await computer.interface.copy_to_clipboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File System Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check file/directory existence\n",
    "file_exists = await computer.interface.file_exists(\"/path/to/file.txt\")\n",
    "dir_exists = await computer.interface.directory_exists(\"/path/to/directory\")\n",
    "\n",
    "# Run shell commands\n",
    "result = await computer.interface.run_command(\"ls -la\")\n",
    "stdout, stderr, returncode = result.stdout, result.stderr, result.returncode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get accessibility tree\n",
    "accessibility_tree = await computer.interface.get_accessibility_tree()\n",
    "print(accessibility_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sharing a directory with the sandbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can share a directory with the sandbox by passing a list of absolute paths to the `shared_directories` argument when initializing the Computer instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "computer = Computer(\n",
    "    display=\"1024x768\",\n",
    "    memory=\"4GB\",\n",
    "    cpu=\"2\",\n",
    "    os_type=\"macos\",\n",
    "    shared_directories=[\"/absolute/path/to/directory\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Host Computer\n",
    "\n",
    "For local development and testing purposes, you can run the Computer API server on the same host machine and target it instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "computer = Computer(\n",
    "    display=\"1024x768\", memory=\"4GB\", cpu=\"2\", os_type=\"macos\", use_host_computer_server=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from computer.computer import Computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async with Computer(display=\"1024x768\", memory=\"4GB\", cpu=\"2\", os_type=\"macos\") as computer:\n",
    "    await computer.run()\n",
    "    res = await computer.interface.run_command(\"ls -a\")\n",
    "\n",
    "    # Get screen dimensions\n",
    "    screen_size = await computer.interface.get_screen_size()\n",
    "\n",
    "    # Move and click\n",
    "    await computer.interface.move_cursor(100, 100)\n",
    "    await computer.interface.left_click()\n",
    "    await computer.interface.right_click(300, 300)\n",
    "    await computer.interface.double_click(400, 400)\n",
    "\n",
    "    # Drag operations\n",
    "    await computer.interface.drag_to(500, 500, duration=1.0)\n",
    "\n",
    "    # Get cursor position\n",
    "    cursor_pos = await computer.interface.get_cursor_position()\n",
    "\n",
    "    # Your automation code here\n",
    "    await computer.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}