computer/notebooks/computer_nb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computer\n",
    "\n",
    "This notebook demonstrates how to use Computer to operate a Lume sandbox VMs programmatically on Apple Silicon macOS systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip uninstall -y cua-computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install cua-computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If locally installed, use this instead:\n",
    "import os\n",
    "\n",
    "os.chdir('../libs/computer')\n",
    "!poetry install\n",
    "!poetry build\n",
    "\n",
    "!pip uninstall cua-computer -y\n",
    "!pip install ./dist/cua_computer-0.1.0-py3-none-any.whl --force-reinstall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lume daemon\n",
    "\n",
    "While a `lume` binary is included with Computer, we recommend installing the standalone version with brew, and starting the lume daemon service before running Computer. Refer to [../libs/lume/README.md](../libs/lume/README.md) for more details on lume cli."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For installing the standalone lume binary, run the following command from a terminal:\n",
    "\n",
    "```bash\n",
    "/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/lume/scripts/install.sh)\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start the lume daemon service (from another terminal):\n",
    "\n",
    "```bash\n",
    "lume serve\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pull the latest pre-built macos-sequoia-cua image. This image, based on macOS Sequoia, contains all dependencies needed to be controlled from the cua-computer interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!lume pull macos-sequoia-cua:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initial download requires 80GB storage, but reduces to ~30GB after first run due to macOS's sparse file system.\n",
    "\n",
    "VMs are stored in `~/.lume`, and locally cached images are stored in `~/.lume/cache`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can always see the list of downloaded VM images with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name                              os      cpu     memory  disk            display     status          ip              vnc                                               \n",
      "macos-sequoia-cua_latest          macOS   12      16.00G  64.5GB/80.0GB   1024x768    running         192.168.64.78   vnc://:kind-forest-zulu-island@127.0.0.1:56085    \n"
     ]
    }
   ],
   "source": [
    "!lume ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the sandbox\n",
    "\n",
    "Once downloaded, you can run the sandbox to test everything is working:\n",
    "\n",
    "```bash\n",
    "lume run macos-sequoia-cua:latest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add additional software and tools to the sandbox - these changes will be saved in the VM disk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize a Computer instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computer allows you to create and control a virtual sandbox instances from your host on Apple Silicon. Here's a basic example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from computer import Computer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can either use the async context manager or initialize the Computer instance directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the async context manager:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async with Computer(\n",
    "    # name=\"my_vm\", # optional, in case you want to use any other VM created using lume\n",
    "    display=\"1024x768\",\n",
    "    memory=\"8GB\",\n",
    "    cpu=\"4\",\n",
    "    os=\"macos\"\n",
    ") as computer:\n",
    "    await computer.run()\n",
    "    # ... do something with the computer interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Direct initialization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Equivalent curl command:\n",
      "curl -X POST \\\n",
      "  'http://localhost:3000/lume/vms/macos-sequoia-cua_latest/run' \\\n",
      "  -H 'Content-Type: application/json' \\\n",
      "  -d '{\"noDisplay\": false, \"sharedDirectories\": []}'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "computer = Computer(\n",
    "    display=\"1024x768\",\n",
    "    memory=\"8GB\",\n",
    "    cpu=\"4\",\n",
    "    os=\"macos\"\n",
    ")\n",
    "\n",
    "await computer.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Computer instance requires a Lume server for communication. Here's how it works:\n",
    "\n",
    "1. First, it attempts to connect to any existing Lume server running on port 3000\n",
    "2. If no Lume server is available, it automatically starts a new one via [lume serve](https://github.com/trycua/lume/?tab=readme-ov-file#local-api-server)\n",
    "\n",
    "The sandbox's lifecycle is tied to the Lume server:\n",
    "- If Computer started the Lume server itself, the server will be terminated when Computer stops\n",
    "- If Computer connected to a pre-existing server, that server remains running after Computer stops\n",
    "\n",
    "If you have scenarios where you need to run multiple sandboxes in parallel, we recommend first starting the Lume server separately with `lume serve` (refer to [Lume](https://github.com/trycua/lume/?tab=readme-ov-file#install) on how to install) and then having each Computer instance connect to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to execute commands targeting the sandbox, the Computer interface communicates through websockets to a Fast API server running on the sandbox."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cursor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Move and click\n",
    "await computer.interface.move_cursor(100, 100)\n",
    "await computer.interface.left_click()\n",
    "await computer.interface.right_click(300, 300)\n",
    "await computer.interface.double_click(400, 400)\n",
    "\n",
    "# Drag operations\n",
    "await computer.interface.drag_to(500, 500, duration=1.0)\n",
    "\n",
    "# Get cursor position\n",
    "cursor_pos = await computer.interface.get_cursor_position()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keyboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type text\n",
    "await computer.interface.type_text(\"Hello, World!\")\n",
    "\n",
    "# Press individual keys\n",
    "await computer.interface.press_key(\"enter\")\n",
    "await computer.interface.press_key(\"escape\")\n",
    "\n",
    "# Use hotkeys\n",
    "await computer.interface.hotkey(\"command\", \"c\")  # Command+C on macOS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Screen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get screen dimensions\n",
    "screen_size = await computer.interface.get_screen_size()\n",
    "\n",
    "# Take basic screenshot\n",
    "screenshot = await computer.interface.screenshot()\n",
    "with open(\"screenshot.png\", \"wb\") as f:\n",
    "    f.write(screenshot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clipboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set clipboard content\n",
    "await computer.interface.set_clipboard(\"Text to copy\")\n",
    "\n",
    "# Get clipboard content\n",
    "clipboard_content = await computer.interface.copy_to_clipboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File System Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check file/directory existence\n",
    "file_exists = await computer.interface.file_exists(\"/path/to/file.txt\")\n",
    "dir_exists = await computer.interface.directory_exists(\"/path/to/directory\")\n",
    "\n",
    "# Run shell commands\n",
    "stdout, stderr = await computer.interface.run_command(\"ls -la\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get accessibility tree\n",
    "accessibility_tree = await computer.interface.get_accessibility_tree()\n",
    "print(accessibility_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sharing a directory with the sandbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can share a directory with the sandbox by passing a list of absolute paths to the `shared_directories` argument when initializing the Computer instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "computer = Computer(\n",
    "    display=\"1024x768\",\n",
    "    memory=\"4GB\",\n",
    "    cpu=\"2\",\n",
    "    os=\"macos\",\n",
    "    shared_directories=[\"/absolute/path/to/directory\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Host Computer\n",
    "\n",
    "For local development and testing purposes, you can run the Computer API server on the same host machine and target it instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "computer = Computer(\n",
    "    display=\"1024x768\",\n",
    "    memory=\"4GB\",\n",
    "    cpu=\"2\",\n",
    "    os=\"macos\",\n",
    "    use_host_computer_server=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from computer.computer import Computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async with Computer(\n",
    "    display=\"1024x768\",\n",
    "    memory=\"4GB\",\n",
    "    cpu=\"2\",\n",
    "    os=\"macos\"\n",
    ") as computer:\n",
    "    await computer.run()\n",
    "    res = await computer.interface.run_command(\"ls -a\")\n",
    "\n",
    "    # Get screen dimensions\n",
    "    screen_size = await computer.interface.get_screen_size()\n",
    "\n",
    "    # Move and click\n",
    "    await computer.interface.move_cursor(100, 100)\n",
    "    await computer.interface.left_click()\n",
    "    await computer.interface.right_click(300, 300)\n",
    "    await computer.interface.double_click(400, 400)\n",
    "\n",
    "    # Drag operations\n",
    "    await computer.interface.drag_to(500, 500, duration=1.0)\n",
    "\n",
    "    # Get cursor position\n",
    "    cursor_pos = await computer.interface.get_cursor_position()\n",
    "\n",
    "    # Your automation code here\n",
    "    await computer.stop()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cua",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}