mirror of
https://github.com/trycua/computer.git
synced 2026-01-02 03:20:22 -06:00
484 lines
11 KiB
Plaintext
484 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Computer\n",
|
|
"\n",
|
|
"This notebook demonstrates how to use Computer to operate a Lume sandbox VMs programmatically on Apple Silicon macOS systems."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Installation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip uninstall -y cua-computer"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!pip install \"cua-computer[all]\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# If locally installed, use this instead:\n",
|
|
"import os\n",
|
|
"\n",
|
|
"os.chdir('../libs/computer')\n",
|
|
"!poetry install\n",
|
|
"!poetry build\n",
|
|
"\n",
|
|
"!pip uninstall cua-computer -y\n",
|
|
"!pip install ./dist/cua_computer-0.1.0-py3-none-any.whl --force-reinstall"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Lume daemon\n",
|
|
"\n",
|
|
"Refer to [../libs/lume/README.md](../libs/lume/README.md) for more details on the lume cli."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"For installing the standalone lume binary, run the following command from a terminal:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/lume/scripts/install.sh)\"\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Pull the latest pre-built macos-sequoia-cua image. This image, based on macOS Sequoia, contains all dependencies needed to be controlled from the cua-computer interface."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"!lume pull macos-sequoia-cua:latest"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"VMs are stored in `~/.lume`, and locally cached images are stored in `~/.lume/cache`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can always see the list of downloaded VM images with:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"name os cpu memory disk display status storage shared_dirs ip vnc \n",
|
|
"macos-sequoia-cua_latest macOS 4 8.00G 25.9GB/80.0GB 1024x768 stopped default - - - \n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"!lume ls"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Testing the sandbox\n",
|
|
"\n",
|
|
"Once downloaded, you can run the sandbox to test everything is working:\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"lume run macos-sequoia-cua:latest\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can add additional software and tools to the sandbox - these changes will be saved in the VM disk."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Initialize a Computer instance"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Computer allows you to create and control a virtual sandbox instances from your host on Apple Silicon. Here's a basic example:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from computer import Computer, VMProviderType"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can either use the async context manager or initialize the Computer instance directly."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"With the async context manager:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async with Computer(\n",
|
|
" # name=\"my_vm\", # optional, in case you want to use any other VM created using lume\n",
|
|
" display=\"1024x768\",\n",
|
|
" memory=\"8GB\",\n",
|
|
" cpu=\"4\",\n",
|
|
" os_type=\"macos\",\n",
|
|
" provider_type=VMProviderType.LUME,\n",
|
|
") as computer:\n",
|
|
" await computer.run()\n",
|
|
" # ... do something with the computer interface"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Direct initialization:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"computer = Computer(\n",
|
|
" display=\"1024x768\",\n",
|
|
" memory=\"8GB\",\n",
|
|
" cpu=\"4\",\n",
|
|
" os_type=\"macos\"\n",
|
|
")\n",
|
|
"\n",
|
|
"await computer.run()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"In order to execute commands targeting the sandbox, the Computer interface communicates through websockets to a Fast API server running on the sandbox."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Cursor"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Move and click\n",
|
|
"await computer.interface.move_cursor(100, 100)\n",
|
|
"await computer.interface.left_click()\n",
|
|
"await computer.interface.right_click(300, 300)\n",
|
|
"await computer.interface.double_click(400, 400)\n",
|
|
"\n",
|
|
"# Drag operations\n",
|
|
"await computer.interface.drag_to(500, 500, duration=1.0)\n",
|
|
"\n",
|
|
"# Get cursor position\n",
|
|
"cursor_pos = await computer.interface.get_cursor_position()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Keyboard"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Type text\n",
|
|
"await computer.interface.type_text(\"Hello, World!\")\n",
|
|
"\n",
|
|
"# Press individual keys\n",
|
|
"await computer.interface.press_key(\"enter\")\n",
|
|
"await computer.interface.press_key(\"escape\")\n",
|
|
"\n",
|
|
"# Use hotkeys\n",
|
|
"await computer.interface.hotkey(\"command\", \"c\") # Command+C on macOS"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Screen"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Get screen dimensions\n",
|
|
"screen_size = await computer.interface.get_screen_size()\n",
|
|
"\n",
|
|
"# Take basic screenshot\n",
|
|
"screenshot = await computer.interface.screenshot()\n",
|
|
"with open(\"screenshot.png\", \"wb\") as f:\n",
|
|
" f.write(screenshot)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Clipboard"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Set clipboard content\n",
|
|
"await computer.interface.set_clipboard(\"Text to copy\")\n",
|
|
"\n",
|
|
"# Get clipboard content\n",
|
|
"clipboard_content = await computer.interface.copy_to_clipboard()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### File System Operations"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Check file/directory existence\n",
|
|
"file_exists = await computer.interface.file_exists(\"/path/to/file.txt\")\n",
|
|
"dir_exists = await computer.interface.directory_exists(\"/path/to/directory\")\n",
|
|
"\n",
|
|
"# Run shell commands\n",
|
|
"stdout, stderr = await computer.interface.run_command(\"ls -la\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Accessibility"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Get accessibility tree\n",
|
|
"accessibility_tree = await computer.interface.get_accessibility_tree()\n",
|
|
"print(accessibility_tree)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Sharing a directory with the sandbox"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can share a directory with the sandbox by passing a list of absolute paths to the `shared_directories` argument when initializing the Computer instance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"computer = Computer(\n",
|
|
" display=\"1024x768\",\n",
|
|
" memory=\"4GB\",\n",
|
|
" cpu=\"2\",\n",
|
|
" os_type=\"macos\",\n",
|
|
" shared_directories=[\"/absolute/path/to/directory\"]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Using Host Computer\n",
|
|
"\n",
|
|
"For local development and testing purposes, you can run the Computer API server on the same host machine and target it instead:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"computer = Computer(\n",
|
|
" display=\"1024x768\",\n",
|
|
" memory=\"4GB\",\n",
|
|
" cpu=\"2\",\n",
|
|
" os_type=\"macos\",\n",
|
|
" use_host_computer_server=True\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Examples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from computer.computer import Computer"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"async with Computer(\n",
|
|
" display=\"1024x768\",\n",
|
|
" memory=\"4GB\",\n",
|
|
" cpu=\"2\",\n",
|
|
" os_type=\"macos\"\n",
|
|
") as computer:\n",
|
|
" await computer.run()\n",
|
|
" res = await computer.interface.run_command(\"ls -a\")\n",
|
|
"\n",
|
|
" # Get screen dimensions\n",
|
|
" screen_size = await computer.interface.get_screen_size()\n",
|
|
"\n",
|
|
" # Move and click\n",
|
|
" await computer.interface.move_cursor(100, 100)\n",
|
|
" await computer.interface.left_click()\n",
|
|
" await computer.interface.right_click(300, 300)\n",
|
|
" await computer.interface.double_click(400, 400)\n",
|
|
"\n",
|
|
" # Drag operations\n",
|
|
" await computer.interface.drag_to(500, 500, duration=1.0)\n",
|
|
"\n",
|
|
" # Get cursor position\n",
|
|
" cursor_pos = await computer.interface.get_cursor_position()\n",
|
|
"\n",
|
|
" # Your automation code here\n",
|
|
" await computer.stop()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "cua312",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|