opendesk

Open Computer Use Agent — gives any AI agent eyes and hands on your desktop. Works with any MCP client (Claude Code, Cursor, Windsurf, Continue) and any AI framework (Anthropic SDK, OpenAI, LangChain). Runs on macOS, Linux, and Windows.

Getting Started

• Getting Started
  • Try it immediately via Claude
• Python Quickstart
  • Install
  • Try it via Claude
  • SDK examples
  • 2. Click a button by name
  • 3. Type text
  • 4. Full agentic loop
  • 5. Permission modes
  • 6. Restrict the sandbox
  • Next steps
• JavaScript / TypeScript Quickstart
  • Install
  • Try it via Claude
  • SDK examples
  • 2. Click a button by name
  • 3. Type text
  • 4. Full agentic loop (Vercel AI SDK)
  • 5. Native MCP server (Node.js)
  • 6. Custom session or permission handler
  • Next steps

MCP

• MCP Integration
  • How it works
  • What the LLM sees
  • Ask Claude
  • Setup
• Install
  • Python
  • JavaScript / TypeScript
• Claude Code
  • Python
  • JavaScript / TypeScript
• Claude Desktop
  • Python
  • JavaScript / TypeScript
• Cursor
  • Python
  • JavaScript / TypeScript
• Continue (VS Code)
  • Python
  • JavaScript / TypeScript
  • Any other MCP client
• Advanced: In-process Server & Restrictions
  • Running the MCP server in Python
  • Restricting what the agent can do
  • Audit log
• Troubleshooting

Integrations

• Integrations
• Anthropic SDK
  • Agentic loop (recommended)
  • Manual control
  • Tool definitions format
• OpenAI Function Calling
  • Agentic loop
  • With Groq
  • With Ollama
  • Manual control
• LangChain / LangGraph
  • With custom context
• JavaScript / TypeScript SDK
  • Install
  • Claude Code / Claude Desktop
  • Programmatic usage
  • Native MCP server
  • With Vercel AI SDK
  • With LangChain.js
  • Custom session or permission handler
• Custom / Generic Integration
  • Sandbox configuration

Tools

• Tools Reference
  • ToolResult
• ui — Accessibility-based UI Interaction
  • Actions
  • Ask Claude
  • SDK examples
  • Platform notes
• screenshot — Screen Capture
  • Parameters
  • Ask Claude
  • SDK examples
  • Output
• mouse — Mouse Control
  • Actions
  • Ask Claude
  • SDK examples
• keyboard — Keyboard Input
  • Actions
  • Ask Claude
  • SDK examples
• app — Application Control
  • Actions
  • Ask Claude
  • SDK examples
• clipboard — Clipboard Read/Write
  • Actions
  • Ask Claude
  • SDK examples
• ocr — Text Extraction
  • Parameters
  • Ask Claude
  • SDK examples
• audit — Session Audit Log & Replay
  • Ask Claude
  • Parameters
  • SDK examples
  • What gets replayed
• learn — Record and Replay Tasks
  • Actions
  • Ask Claude
  • SDK examples

Automation

• Automation — Learn, Replay & Schedule
  • How it works
  • Ask Claude
  • Guides
• Automation in Claude Code
  • Record
  • Replay
  • Schedule
  • Manage
• Python API
  • Learn & Replay
  • Schedule
• Scheduler
  • Ask Claude
  • Start the background runner
  • Timing formats
  • Installation
  • Storage
  • Platform support

Architecture

• Architecture
  • Layer guide
• Computer & Tools Layers
  • Layer 1: computer/
  • Layer 2: tools/
• Protocol Layer
  • Layer 3: protocol/
• Remote & Integrations Layers
  • Layer 4: remote/
  • Layer 5: integrations/
• Data Flow
  • Local call
  • Remote call
• Adding a Custom Tool

Protocol

• Protocol Reference
• Transport & Encoding
  • 1. Transport
  • 2. Encoding
• Encryption & Identity
  • 3. Encryption
  • 4. Identity and key material
• Pairing Handshake
  • 5. Pairing handshake (first contact)
• Auth Handshake
  • 6. Auth handshake (reconnect)
• Frames & Errors
  • 7. Frame types
  • 8. Error codes
• Session Lifecycle & Method Namespace
  • 9. Session lifecycle
  • 10. Method namespace
• Discovery, Admin IPC & Cryptographic Summary
  • 11. mDNS discovery
  • 12. Capability manifest
  • 13. Admin IPC
  • 14. Cryptographic summary

Remote — Python

• Remote Computer Use — Python / CLI
  • Mental model
  • Guides
• One-time Setup
  • On the controlled machine
  • On the controller
• Running
  • Controlled machine
  • Ask Claude (on the controller)
  • Controller
• MCP Integration
  • Ask Claude
  • Default peer resolution
  • Agent example
  • Why pairing isn’t an MCP tool
• Trust & Security
  • What the threat model covers
  • What’s not in v1
  • Pairing-code strength
  • Files written to disk
• CLI Reference
  • Controlled machine
  • Controller
• Service Install
• Concurrency: One Controller at a Time
  • Disconnect vs unpair — the trust model
• Programmatic Use
  • Python
  • JavaScript / TypeScript
• Troubleshooting

Remote — JS/TS

• Remote Computer Use — JavaScript / TypeScript SDK
  • Mental model
  • Guides
• One-time Setup
  • On the controlled machine
  • On the controller
• Running
  • Controlled machine
  • Controller
• MCP Integration
  • Ask Claude
  • Default peer resolution
  • Agent example
  • Why pairing isn’t an MCP tool
• Audit Log
  • CLI viewer
  • Programmatic access
• Trust & Security
  • What the threat model covers
  • What’s not in v1
  • Pairing-code strength
  • Files written to disk
• CLI Reference
  • Controlled machine
  • Controller
  • Shared
• Concurrency: One Controller at a Time
  • disconnect vs unpair
• Programmatic Use
  • Connect to a paired peer
  • Connect via DiscoveredPeer or explicit URL
  • Pair programmatically
  • Run the server daemon from Node.js
  • Identity and peer storage
• Troubleshooting