Giving an AI agent eyes and hands on your desktop

OpenDesk is a computer use framework. It lets an AI agent see the screen, move the mouse, type, scroll, and read what came back, the same way a person would. One machine or a fleet. Mac, Linux, or Windows.

Why this matters

Most enterprise software has no useful API. The browser tool the team uses every day is closed. The accounting system is locked. The internal portal would take six months of paperwork to integrate with. So the team clicks through screens by hand, and there is no way to automate any of it.

OpenDesk takes the position that the screen itself is the API. If a person can do the task, an agent can do it too. That changes what kinds of work can be automated.

What the agent gets

• Screenshots. The agent sees the screen the way you do.
• Mouse and keyboard. It clicks, types, scrolls, and drags exactly as a person would.
• OCR. When the text is in an image, the agent can still read it.
• UI interaction. Accessibility APIs give the agent a structured view of buttons, fields, and menus, so it does not have to guess pixel coordinates.
• Workflow recording. Record a flow once and replay it. Mix recorded steps with on-the-fly decisions.
• Scheduling. Run a flow on a cron, on demand, or when something on the screen changes.

One machine or many

The same SDK that drives the local desktop drives a remote one. Each remote machine runs a small agent that exposes its screen and inputs over an encrypted WebSocket. The team can stand up a fleet, point one controller at all of them, and run the same workflow in parallel.

That is what turns OpenDesk from a useful local tool into something that can replace a room full of operators. The pattern is one agent, twenty machines, all doing the same work.

Where teams use it

Three patterns come up again and again.

Bulk operations on closed tools. A team needs to update a thousand records inside a vendor portal. The portal has no API. OpenDesk drives the portal the way the team would, one record at a time, across as many machines as needed.

End-to-end QA on real apps. No selectors. No CSS paths. The agent walks the app the way a user would, takes a screenshot at each step, and flags anything that looks off. When the UI changes, the agent adapts. The tests do not break.

Workflow automation across legacy systems. The classic RPA use case, without the fragility. Because the agent reads the screen with a real model behind it, small layout changes do not knock it over.

How it stays safe

The team picks what the agent is allowed to do. Read-only by default. Granular permissions on bash, file writes, network access. The full session is recorded, including every screenshot the agent took and every input it sent. If something looks wrong six weeks later, the team can replay the run and see exactly what happened.