Native helper (opendesk-ax)

Native helper (opendesk-ax)#

On macOS, opendesk reads the accessibility tree and delivers background input through a small Swift program, opendesk-ax, instead of running an AppleScript per query. This page explains what it does and how both SDKs use it.

Why#

The AppleScript path (osascript + System Events entire contents) pays for a new process and the AppleScript runtime on every call, then makes several cross-process round trips per element. It is slow, has to be capped (200 elements), and sees almost nothing in Chromium/Electron apps, which only build their accessibility tree when an assistive client asks for it.

The helper:

  • stays running — opendesk-ax serve answers JSON-line requests over stdin/stdout, so there is no start-up cost per query;

  • batches attribute reads — role, subrole, title, value, description, identifier, position, size, enabled, focused and children come back from one AXUIElementCopyMultipleAttributeValues call per element;

  • wakes Chromium/Electron — sets AXManualAccessibility and AXEnhancedUserInterface on Chromium-based apps (detected by bundle id or an Electron/Chromium framework in the bundle) and retries for up to ~450 ms while the tree is built;

  • hands out stable element refs so an agent can act on the exact element it saw;

  • delivers input in the background with CGEvent.postToPid, without moving the user’s cursor or activating the target app.

get_tree, median of 5 runs, macOS 26 on Apple silicon (python python/scripts/bench_ax.py --open):

App

AppleScript

opendesk-ax

Speed-up

Elements (AppleScript / helper)

Finder (Applications window)

7,649 ms

75 ms

~100x

28 / 175

TextEdit (blank document)

2,715 ms

48 ms

~55x

3 / 45

Google Chrome (one tab)

—

120–190 ms

—

— / 599

Build and lifecycle#

The Swift source is embedded in both SDKs — python/opendesk/computer/macos_native.py (SWIFT_SOURCE) and js/src/computer/macosNative.ts (SWIFT_SOURCE). The two copies are byte-identical, and a test in each SDK enforces it.

On first use the source is compiled with swiftc -O to <home>/bin/opendesk-ax-v1 (<home> is $OPENDESK_HOME or ~/.opendesk) and a opendesk-ax-v1.sha256 stamp of the source is written next to it. Compilation takes ~10–15 s once; whichever SDK runs first builds the binary and the other reuses it. A changed source rebuilds automatically, and the binary is written to a temp file and renamed into place, so concurrent processes never see a partial file.

Each SDK keeps one helper process per host process:

  • requests are serialised and matched to responses by id;

  • every request has a timeout (10 s, 30 s for trees); a timed-out helper is killed and the next request starts a fresh one;

  • a crashed helper is restarted on the next request;

  • the helper exits when its stdin closes, so it never outlives its parent. The JS client unrefs the process while idle so it never keeps Node alive.

If the helper can’t run — not macOS, no swiftc (install the Xcode Command Line Tools), or no Accessibility permission — available() is false, the ui tool falls back to AppleScript, and window_text / windowText return None / null. Set OPENDESK_NATIVE_AX=0 to disable it entirely.

The helper inherits Accessibility permission from the app that runs opendesk (Terminal, your IDE, Claude Desktop): grant it there under System Settings → Privacy & Security → Accessibility.

Protocol#

→ {"id": 7, "cmd": "tree", "app": "TextEdit", "max_nodes": 500}
← {"id": 7, "ok": true, "result": {"pid": 64076, "app": "TextEdit", "window_title": "Untitled",
                                   "truncated": false, "nodes": [...]}}
← {"id": 8, "ok": false, "error": "unknown ref zzzzzz; call tree again"}

Command

Arguments

Result

ping

—

{version, trusted, pid}

frontmost

—

{app, pid, bundle_id, window_title}

tree

pid or app (default frontmost), max_depth (12), max_nodes (1500), max_value (300), scope (window | app), actions (true)

{pid, app, window_title, truncated, nodes}

get

ref, max_value

one node, freshly read

perform

ref, action (default AXPress)

{ref, action, pid}

set_value

ref, value

{ref, pid}

focus

ref

{ref, pid}

element_at

x, y (logical screen points)

node + pid, app, pressable (nearest element with AXPress)

window_text

pid (default frontmost), max_chars (20000)

{text, app, pid, window_title, nodes}

click_pid

x, y, pid or app (default: owner of the window under the point), button, count

{pid, window, ...}

key_pid

pid, app or ref; text, or key / keycode + modifiers

{pid, events}

Nodes are {ref, parent, role, subrole?, title?, value?, description?, identifier?, frame?{x,y,w,h}, enabled, focused, actions?} in breadth-first order, so one deep subtree can’t starve the rest of the window. max_depth counts meaningful levels only: unlabelled AXGroup, AXScrollArea, AXSplitGroup and AXLayoutArea wrappers (common in web content) don’t add depth.

opendesk-ax call '<json>' runs a single request and prints the response, which is handy for debugging.

Element refs#

A ref is a 6-character base-36 hash of the element’s pid and its path from the window: one (role, index among same-role siblings, identifier-or-title) step per level. The window step deliberately omits the window title, which changes as documents are edited. So:

  • the same UI produces the same refs across calls and across helper restarts;

  • a ref survives unrelated changes elsewhere in the window;

  • renaming or moving the element (or an ancestor) changes it.

The helper keeps ref → element for the latest tree of each app. When a stored element has gone stale it re-resolves the ref by walking its path again; an unknown ref (for example after a helper restart) triggers a fresh tree read of the known apps and the frontmost app before giving up.

On the Python wire protocol UIElement.ref travels inside metadata["ref"], so peers that predate the field (they reject unknown keys) keep working in both directions.

Background input#

  • perform / set_value / focus act through accessibility and never touch the cursor.

  • key_pid posts keyboard events to one process (CGEventPostToPid) using a private event source, character by character with the Unicode string attached, so any text works without the clipboard. With ref, the element is focused first.

  • click_pid posts mouse-down/up to one process with the target window’s number attached. Apps that are inactive or on another Space may ignore it; the mouse tool therefore tries element_at + AXPress first.

OPENDESK_BACKGROUND=1 makes the ui and mouse tools use only these paths and refuse anything that would move the cursor.

window_text#

window_text(pid=None, max_chars=20000) (Python) / windowText(pid?, maxChars = 20000) (JS) returns the visible text of the frontmost (or given) window in accessibility reading order: titles, static text, and text-field / text-area / heading values, deduplicated, one item per line. Menus are skipped, tables and lists use their visible rows, and elements outside the window frame are ignored. Screen memory uses it as a cheap, exact alternative to OCR.

Other platforms#

  • Windows: LocalComputer.ui_tree fetches the whole window subtree with one UI Automation CacheRequest (TreeScope_Subtree) when pywinauto’s UIA backend is available, instead of several round trips per control, and falls back to the per-control walk otherwise.

  • Linux Wayland: capture via grim, the xdg-desktop-portal Screenshot API (gdbus), or gnome-screenshot; input via ydotool. See mouse → Wayland.