computer-use – Agents & Automation skill for AI agents

Computer Use (universal, any-model, cross-platform)

You have a computer_use tool that drives the user's desktop in the

background — your actions do NOT move the user's cursor, steal

keyboard focus, or switch virtual desktops / Spaces. The user can keep

typing in their editor while you click around in a browser in another

window. This is the opposite of pyautogui-style automation.

Everything here works with any tool-capable model — Claude, GPT, Gemini,

or an open model on a local OpenAI-compatible endpoint. There is no

Anthropic-native schema to learn.

Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood.

This skill teaches the Hermes computer_use action vocabulary, which is

NOT the driver's raw MCP vocabulary. Call the actions documented below and

never the driver's tools by name: capture is a Hermes action that maps to

the driver's get_window_state; element=N is a Hermes argument that the

wrapper translates into the driver's element_token handle. If you see a

driver-side error mentioning snapshot_id, element_token, or "no reviewed

risk classification", you (or a stale description) called the raw driver

vocabulary — go back to the actions below.

The canonical workflow

Step 1 — Capture first. Almost every task starts with:

computer_use(action="capture", mode="som", app="<the app you're driving>")

Returns a screenshot plus an indexed element list like:

#1  AXButton 'Back' @ (12, 80, 28, 28) [Chrome]
#2  AXTextField 'Address bar' @ (80, 80, 900, 32) [Chrome]
#7  Link 'Sign In' @ (900, 420, 80, 24) [Chrome]
...

The #N index is the ONLY element handle you use. Behind it the wrapper

keeps this snapshot's opaque per-element token and sends it with every

element=N action, so a click on an index from a superseded snapshot is

refused explicitly (stale) instead of landing on the wrong control.

Re-capture after anything that changes the screen; indices do not survive it.

The role names match the host platform's accessibility framework

(AXButton on macOS, Button on Windows UIA, push button on Linux

AT-SPI) — treat them as labels, not as strict types.

Step 2 — Click by element index. This is the single most important

habit:

computer_use(action="click", element=7)

Much more reliable than pixel coordinates for every model. Claude was

trained on both; other models are often only reliable with indices.

Step 3 — Verify. After any state-changing action, re-capture. You

can save a round-trip by asking for the post-action capture inline:

computer_use(action="click", element=7, capture_after=True)

Capture modes

| mode | Returns | Best for |

|---|---|---|

| som (default) | Screenshot + indexed element list | Vision models; preferred default |

| vision | Plain screenshot, no elements | When you only need pixels (then click by coordinate=) |

| ax | Element list only, no image | Text-only models, or when you don't need to see pixels |

Current drivers always return the screenshot AND the tree in one call;

mode decides what Hermes hands back to you, not what the driver does.

There is no numbered overlay burned into the screenshot — the index list is

the map; ground on both and cross-check (the tree lies on some surfaces).

No vision model? If your main model can't read images (or the provider

rejects image tool results), Hermes routes the screenshot through the

auxiliary vision model and you get a text description instead of pixels.

Configure auxiliary.vision in config.yaml to pick that model, or use

mode="ax" and drive by element index without a screenshot at all.

Actions

capture           mode=som|vision|ax   app=…  (default: current app)
click             element=N     OR     coordinate=[x, y]    button=left|right|middle
double_click      element=N     OR     coordinate=[x, y]
right_click       element=N     OR     coordinate=[x, y]
middle_click      element=N     OR     coordinate=[x, y]
drag              from_element=N, to_element=M        (or from/to_coordinate)
scroll            direction=up|down|left|right   amount=3 (ticks)
type              text="…"
key               keys="<save shortcut>" | "return" | "escape" | "<modifier>+t"
set_value         element=N  value="…"     (selects/sliders without opening the menu)
wait              seconds=0.5
list_apps
list_windows
focus_app         app="<app name>"   raise_window=false   (default: don't raise)

All actions accept optional capture_after=True to get a follow-up

screenshot in the same tool call. All actions that target an element

accept modifiers=[…] for held keys.

The input actions (click, double_click, right_click, middle_click,

drag, scroll, type, key) also accept delivery_mode. The optional

bring_to_front=True request invokes a separately approved standalone focus

tool before foreground input; it is never an input-action property.

The verify → escalate ladder (background-first)

cua-driver delivers input in the background by default (no focus steal),

but that is the first rung, not the only one. Every input action returns a

structured verdict; read it and climb only when the driver tells you to.

Returned fields (present when the driver supports them):

• effect: "confirmed" (driver read the result back — done), "unverifiable"

(delivered, but confirm it yourself by re-capturing), or "suspected_noop"

(ran but almost certainly did nothing).

• escalation: {recommended: "px" | "foreground", reason} — present

only when there's a next rung to try.

• code: a structured refusal like "background_unavailable",

"foreground_unsupported", or "stale" (re-capture, then retry by index).

• verified: true only on AX read-back.

Walk it in order:

1. Element, background (default). click(element=N). If effect:"confirmed",

you're done.

2. Fresh verification. effect:"unverifiable" means inspect a fresh

capture/state before any retry. Do this even when escalation.recommended

is present; it is advisory, not proof that successful input should repeat.

3. Pixel, background. After effect:"suspected_noop" or a structured

refusal recommends "px" (or a degraded capture has no elements), click

by coordinate=[x,y] instead of element.

4. Foreground. After effect:"suspected_noop",

code:"background_unavailable", or a verified pixel no-op,

re-issue the SAME action with delivery_mode="foreground". This briefly

raises the window and restores focus after; pair with bring_to_front=True

for a short sequence to avoid per-call flashes. It needs its own approval

(it's a visible focus change) and is only appropriate when the user isn't

actively working. Classic cases: Electron/Chromium consent dialogs (e.g.

tldraw offline's "Run Script"), DirectInput games, raw-input canvases.

5. Keystrokes verified-lost on a KDE/Qt editor → use the app's own I/O.

Some Qt text components (KTextEditor: Kate, KWrite, KDevelop) discard

SYNTHETIC X keystrokes entirely — foreground type reports ok

("Typed N characters into the focused widget", effect:"unverifiable")

but a fresh AX capture shows the text never arrived, and raw XTest fails

identically (proven live, Aug 2026 — it is the toolkit, not the driver;

the same foreground route works on kcalc/Chrome). After ONE such

verified-lost round trip, stop retrying input rungs: write the file with

terminal/file tools and let the editor reload it, or drive the app's

DBus/CLI interface. Never loop the ladder against a surface that

verifiably swallows synthetic input.

computer_use(action="click", element=7)
# → {effect: "suspected_noop", escalation: {recommended: "foreground", ...}}
computer_use(action="click", element=7, delivery_mode="foreground")
# → {effect: "unverifiable", path: "x11_pixel_fg"}   then re-capture to confirm

**Escalate to foreground as a REACTION to a returned signal, never as a

prediction** from the app being Electron/Chromium/GTK. A confirmed effect is

done and must not be duplicated. Different controls in

the same app behave differently. Do NOT silently retry the same rung, and do

NOT conclude "cua-driver can't drive this app" — climb the ladder. If

delivery_mode="foreground" returns code:"foreground_unsupported", the live

action schema lacks that property; choose another verified rung without

inferring support from the executable's reported version.

Page content is a separate toolset

computer_use is desktop-only: it does not expose a typed route for browser

page content (no cua_browser_* actions). For reading or acting on a page's

DOM — navigation, clicking a link by text, typed input into a form field —

use the separate browser_exec tool (Python CDP helpers driving a headless

Chromium); its own schema documents the current contract. Reserve computer_use for browser

*chrome* (the address bar, permission prompts, extension popups, nativ

… (full content is in the download package)