Skills Reference
Computer Use
What it does
Controls a connected desktop directly. It observes the screen through accessibility APIs and screenshots, then clicks, types, and scrolls. Some actions depend on the desktop operating system.
Setup required
None (built into the desktop app). The app will request the operating-system permissions it needs.
Permissions
- Accessibility (mouse/keyboard control)
- Screen capture (seeing screen content)
- Each action is prompted individually for approval
Common prompts
| You say... | What happens |
|---|---|
| “Open my browser and go to my bank's website” | Opens app and navigates |
| “Click the Submit button” | Clicks a specific UI element |
| “Fill out this form with my info” | Types into form fields |
| “Take a screenshot of what's on screen” | Captures current screen state |
| “Switch to Slack and check my DMs” | Navigates between apps |
| “Scroll down to the pricing section” | Scrolls within an app |
Configuration
- No configuration needed
- Step limit of 50 actions per session
- Each action requires approval unless you create trust rules via the Allow button
Tips & gotchas
- Accessibility tree + screenshots. The assistant reads the accessibility tree (same API screen readers use) on every step and takes a screenshot after each action, so it can see what the action did.
- Element-based clicking. It prefers clicking by element name rather than coordinates for reliability.
- Session caps. Sessions are capped at 50 steps with loop detection.
- Platform differences. Dragging, opening apps by name, and AppleScript are available on macOS. Windows exposes only actions its desktop helper supports, so unsupported tools are not offered to the assistant.
- Typing is not sending. Asked to type or draft a message, the assistant types it and stops before pressing Enter. It sends only when asked to.
- Your pointer comes back on macOS. The pointer stays where the assistant last clicked while it works, so controls that only show on hover keep working. It returns to where you left it when the task ends, unless you have already moved it.
- Batched actions on macOS. When the assistant already knows several steps, such as opening a window, typing a URL, and pressing Enter, it can run them in one step and look at the screen once afterward. It stops at the first action that fails. This needs a desktop app that supports batched actions.
- Single-window observations on macOS. The observe tool accepts
capture_window_id, a current native CGWindowID, not a browser tab or accessibility element ID. It captures only that window and its accessibility tree, even behind another app, without secondary windows or a desktop fallback. A compatible desktop app must explicitly advertise window-capture support; older or unsupported clients are rejected before capture. - Observation-only scope. Window selection applies to one observe call, not the whole session: later click, type, and scroll actions still return normal desktop observations. Selection does not focus the window or restrict later input to it. Cropped screenshot coordinates are window-relative, not full-display coordinates; prefer accessibility element IDs and focus the intended window before acting.
- Screen visibility. Be mindful of what's visible on screen. Screenshots are sent to the AI model.