Tools
Computer use and the screen
Opening and arranging apps, pressing buttons by their names, and working on the screen by sight only when nothing more direct can reach.
The screen comes last
When there is a direct way, Mori takes it: a tool, a connected app, a command or the app's own scripting. This family is for what only happens on screen, a dialog, a panel, an app with no scripting, or when you ask about the screen itself. The full picture, with the bar that shows what she is doing and the app tiers, is on Computer use and the browser.
By sight, when names run out
use_computer runs the vision loop on one small part of a task: look, act, check, step by step, in the app you name, then hand back what it did and the last screenshot. It needs a model that reads pictures. The single mouse and keyboard tools, clicks, drags, scrolls, keys and chords, are its hands when Mori steers one step at a time. Coordinates match the screenshots one to one.
take_screenshot is the only way to show you a picture of the screen; a file written by a command shows you nothing. A page she built is checked in the Preview window instead, never on your desktop.
Apps and windows
Mori lists running apps, opens one by its everyday name, focuses it, and quits it. Quitting is for a whole app; a browser tab is closed with the tab tools, because quitting Chrome takes all your other tabs with it. She lists every window, reads and sets its size and place, minimises, maximises and tiles them, and knows which window you were in before you switched to her.
Which apps she may press
Every click and keystroke is checked against the app it lands in:
- Look only: banking and trading apps, browsers driven through the screen, and Mori's own window.
- Click only: terminals and code editors.
- Full control: everything else.
No setting changes a tier. With asking on, Computer-use autonomy in Settings > Agent decides whether she asks the first time she needs an app (Ask first, with Allow for this task or Always) or never asks (Full control). With asking off there is no card, and the tiers still hold.
The keyboard is its own switch
Typing and pressing keys in another app follows Keyboard Control in Settings > Permissions, and it is off until you turn it on, at every permission level. With it off, every typing tool refuses and nothing is typed, and Mori is told not to reach around it through the shell. Typing into a terminal counts as running a command and meets the command rails.
Unattended runs and other limits
A scheduled run or anything nobody is watching never drives your screen and never types, whatever was allowed before: a stored yes was given by someone who was there to see it. On a cloud machine there is no screen of yours, so screenshots and opening apps are not offered there and pages are read with the web tools. Opening, focusing and quitting apps follow the App Launching switch; the mouse and window tools run at the Standard and Full Agent levels, never at Safe.
Each description below is the exact text her model reads when it decides which tool to call.
PREFERRED way to drive apps: list the interactive elements (buttons, fields, links, menus) of the frontmost window as a numbered list, read from the OS accessibility tree. Works with ANY model, no screenshot or vision needed. Then act with click_element. Re-run after every navigation: ids go stale when the screen changes.
Click an element from the last screen_elements list, by id (preferred) or by partial name. Resolves to the element centre and clicks it. Call screen_elements first; re-list after the screen changes.
Work one part of the task on the screen, by sight: the vision loop looks, clicks, types, drags and checks every step in the app you name, then hands back what it did, whether it is done, and the final screenshot. The LAST resort: use a tool, an API, a command or the app's own scripting (osascript, PowerShell, blender --python, ExtendScript) first, and this only for what they cannot reach (a panel, a dialog, a UI with no scripting). Give one small, checkable part, not the whole job.
FALLBACK ONLY, Capture and inspect the current screen state. Do NOT use this as a first action. Use direct tools first. Only use when you need to observe the screen after direct tools have failed.
Capture the screen and SHOW it to the user. It is the ONLY way to deliver a picture: a screenshot written to a file by a shell command sends nothing. Use it whenever they ask to see the screen, and to check your own work after doing something visual on the desktop; a page you BUILT is checked with preview_screenshot. The image's pixels map 1:1 onto mouse_click/mouse_move coordinates. Pass x,y,width,height to capture just that REGION at full detail. Do NOT reach for it to explore when a direct tool (open_app, read_file, list_files) answers better.
Move the mouse cursor to screen coordinates (x, y).
Move mouse to (x, y) and perform a left click. Use for clicking buttons, links, input fields, etc.
Move mouse to (x, y) and perform a double left click.
Move mouse to (x, y) and perform a right click (context menu).
Move mouse to (x, y) and click the middle button (open a link in a new tab, autoscroll).
Move mouse to (x, y) and triple-click (selects a whole line or paragraph).
Press and HOLD the left mouse button at (x, y). Pair with mouse_move and mouse_up to compose a custom drag; mouse_drag does a plain press-glide-release.
Release the left mouse button at (x, y).
Drag the mouse from one screen point to another (press, move, release), for drag-and-drop, sliders, moving items between panes, drawing. Coordinates are absolute screen pixels.
Scroll the mouse wheel. Positive = down, negative = up. Amount is in "clicks" (1-10 typical). Pass x,y to scroll a SPECIFIC pane: the wheel spins over whatever is under the cursor. axis "h" scrolls horizontally (positive = right).
Where the mouse pointer is right now, in the same coordinates the mouse tools and screenshots use. Returns "x,y".
Type text using keyboard simulation
Press one keyboard key or chord: "enter", "tab", "f5", "cmd+s", "ctrl+shift+t". Chord syntax works on both platforms (cmd on Mac, ctrl on Windows). To type text into a field use type_text, free text passed here is typed, not pressed.
Press a keyboard hotkey/shortcut using SendKeys format (e.g. "^s" for Ctrl+S, "%{F4}" for Alt+F4, "{ENTER}", "^a" for Ctrl+A).
Hold a key or chord for a duration then release it, e.g. a held arrow key in a game or a long-press shortcut.
Focus a running application and send keyboard keys to it. Use SendKeys format (e.g. "^k" for Ctrl+K, "{ENTER}", "^a").
Focus a running application and type or paste text into it. Better than type_text for long text or special characters, because it uses clipboard paste.
Set the FOCUSED text field's value directly through accessibility, without keystrokes: long text, or a field that rejects typing. Click or focus the field first.
List the running applications that have a user interface (name, pid, which is frontmost). Use before focus_app when unsure of a name.
Open any desktop application by friendly name: Discord, Chrome, Edge, Firefox, Brave, VS Code, Notepad, Calculator, Paint, Explorer, PowerShell, CMD, Terminal, Spotify, VLC, Slack, Telegram, Teams, Zoom, Steam, Word, Excel, OBS, Task Manager and more.
Bring a running application window to the foreground
Find a running application by process name and bring its window to the foreground. Returns NOT_RUNNING if the app is not open.
Quit a whole running APPLICATION by its app or process name ("Preview", "Slack"). Never use this to close a web page or browser tab, that is chrome_tabs with action=close; quitting the browser takes all the user's other tabs with it.
Check if an application is currently running. Returns RUNNING:count or NOT_RUNNING.
Open Windows Terminal or Command Prompt
Open Command Prompt
Open PowerShell
List every open window: which app owns it, whether it is minimized/maximized/normal, its position and size, and its title. Use before arranging windows, or to find what the user means by "that window".
Position, size, minimized and frontmost state of an application's front window.
Minimize, maximize or restore an app's main window. Use for "minimize Chrome", "maximize the editor", "bring back Notepad".
Move and resize an app's main window. Origin is the top-left of the primary display. Use for tiling or "put the browser on the left half".
The app and window title in front right now, and the one the user was in before switching to Mori. Your turn context already carries this under RIGHT NOW ON THIS MACHINE; call this only to re-check after an app switch.
Read the current text on the system clipboard, whatever the user last copied. Use when they say 'the thing I copied', 'paste it here', or hand you content by copying it.
Put text ON the system clipboard so the user can paste it anywhere with Cmd/Ctrl+V, a link, a snippet, anything they will paste into another app. Verified after writing.
Wait/pause for a specified number of milliseconds before the next action.
Run a shell command inside a throwaway sandbox Mac (cua's Lume VM) instead of on the person's own computer, then delete the sandbox. For untrusted downloads, scripts from the web, installers, and unattended runs. Needs Lume installed by the person and Sandbox runs switched on in Settings; status says whether it is ready and why not. The sandbox has its own disk: nothing it does reaches this computer's files.