Mori has three ways to act on the web and your screen — a router picks the right one per request, and every purchase stops at a hard gate you control.
Before any model call, Mori classifies your message and routes web and screen work to one of three engines. All three run locally: the desktop app talks directly to the provider with your own key, and nothing is proxied through Studio Mori servers.
Both routes require agent mode to be on (it is by default). Anything that is not a web or screen task falls through to the normal tool runtime.
Routing is deterministic, not model-guessed. isBrowserIntent matches shopping language (buy, order, add to cart, checkout), major store names (Amazon, eBay, Walmart, Target, Etsy and others), webmail work that names a mail site ("check my gmail", inbox triage, drafting or replying in webmail), and go-to-a-site-and-act patterns ("go to X and search / fill / book / sign in / post"). isComputerUseIntent fires only on explicit operate-the-machine phrasing — "use the computer", "control my screen", "take over the screen", "click around".
Two exclusions keep tasks from being hijacked into the wrong engine. Build-a-deliverable requests ("make a PDF and email it") stay in the main runtime, which can create files and attach them — the browser agent cannot. And email addresses are stripped before matching, so "send an email to sam@gmail.com" goes to the send_email tool rather than triggering a Gmail browser run; the DOM agent only gets email jobs that explicitly name the webmail site.
Inside the main runtime, desktop-app work follows a fixed preference order: direct tools first (open a file, run a command, call the tool that does the job with no UI at all), then element control (screen_elements + click_element, below), and screenshots only as a last resort. The habit is enforced in code: a turn that opens with a screenshot as its very first tool call is rejected and re-asked with instructions to use direct tools or element control instead.
"order AA batteries on amazon" → browser engine (shopping)
"check my gmail for anything urgent" → browser engine (webmail triage)
"go to news.ycombinator.com and find the top post" → browser engine
"use the computer to clean up my desktop" → computer-use loop
"build a report as a PDF and email it to Sam" → normal runtime (not the browser)When a web task arrives, Mori checks whether the Mori Chrome extension is online. If it is, the task runs in your real Chrome — your logins, your sessions, your autofill — instead of the operator browser. The desktop and the extension join an account-scoped realtime channel (mori-chrome-<uid>), so signing into the same account on both is the only pairing step; presence detection recognizes them automatically, with no pairing code. The desktop broadcasts the task, the extension streams step updates back, and the run times out after 3 minutes if the extension stops responding.
This path needs no Anthropic key on the desktop side — the extension does the work. It is also exposed to the main runtime as the use_mori_chrome tool, offered to the model only while the extension is connected, so Mori can hand logged-in web work to your real browser mid-task. If the extension is not connected and no Anthropic key is set, Mori tells you plainly which of the two to add rather than failing silently.
Without Mori Chrome, web tasks run in Mori's built-in operator browser through a fast DOM loop. Each turn, Claude receives the current URL, page title, an indexed list of every interactive element with its exact text, and the page text — no screenshots in the hot loop. It acts by referencing elements by index through a single browser tool with actions goto, click, type, submit, press, select, scroll, and done. This is roughly 10–50x faster than a vision loop, and the spend guard is stronger because Mori knows the exact label of every button before it is clicked.
The loop caps at 45 steps, streams a screenshot to the UI every few actions so you can watch it work, and honors the Stop button between every action. It requires an Anthropic API key (default model claude-sonnet-4-6), called directly from your machine.
When Mori drives a desktop app through her normal tool runtime, the preferred instrument is a pair of tools that read the operating system's accessibility tree instead of taking screenshots. screen_elements asks the OS — System Events over the AXUIElement tree on macOS, UI Automation on Windows — for the frontmost window's interactive elements, and returns them as a numbered text list: buttons, text fields, links, menus, checkboxes and sliders, each with its role and name, up to 120 per listing. click_element then acts on one by id (or by exact or partial name), resolving the cached entry to its center coordinates and clicking it — pass double for the double-click that opens files and folders.
Text in, text out — and that has three concrete consequences:
The list is a snapshot, and it goes stale the moment the screen changes — so every tool result reminds the model to re-run screen_elements after each navigation, and click_element refuses to act when no current list exists. When an app exposes no accessibility tree at all (some Electron apps), the tool says so explicitly and Mori falls back to take_screenshot plus coordinate clicks — screenshots are the fallback, never the default. On macOS this path uses the same Accessibility permission as the rest of computer use, and Mori sets the AXManualAccessibility attribute on the target process, which coaxes many Electron apps into exposing their tree.
screen_elements
→ Interactive elements of "System Settings" (23):
1. [Button] General
2. [Button] Appearance
3. [TextField] Search
…
Act with click_element(id) or click_element(name). The list goes
STALE when the screen changes — re-run screen_elements after
navigation.
click_element { id: "2" }
→ clicked — [2] Button "Appearance". If the screen changed, call
screen_elements again before the next click.For explicit operate-the-machine requests, Mori runs a vision loop: screenshot, Claude picks one action (click, double-click, right-click, type, key combo, scroll, move, wait), Mori executes it on the real OS, screenshots again, repeats — up to 70 steps. Before starting, it prechecks macOS permissions: if Screen Recording is not granted it stops and tells you exactly which toggle to flip, and if Accessibility is missing it opens that settings pane and registers the app for you, since without it synthetic clicks are silently discarded by macOS.
Autonomy defaults to ask: every click, keystroke, and typed string is described and requires your approval before it executes, and a declined action is reported back to the model rather than performed. You can switch to auto for trusted runs, but the money rules below hold in both modes. This engine requires an Anthropic API key.
Mori will drive shopping all the way to the final review screen and no further. The rule is layered so no single failure can break it.
Completing the purchase requires your explicit approval of the exact order in chat. Only then does Mori re-read the review page, find the place-order button by its exact label, click it once, and verify the confirmation page before reporting success. Payment details always come from your own saved sessions and autofill — Mori never types a card number, and it never executes trades.
Mori: The order's ready at the final review screen. I won't place it
myself — tap "Place the order" below (or reply "place my order")
and I'll click "Place your order". Tap Cancel to leave it.Mori is bring-your-own-keys, and the three engines have different requirements.
If a web task arrives with neither the extension connected nor a key configured, Mori says so and names both options — it never falls back to a degraded engine without telling you.