agent-desktop Moves to Vercel Labs: An Open-Source Tool for Agents to Control Desktop Apps
agent-desktop, a Rust-based tool that enables AI agents to read and control desktop applications via the operating system's accessibility tree rather than screenshots, reached version 1.0 on October 28 and was transferred to the Vercel Labs organization. It runs on macOS, Windows, and Linux.
AI agents operating a computer often “look” at the screen through screenshots, then click at specific coordinates. agent-desktop takes a different path: this open-source tool reads the operating system’s accessibility tree, the structured description of the interface that applications already expose for screen readers. The project reached version 1.0 on October 28, 2026, and was transferred to the Vercel Labs organization, as noted in the release notes for version 1.0.2, published on October 30.
Observe, Act, Verify
agent-desktop is a command-line tool and a C library, written in Rust and released under the Apache 2.0 license, for macOS, Windows, and Linux. It relies on each platform’s accessibility API: AX on macOS, UI Automation on Windows, and AT-SPI on Linux. A snapshot command returns JSON with a stable reference for each actionable element; the agent clicks or enters text using a reference, then takes another snapshot to verify the outcome.
The tool is not an agent itself: it does not bundle any language model and leaves the decision loop to the agent calling it. It features 60 command names (observing, clicking, typing, managing windows and applications, clipboard, notifications…), installs with npm install -g agent-desktop, and provides its own prompt instructions for agents to load before first use. It complements agent-browser, Vercel Labs’ tool dedicated to web pages, and can hand off control to it for web content inside Chromium or Electron applications.
Limitations
This approach relies on the quality of an application’s accessibility support: an interface that does not expose an accessibility tree remains invisible. macOS requires Accessibility permission; on Linux, a graphical session with AT-SPI is required, and certain actions, such as mouse control, only work under X11. Official Homebrew and Scoop packages are still pending submission. Finally, Vercel has not published a dedicated announcement: the project, whose releases are published by developer lahfir, joined Vercel Labs through a straightforward repository transfer.
Our take
Reading the accessibility tree rather than screenshots makes an agent’s actions more reliable and easier to verify: an element reference does not drift like click coordinates do. The downside is well known to screen reader users: many applications expose their interfaces poorly, leaving the tool completely blind to them. By bringing it into Vercel Labs alongside agent-browser, Vercel is assembling a complete stack for agents to interact across both the web and the desktop, stepping directly into territory staked out by computer-use features from major AI labs.
Sources
- GitHub — vercel-labs/agent-desktop Primary source
- GitHub — vercel-labs/agent-desktop Releases (v1.0.0 to v1.0.2) Primary source
- agent-browser.dev — agent-desktop documentation Primary source
A primary source comes directly from the company, lab or organization concerned. A secondary source reports the information without being its author: we flag it as such. Spotted an error? See our corrections policy.
