FilipNowakowicz/agent-desktop

Give your agent its own Linux desktop while you keep using your computer.
Agent Desktop runs persistent, separate graphical sessions controlled through a
CLI or a local MCP server. Agents can launch applications, inspect windows and
accessible UI elements, take screenshots, and send keyboard and pointer input.
You can watch a session or take control when a task needs human interaction.
Experimental, Linux-first. Headless sessions have been tested locally on
NixOS/Hyprland and in Ubuntu, Fedora and Arch CI environments. Those checks cover
specific tasks, not every application or desktop installation. See the
compatibility matrix for versions and known failures.
Applications run as your user with host filesystem and network access. Separate
display sockets, application profiles and input routing do not provide a
security sandbox. The optional host guard reduces accidental host changes.
## What it provides
Persistent headless labwc sessions with private display sockets and D-Bus.
Explicit session routing for window focus, screenshots, Unicode typing,
key chords, clicks, dragging and scrolling.
Screenshot observation tokens, waits and bounded action sequences to help
detect changes before acting; verify outcomes after each action.
Optional AT-SPI element inspection and semantic actions.
A read-only observer, cooperative human takeover (with clipboard paste for
passwords) and named login profiles that keep logins between sessions.
Process supervision, cleanup, bounded screenshot retention and diagnostics.
The runtime uses existing Linux desktop infrastructure and works with existing
agents. It does not require a particular model or cloud service.
## Installation
Use Python 3.12 or newer managed by uv. From a local
checkout of this repository, install the locked Python dependencies:
sh
uv sync --managed-python --locked
Python packages alone do not supply the compositor or capture tools. Two
installation paths are supported and tested in CI; see
runtime/INSTALL.md:
NixOS or any Linux with Nix: nix build ./runtime/nix --out-link
~/.local/share/agent-desktop/runtime builds the runtime, including the X11
repair. The CLI and MCP server use it automatically.
Ubuntu 24.04: sh runtime/install-ubuntu.sh installs the packages below.
On other distributions, install the same tools with the package manager:
| Dependency | Purpose |
| labwc, grim, dbus-daemon | Required: compositor, screenshots, private session bus |
| foot | Terminal for the quick start and test fixtures |
| xwayland | Optional X11 application support |
| wayvnc, TigerVNC vncviewer | Optional observer and interactive takeover |
| at-spi2-core | Optional semantic UI access; coverage depends on the application |
Package names and versions differ between distributions. Run preflight before
creating a session:
sh
uv run agent-desktop doctor
uv run agent-desktop doctor --smoke # create, capture and destroy a test session
Stock wlroots 0.19.3 and 0.20.2 have a reproduced intermittent X11 mapping failure.
See the optional project-local runtime repair when using
those versions. A passing smoke check does not establish sustained reliability.
Runtime paths can be set with --labwc, --grim or corresponding