citydirector/ui-locate-mcp

English · 简体中文
Find on-screen UI text locally and return pixel coordinates for desktop interaction. Combined tools can locate, click, type, scroll and attempt to verify the result.
Detection, OCR and actions run locally. Initial setup downloads Python dependencies and a detection model of about 80 MB. A persistent service runs inference; a lightweight MCP process forwards requests.
## Quick start
1. Enter ui-locate-mcp in DSH Plugins → Add plugin, then enable it.
2. Open ui-locate runtime in Settings, choose paths and a backend, and prepare the environment. The Host tool uilocatesetup provides the same preparation path.
3. After preparation, connect or re-enable mcp-ui-locate. Restart if DSH requests it.
4. Call health, then try locatetext(text="Save").
The bundle contains an MCP client, a Host runtime-management plugin and a Client settings page. npm installation supplies the code; Python dependencies and the model require separate setup.
### Requirements
The current bundle's MCP executable path is configured for Windows. Cross-platform Python code does not imply a cross-platform bundle configuration.
The Python project requires Python 3.12+. The settings preparation uses a Python runtime located by the host to create a dedicated environment. Check the desktop installation's bundled runtime if no interpreter can be found.
DirectML is the default backend; cuda and cpu are alternatives. CUDA needs the corresponding NVIDIA environment and larger dependencies. Changing backends requires rebuilding the environment.
package.json currently declares no engines.dsh range. This is not a guarantee of compatibility with every DSH version: the bundle needs tool registration, an MCP client and settings services.
For a Web profile:
~~~sh
dsh plugin --profile web add ui-locate-mcp
~~~
Stop the instance before CLI changes and restart afterwards. Prefer the Plugins page on Desktop; managing its desktop profile requires that installation's CLI and the same DSHHOME.
## Tools
| Tool | Purpose |
| locatetext | Find matching UI text and its center coordinates |
| findall | Return multiple matching candidates |
| detectelements | Enumerate detected elements |
| uilocatemarks | Screenshot with numbered marks (and an optional named grid) so the model picks a number instead of guessing pixels |
| uilocateact | Locate, perform an action and attempt verification |
| uilocatesteps | Run ordered steps and identify failures |
| uilocatewindows | List top-level windows (title, handle, client rect, foreground) |
| uilocatediff | Where the screen changed since the last call, as rectangles (no image, no vision tokens) |
| uilocatewatch | Record a few seconds and return one image of the keyframes, cropped to what changed |
| uilocateframes / uilocateframe | List or retrieve recent action frames |
| listmonitors | Display sizes, origins and indices |
| health | Check service readiness |
| uilocatesetup | Prepare or rebuild Python dependencies and models |
The first twelve entries come from MCP; the last two are Host tools. The host may prefix tool names with a namespace.
~~~python
locatetext(text="Save", monitor="primary")
uilocateact(text="Save", action="click", monitor="primary")