Phone-Eye 📱👁️

Let your AI agent see — and operate — a real Android phone.

Your coding agent can read code, run commands, and browse docs. Can it see the
app it's building, on a real phone, with real pixels? Phone-Eye gives it eyes
and a finger.

phone_look("what's on screen, and where is the login button?")
  → "Login button at (540, 1830) — a green 'Sign in' …"
phone_tap(540, 1830)
phone_look("did the next page load?")

Five verbs, no framework: your agent composes them into whatever workflow it
needs. Works with any MCP client — Claude Code, Codex, Cursor, dsh, and
friends.

Why

If you build or test anything that ends up on a phone, you know this loop:
the layout is broken on the real device, you screenshot by hand, describe
screens in words ("the gear, top right, next to the account thing"), and play
coordinate-decoder between your agent and your phone. Phone-Eye closes that
loop — the agent iterates with the device the way it already iterates with
your codebase.

Two channels, fused:

• Vision — a vision model answers natural-language questions about the
  live screenshot (works on game canvases, images, anything pixels can show).
• UI tree — uiautomator dump for exact text and bounds when the
  accessibility tree has them.

Vision is pluggable. Any MCP server exposing describe_image(path, question) works — cloud GLM vision out of the box, or point it at a local
Qwen-VL endpoint so no pixel ever leaves your LAN.

Install

Requirements: Python 3.10+, adb (platform-tools / android-tools) on PATH,
a phone with USB debugging on, and any MCP vision server.

git clone https://github.com/boheastill/phone-eye
cd phone-eye
pip install -r requirements.txt

# env (defaults shown):
export ANDROID_SERIAL=""                      # empty = first adb device; or 192.168.x.x:5555
export PHONE_EYE_VISION_URL="http://127.0.0.1:8102/mcp"  # your vision MCP

Wire it into your client (stdio):

// Claude Code / dsh-mcp-client / any stdio MCP config
{
  "mcpServers": {
    "phone-eye": { "command": "python", "args": ["/path/to/phone-eye/server.py"] }
  }
}

Or run it as an HTTP service (streamable-http) behind your own fleet and add
http://<host>:8122/mcp — see docs/fleet.md.

Connect the phone (one-time)

USB once, then Wi-Fi forever:

adb devices                      # USB: accept the debugging prompt on the phone
adb tcpip 5555                   # switch to Wi-Fi mode
adb connect <phone-ip>:5555      # unplug and go

Tools

What it is / isn't

✔ agent eyes + hands on one real Android device, zero on-device install, no root
✔ vision-first (works where UI trees can't see) with tree-fusion for precision
✔ privacy option: point vision at a LAN-only model

✘ not a test framework (no DSL/recorder — the agent is the logic)
✘ not iOS, not device farms (yet)
✘ not a mobile UI for humans (that's a different product)

Status & roadmap

Phase 1 (now): the five verbs, single device, pluggable vision — verified
end-to-end on real hardware (Redmi K40 Gaming / Android 13). Its favorite
party trick so far: it discovered a USB-debugging authorization dialog on its
own screen, read the buttons, and tapped "Allow" itself.

• [ ] Phase 2: offline vision quick-start, multi-device addressing,
  retry/verify wrappers
• [ ] Everything else: request-driven — open an issue and it moves up the queue.

License

MIT

FAQ

Vision server? I don't have one. Any MCP server exposing
describe_image(path, question) works. The quickest cloud option is a GLM
vision endpoint; for fully-offline, a local Qwen-VL (llama.cpp / Ollama
OpenAI-compatible + a 20-line adapter) keeps every pixel on your LAN.

Why not a dsh-native plugin? MCP-first means the same five verbs work in
every client. dsh users can wire it via @deepseek-ai/dsh-mcp-client or
dsh plugin add.

Verified on

Add yours via a PR to this table.