dsh-tool-vision

GitHub: bcdahb0-jpg/dsh-tool-vision

External vision model for DeepSeek Harness.

DeepSeek's own models are text-only, and the harness derives every model
request strictly from the session log (llm/stream requests must equal the
durable derivation — the agent-loop invariant). This plugin bridges the gap
in two ways:

1. inspect_image tool — sends an image (local file, or http(s) URL) to
   any OpenAI-compatible /chat/completions endpoint that supports
   image_url content parts, and returns the vision model's textual answer
   into the agent loop.

2. Image bridge (v0.2.1) — pasted images are turned into inspect_image
   hints before they enter the durable log, on the agent/pre-step
   waterfall (the one seam where the harness lets a plugin replace the
   messages of a proposed step). Images already logged by an older version
   are repaired lazily with a surface replace on the session's first
   pre-step. Only models listed in multimodalModels receive image blocks
   directly; a model's declared inputModalities are never consulted,
   because profiles routinely declare input: [text, image] on text-only
   models just to pass the harness's prompt-admission check.

   Images render in chat (v0.3.8): when the session model runs on the
   bridge route (default provider tool-vision), the durable log KEEPS the
   original image blocks, so the chat renders the pasted image instead of a
   [User sent an image ... exported to: <path>] hint text. The bridging
   logic stays fully behind the scenes: the bridge adapter rewrites image
   blocks into inspect_image hints at stream time, so the text-only
   upstream still receives exactly the same hint as before.

• Zero dependencies beyond the dsh SDK — works with any compatible endpoint:
  OpenAI GPT-4o, Qwen-VL (DashScope), GLM-4V (Zhipu), Moonshot, Gemini
  compatible endpoints, local Ollama, etc.
• Registered on the global tools layer: every agent in the process can
  call inspect_image.
• Web UI settings section (v0.3.0): Settings → 视觉模型 edits the
  tool-vision namespace (API endpoint, write-only key, model, bridge
  options) in settings.yaml; changes hot-apply without a restart. The API
  key lives in settings.yaml, not the profile patch. Mount by package name
  (name: 'dsh-tool-vision') so the web client bundle is discovered.

Install

Install from GitHub (recommended):

dsh plugin --profile <profile> add github:bcdahb0-jpg/dsh-tool-vision

Then mount it in a profile patch ($DSH_HOME/profiles/<name>/cordis.patch.yml):

- insert:
    - id: tool-vision
      name: 'dsh-tool-vision'
      config:
        baseURL: 'https://api.openai.com/v1'
        apiKeyEnv: 'VISION_API_KEY'
        model: 'gpt-4o-mini'

Or load it from a local path without installing the package:

    - id: tool-vision
      name: './plugins/dsh-tool-vision/index.js'

Config

Bridge model entry (self-contained, v0.4.0)

When enabled, the plugin registers an image-admission route in the model picker
(default provider tool-vision, names like DeepSeek V4 Flash（tool-vision 桥接）).
Selecting it:

1. lets pasted/dropped images pass the harness prompt admission (previously you
   had to declare input: [text, image] under llm-pi-ai in settings.yaml —
   now the plugin owns that gate end to end);
2. the image bridge (agent/pre-step) turns images into inspect_image hints,
   or keeps them as image blocks via the multimodalModels whitelist and
   rewrites them at request time;
3. text turns delegate unchanged to the real bridgeProvider adapter (by
   default the official DeepSeek route).

Decoupling: the bridge model entry, the image bridge, the inspect_image
tool, and the settings namespace are all registered/unregistered by this plugin.
Removing the plugin removes the picker entry, the bridge, the tool, and the
settings together — nothing survives in settings.yaml; swap vision solutions by
swapping the plugin.

Image bridge setup

1. In your model settings, declare image input on the models you paste
   images onto, so the harness admits image messages (pi-ai style):

   llm-pi-ai:
    providers:
      your-provider:
        models:
          - id: deepseek-v4-flash
            input: [text, image]

2. List genuinely multimodal models in the plugin config so they receive
   image blocks untouched:

   - id: tool-vision
    name: 'dsh-tool-vision'
    config:
      multimodalModels: ['mimo-v2.5', 'grok-4.5']

Then pasting an image:

• on the bridge route (recommended): the chat renders the pasted image
  directly; the text-only model still receives the inspect_image hint
  ([User sent an image, exported to: <path>. Inspect it with the inspect_image tool...]), and the agent inspects it through the configured
  vision endpoint;
• on any other text-only model: the transcript stores that hint text (the
  pasted image no longer renders as pixels in that message), and the agent
  inspects it through the configured vision endpoint.

Why not llm/stream? The harness freezes every request and the agent-loop
invariant fails any request whose messages diverge from the session-log
derivation (log-reconstruction desync), and this cordis waterfall's
next() cannot replace request arguments. The agent/pre-step waterfall is
the supported seam: its decision messages become the durable log, so the
invariant stays satisfied.

Key resolution order: config.apiKey → process.env[apiKeyEnv] →
process.env.OPENAI_API_KEY.

Tool: inspect_image

Example endpoints (baseURL):

• OpenAI: https://api.openai.com/v1 — gpt-4o, gpt-4o-mini
• Alibaba DashScope (Qwen-VL): https://dashscope.aliyuncs.com/compatible-mode/v1 — qwen-vl-plus, qwen-vl-max
• Zhipu (GLM-4V): https://open.bigmodel.cn/api/paas/v4 — glm-4v-flash (free tier), glm-4v-plus
• Moonshot (Kimi): https://api.moonshot.cn/v1 — moonshot-v1-8k-vision-preview
• Ollama local: http://localhost:11434/v1 — llama3.2-vision (no key)

Limitations

• On the bridge route the image blocks stay in the session log (the chat
  renders the image), but the text-only model still receives a text hint —
  pixel-precise in-context reasoning is not available to text-only models;
  the vision model's description comes back through inspect_image. On any
  other text-only model the transcript stores the hint text itself.
• Images are base64-transferred; mind privacy and size limits.
• Independent of the dsh-llm routing/retry system; failures return clear
  errors to the agent.

License

MIT