dsh-plugin-vision

Auxiliary vision for DeepSeek Harness (dsh):
analyze images through an external OpenAI-compatible vision endpoint and get a text answer back.
Works with any main model — including DeepSeek, whose own API is text-only.

Why

dsh's built-in read_image tool injects the image
into the main model's context, which requires the main model to declare image input. The DeepSeek
adapter does not (inputModalities: ['text']), so read_image refuses to run with it. This plugin
takes the complementary auxiliary vision route: it sends the image to a separate vision model and
returns the answer as text. No main-model vision support needed.

How it works

1. Read the image — a local file path or an http(s) URL (size-capped, magic-byte MIME sniffing).
2. Encode it as an inline base64 data URL.
3. POST an OpenAI-compatible chat/completions request with [{type:text}, {type:image_url}] content.
4. Return the vision model's answer (falls back to reasoning_content for thinking models).
5. Transcription cache (new in v0.1.1): each (image, question) pair is transcribed once — keyed by
   image content sha256 (deduplicated across paste / read_image archive copies / resumed history) plus
   question sha256 — and later requests reuse the cached text. The vision endpoint sampling temperature is
   pinned to 0 (temperature) for deterministic output. The result:
   • once an image is in the conversation history, later turns no longer call the vision endpoint again
     → time-to-first-token (TTFT) stays flat;
   • the transcribed text never changes → the main model's (DeepSeek) prompt-prefix cache is not broken
     by transcription variance, so the cache hit rate no longer decays.

Capabilities

1. vision_analyze tool

Send an image (local path or URL) to the configured vision endpoint and get a
text answer — works with any main model, including text-only DeepSeek.

2. Image-attachment transcription (pasting images in the Web UI)

The Web UI accepts pasted/dropped images, but the DeepSeek adapter rejects
image content (UNSUPPORTED_CONTENT). This plugin hooks agent/pre-step —
the documented seam for replacing the messages that enter a step — and, when a
user message carries image blocks, calls the vision endpoint to describe them,
then swaps each image block for a [User-attached image description] text
block before the request reaches the model. DeepSeek only ever sees text,
so image attachments just work. When the active model route declares image
input, transcription is skipped and native vision takes over.

Toggle with attachImages (default true). Transcription failures degrade to
an explicit note instead of blocking the turn.

attachMode controls how the vision model is asked about attached images:

• auto (default): when the user wrote text with the image, that text is passed
  to the vision model verbatim as its question — "who is this?", "translate
  the text", "what is wrong on this page?" are answered directly, plus a
  one-line image summary for follow-ups. Without user text, a generic
  description is produced.
• describe: always use the generic description prompt, ignoring user text.

3. deepseek-vision provider route (pasting images in the Web UI, root fix)

The Web UI's upload preflight rejects images when the selected model does not
declare image input — so pasted images never even reach the agent on the
plain DeepSeek route. This plugin registers a deepseek-vision provider: a
DeepSeekAdapter subclass that declares image input (preflight passes) and
transcribes attached images to text at request time before delegating to the
real DeepSeek chat-completions endpoint. The main model is still DeepSeek —
same endpoint, same key, same models. Select "DeepSeek (vision via plugin)"
in the model picker, then paste/drop images as usual.

Toggle with deepseekVision.enabled (default true); the provider id is
deepseekVision.providerId (default deepseek-vision). Transcription goes
through the content-hash cache, so once an image is in the history every later
turn reuses the cached text instead of calling the vision endpoint again.

4. opencode-vision provider route (OpenCode Go + image transcription)

The same wrapper idea over OpenCode Go (opencode.ai/zen/go),
which is a pi-ai built-in provider in dsh. This plugin registers an
opencode-vision provider: a PiAiAdapter subclass that serves the full
opencode-go catalog (deepseek-v4-flash/pro, qwen3.7-max/plus, kimi-k3,
glm-5.1/5.2, minimax-m3, grok-4.5, …), declares image input so the Web UI
upload preflight accepts pasted images, and transcribes attached images to text
at request time for text-only models. Models the catalog marks as natively
image-capable (e.g. qwen3.7-plus, minimax-m3) stream the image through
untouched. Select "OpenCode Go (vision via plugin)" in the model picker,
then paste/drop images as usual.

Toggle with opencodeVision.enabled (default true); the provider id is
opencodeVision.providerId (default opencode-vision). The credential is the
same OPENCODE_API_KEY as the base opencode-go route, so store the key once
(via the web Models page or OPENCODE_API_KEY) and both routes work.

Models the installed pi-ai catalog does not describe yet (your account may
support newer ones) can be added outright with opencodeVision.extraModels —
the wire protocol is required, endpoint and caps default to the OpenCode Go
gateway:

config:
  opencodeVision:
    extraModels:
      - id: glm-5.3
        api: openai-completions
        contextWindow: 1000000
        maxTokens: 131072
      - id: qwen3.8-max
        api: anthropic-messages

To also enable the plain (non-vision) opencode-go route with all its models,
add a llm-pi-ai section to $DSH_HOME/settings.yaml (hot-reloaded, no
restart):

# $DSH_HOME/settings.yaml
llm-pi-ai:
  providers:
    opencode-go:
      apiKeyEnv: OPENCODE_API_KEY
      displayName: OpenCode Go
      # Omit `models` to serve the full installed catalog instead.
      models:
        - id: deepseek-v4-flash
        - id: deepseek-v4-pro
        - id: qwen3.7-plus
        - id: grok-4.5
        # …all opencode-go models you want to use

No-second-port setup: load the plugin into your existing GUI

The vision API key resolves through the harness credentials service first (the
Web UI's stored keys) and falls back to the environment — no export needed in
the GUI. Write the plugin into the home-level user patch, which applies to
every profile (including the web GUI you already run):

# $DSH_HOME/cordis.patch.yml
- insert:
    - id: vision
      name: 'file:///<path-to-plugin>/lib/index.js'

Store your vision API key in credentials (or export VISION_API_KEY), then restart the
GUI once. Use the built lib/index.js — the published CLI loads .ts
entries but cannot resolve their .js-suffixed sibling imports from src/.

4. Image archive (auto-save pasted images + index.json)

Every pasted image is automatically saved to ~/.dsh/image-archive/ with a
date-numbered name (2026-08-14_120331_001.png), recorded in an index.json
manifest (path, sha256, size, source, optional note), and its location is
annotated for the model ([图片已存档: …]). Two tools complete the flow:

• image_archive — the agent archives an important image (user preferences,
  receipts, key data) into a named folder with an optional note: it saves
  <archiveDir>/<folder>/<name-or-date>.png and updates index.json.
• image_archive_find — search the manifest by name, folder, or note.

Configure with archive.enabled (default true) and archive.dir (default
~/.dsh/image-archive). Deduplicated by attachment id.

Try it

Install the bundle into any profile (the prepare script builds on install):

dsh plugin --profile web add github:Tianbaidi/dsh-plugin-vision

Store the vision API key in your credentials or environment
(ALIBABA_CODING_PLAN_API_KEY), then restart the GUI. Paste/drop an image and
ask — or use the vision_analyze tool directly. To use the opencode-vision
route, also store OPENCODE_API_KEY (the web Models page accepts both).

Prefer a dev overlay instead? Point it at your local checkout:

- insert:
    - id: vision
      name: 'file:///<path-to-plugin>/lib/index.js'

Windows note: plugin paths in overlays must be file:// URLs
(file:///D:/...%20...), not bare D:/... paths — the ESM loader rejects
the latter as scheme d:.

Configuration

Any OpenAI-compatible vision endpoint works. The defaults are deliberately
empty so no provider is assumed; pick one:

Configure per deployment (e.g., your profile's cordis.patch.yml or the plugin
row's config):

- id: vision
  name: dsh-plugin-vision
  config:
    baseUrl: https://open.bigmodel.cn/api/paas/v4
    apiKeyEnv: VISION_API_KEY
    model: glm-4.6v-flash
    timeoutMs: 120000

Multiple transcription endpoints (registry)

When one endpoint is not enough — e.g. a token plan that may expire, with
OpenCode Go vision models as the fallback — register several named endpoints
and pick the active one with visionEndpoint. Every field a preset omits
inherits the top-level value, so a preset usually only names what differs:

- id: vision
  name: dsh-plugin-vision
  config:
    # Active transcription endpoint (switch here when the plan expires).
    visionEndpoint: aliyun
    visionEndpoints:
      - id: aliyun
        baseUrl: https://token-plan.cn-beijing.maas.aliyuncs.com/compatible-mode/v1
        apiKeyEnv: ALIBABA_CODING_PLAN_API_KEY
        model: qwen3.7-plus
      - id: opencode-kimi
        baseUrl: https://opencode.ai/zen/go/v1
        apiKeyEnv: OPENCODE_API_KEY
        model: kimi-k3
        temperature: 1   # required: kimi models reject the default temperature 0
      - id: opencode-qwen
        baseUrl: https://opencode.ai/zen/go/v1
        apiKeyEnv: OPENCODE_API_KEY
        model: qwen3.7-plus

OpenCode Go candidates verified against the live endpoint (2026-08-16, real
key, image payload): of the catalog's image-capable go models, only kimi-k3
and kimi-k2.7-code work as transcription endpoints today, and both require
temperature: 1 — the plugin's default temperature: 0 is rejected with
"invalid temperature: only 1 is allowed for this model". mimo-v2.5 answers
but intermittently returns empty text; minimax-m3 answers but emits its
reasoning inline (<think>…) into the transcription. kimi-k2.6,
qwen3.6-plus, and qwen3.7-plus currently fail with upstream 503 "Endpoint
is unavailable" (text and image, direct and proxied) — not a client issue;
retry when the gateway recovers. Switching endpoints never reuses cached text
from another model: each preset's id namespaces its transcription-cache
entries.

Develop

pnpm install        # installs the published @deepseek-ai peer packages
pnpm typecheck
pnpm test           # 58 vitest cases: MIME sniffing, payload, parsing, source loading, execute, transcription cache, provider routes, endpoint registry

Changelog

v0.3.0 — transcription endpoint registry

New visionEndpoints / visionEndpoint config: register any number of named
transcription endpoints and switch the active one with visionEndpoint. Each
preset inherits unset fields from the top-level config, so a preset usually
only names baseUrl / apiKeyEnv / model. The preset id namespaces
transcription-cache entries — switching endpoints never reuses another model's
cached text. activeVisionEndpoint(config) resolves the active endpoint and is
exported. Legacy single-endpoint config (baseUrl / apiKeyEnv / model)
keeps working unchanged (no id → default cache bucket, on-disk entries stay
valid).

v0.2.0 (2026-08-16) — opencode-vision provider route

New opencode-vision provider route: serves the full pi-ai OpenCode Go
catalog under one picker entry, claims image input for the Web UI preflight,
and transcribes attached images at request time for text-only models
(natively image-capable models such as qwen3.7-plus / minimax-m3 stream
the image through untouched). Built as a PiAiAdapter subclass over the
installed pi-ai catalog, so all three wire protocols (openai-completions,
anthropic-messages, openai-responses) and the provider's reasoning
special-casing keep working. Same OPENCODE_API_KEY credential as the base
opencode-go route. opencodeVision.extraModels merges uncatalogued account
models (e.g. glm-5.3, qwen3.8-max) into the route with an explicit wire
protocol. 7 new tests (catalog listing, extra-model merge, image claim,
transcription, native-vision passthrough, replay-state alignment,
registration).

Fixed in v0.2.0: the delegate rewrites model.provider to the catalog route
at wire time, which made pi-ai stamp the session replay state with
opencode-go while the harness recorded the message source under
opencode-vision — a resumed session then failed with "invalid pi-ai replay
state: provider does not match assistant source". The adapter now realigns
every finish chunk's replayState.provider to the route id. Sessions written
before the fix can be repaired by rewriting each message's
replayState.provider to its source.provider in the exported JSONL.

v0.1.1 (2026-08-16) — Major fix: TTFT and cache hit rate

Two performance issues in the deepseek-vision provider, located from real
session telemetry:

• TTFT inflation: once an image entered the conversation history, every
  request re-transcribed it synchronously (measured TTFT degraded from ~1s to
  22–69s). Added transcriptionCache (image content sha256 + question sha256,
  persisted to ~/.dsh/vision-transcription-cache.json): each (image, question)
  pair is transcribed once, later requests reuse the cached text.
• Cache hit rate decay: the non-deterministic transcription text permanently
  broke the main model's prompt-prefix cache from the image position onward
  (measured hit rate decayed from 99.8% to 91.1% and kept falling). Added
  temperature (default 0) and optional seed for deterministic output.
• Added 5 tests covering cache hits, content-hash keying (paste vs read_image
  archive copy), persistence, eviction, and deterministic payloads.

v0.1.0

Initial release: vision_analyze / vision_reask / image_archive tools,
image-attachment transcription, deepseek-vision provider route, image archive.

Known limitations

• Image sources are file paths (resolved against the harness cwd) or plain http(s) URLs. Remote
  URLs are fetched directly without SSRF hardening — restrict this tool to trusted networks if you
  use it with URL inputs.
• The image is sent to the configured endpoint as-is; oversized images are rejected rather than
  downscaled (no Pillow dependency). Compress first for very large screenshots.
• Token costs of the vision call are charged to the configured endpoint's plan.
• The transcription cache is keyed by image content sha256 + question sha256: a new question about
  the same image triggers one fresh vision call (which is then cached). To force a refresh, remove
  the matching entry from (or delete) ~/.dsh/vision-transcription-cache.json.
• Once an image is in the conversation history (pasted by the user, a read_image tool result, or
  a resumed session), every later request carries it; the cache ensures the vision endpoint is only
  called on the first occurrence, but the transcribed text still occupies tokens in the history.

Publish

This project is a bundle (dsh.bundle.patch). Install with dsh plugin add, share on GitHub with
the dsh-plugin topic, or npm publish. See
PUBLISH.md in the companion scaffold repo for the full checklist.

License

MIT