dsh-plugin-vision

🤔 What is this?

dsh-plugin-vision is a plugin for DeepSeek Harness that enables pure-text LLMs to analyze images.

When your underlying language model doesn't support native image input (e.g., DeepSeek-V3 text-only), this plugin bridges the gap by calling Zhipu AI's GLM-4V-Flash — a free vision model — to analyze images and return text descriptions.

User pastes image → Agent calls vision_analyze → GLM-4V-Flash analyzes → Text description returned

✨ Features

• 🖼️ Analyze local image files (PNG/JPEG/WebP/GIF)
• 🌐 Analyze HTTP(S) URL images
• 🔤 Automatic base64 encoding for local files
• 🆓 Uses Zhipu's free vision model — no payment required
• 🔌 Follows DSH Capability Seam architecture
• 🧩 Drop-in Cordis plugin — works with cordis.yml

📦 Installation

Option 1: Clone and link

git clone https://github.com/co-Elly/dsh-plugin-vision.git
cd dsh-plugin-vision
pnpm install
pnpm build

Then in your DSH project's cordis.yml:

plugins:
  - name: /path/to/dsh-plugin-vision/dist/cordis.js
    config:
      apiKey: "your-zhipu-api-key"

Option 2: npm (coming soon)

pnpm add dsh-plugin-vision

⚙️ Configuration

Get a free API Key

1. Visit Zhipu Open Platform
2. Register and log in
3. Create an API Key in the API Keys page
4. The glm-4v-flash model is free — no payment needed

Environment variables

export ZHIPU_API_KEY="your-api-key"

cordis.yml

plugins:
  - name: dsh-plugin-vision/cordis
    config:
      apiKey: "your-api-key"           # or use $ZHIPU_API_KEY
      baseUrl: "https://api.z.ai/api/paas/v4/chat/completions"  # default
      model: "glm-4v-flash"            # default

🚀 Usage

Once registered, the agent automatically gets the vision_analyze tool:

vision_analyze({
  image_path: "/path/to/screenshot.png",
  question: "What is shown in this chart?"
})

Parameters

Returns

{
  "description": "A bar chart showing monthly revenue..."
}

🏗️ Architecture

This plugin follows DSH's Capability Seam pattern:

┌─────────────────────────────────────────────────┐
│                  dsh-plugin-vision               │
├─────────────┬─────────────────┬─────────────────┤
│  vision.ts  │  vision-glm.ts  │  tool-vision.ts │
│  (Service   │  (Provider:     │  (Consumer:     │
│  Definition)│  GLM-4V-Flash)  │  vision_analyze)│
├─────────────┼─────────────────┼─────────────────┤
│  Defines    │  Calls Zhipu    │  Exposes tool   │
│  ctx.vision │  API to analyze │  to the model   │
│  interface  │  images         │                 │
└─────────────┴─────────────────┴─────────────────┘

Files

🗺️ Roadmap

• [x] Core vision capability (Service Definition / Provider / Consumer)
• [x] GLM-4V-Flash provider (free)
• [x] Local file support (base64 encoding)
• [x] HTTP(S) URL support
• [x] Cordis plugin integration
• [ ] OpenAI-compatible provider (GPT-4o, Claude 3, etc.)
• [ ] Local vision model support (moondream2, LLaVA)
• [ ] Batch image analysis
• [ ] Image language detection and auto-prompting
• [ ] Video frame extraction and analysis
• [ ] OCR-specific provider with layout analysis

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

📄 License

MIT