dsh-llm-qwen-local

English | 中文

DeepSeek Harness LLM adapter plugin for a locally deployed Qwen model (e.g. Qwen3.8) served by vLLM behind its OpenAI-compatible /v1/chat/completions endpoint.

Two deployment-specific knobs are first-class:

• Per-model multimodal switch (multimodal: true/false) — declares whether the deployment serves the model with vision.
• Fully configurable reasoning efforts — every selectable level, its display name, its reasoning_effort wire spelling, the default level, and how off is expressed on the wire all come from configuration, matching whatever vocabulary your vLLM build accepts.

Plus, since the 0.1.1-rc.2 harness upgrade:

• One-generation call binding — the adapter overrides LlmAdapter.prepareCall to snapshot connection facts (endpoint, catalog, budgets) once and bind both model metadata and the eventual dispatch to that snapshot, so a settings commit between preparation and dispatch can never combine two configuration generations.
• Request-image pipeline — image bytes go through the durable attachment service's readImageRequest projection (deterministic pixel/byte budgets, cached variants) when the mounted provider implements it, falling back to the normalized master bytes otherwise.

- id: llm-qwen-local
  name: dsh-llm-qwen-local
  config:
    baseURL: http://127.0.0.1:8000/v1
    models:
      - id: qwen3.8
        name: Qwen3.8 (local)
        multimodal: true
        reasoning:
          efforts:
            - { id: off, wire: none }
            - { id: low, wire: low }
            - { id: medium, wire: medium }
            - { id: xhigh, wire: xhigh }
          defaultEffort: xhigh

Requirements

• An installed dsh (the CLI) 0.1.1-rc.2 or newer (the adapter uses the
  LlmAdapter.prepareCall seam and the harness-side text-only image
  projection introduced there), and a vLLM instance serving your Qwen model
  with the OpenAI-compatible API.
• Node.js with global fetch (18+).
• A profile whose composition mounts @deepseek-ai/dsh-attachment — the
  standard web and headless profiles do, via dsh-base. The adapter
  resolves the attachment service lazily at request time (a text-only
  deployment never touches it), but the module import itself is resolved at
  plugin load, so a composition without the package fails to load.

Install

# install from git (the prepare script builds lib/ on install):
dsh plugin --profile web add github:starefinger/dsh-llm-qwen-local

# or from a local checkout (same prepare build runs on install):
dsh plugin --profile web add ./path/to/qwen3.8-LLM-plugin

# or from a packed tarball (prebuilt — no build step on install):
dsh plugin --profile web add ./dsh-llm-qwen-local-0.3.0.tgz

# verify the contributed layer, then start:
dsh --profile web --dump-config
dsh --profile web

Version-pinned install (tag)

Each compatibility snapshot is tagged with the dsh version it targets (dsh-<dsh-version>). To install a specific snapshot, append #<tag> to the git URL — pnpm resolves the tag to the exact commit, so the install is reproducible and independent of main's current state:

# install the snapshot pinned to dsh 0.1.1-rc.2:
dsh plugin --profile web add "git+https://github.com/starefinger/dsh-llm-qwen-local.git#dsh-0.1.1-rc.2"

Pick the tag matching your dsh version (dsh --version). After upgrading dsh, remove and re-add with the tag for the new version:

dsh plugin --profile web remove dsh-llm-qwen-local
dsh plugin --profile web add "git+https://github.com/starefinger/dsh-llm-qwen-local.git#dsh-<new-dsh-version>"

Tags are immutable snapshots: a fix for an already-published tag ships as a new tag, never by moving an existing one.

Git and local-path installs run the package's prepare script (→ pnpm build) to produce lib/ during install. pnpm v10 blocks dependency build scripts until they are allowed: if the first install fails with a "blocked build scripts" notice, add the exact key pnpm printed under allowBuilds in the profile's pnpm-workspace.yaml, then re-run the same dsh plugin add command. The tarball install is prebuilt and never needs this.

The bundle's cordis.patch.yml inserts a baseline llm-qwen-local line (model qwen3.8, multimodal: true, off/low/medium/xhigh efforts, default xhigh). Select the model in the Web UI's model selector once installed; the adapter advertises it through listModels().

To change anything, override the line from your profile's cordis.patch.yml by id: llm-qwen-local — a patch replaces the target line's entire config (no deep merge), so restate every key you keep.

Configuration reference

All fields except models are optional in cordis.yml; schema defaults fill the rest.

Model entries

The multimodal switch

multimodal is a claim about your endpoint, not a check of it — nothing interrogates vLLM for what it accepts. Since the 0.1.1-rc.2 harness upgrade, the harness LLM runtime itself handles the under-claim case:

• false (default): the model is advertised text-only (inputModalities: ['text']). The harness runtime now projects images into a deterministic text placeholder ([image omitted because this model accepts text only; attachment sha256:…]) before the adapter sees them — the request proceeds text-only instead of being refused. The adapter keeps its own UNSUPPORTED_CONTENT gate at serialization time for direct (non-runtime) use and for history assembled outside the runtime projection.
• true: the model is advertised with ['text', 'image']. Image bytes are resolved through the durable attachment service (ctx.attachments); a composition without that service refuses any image with UNSUPPORTED_CONTENT instead of guessing a source.

The two wrong answers do not cost the same: over-claiming admits an image the provider then rejects mid-turn, after the message is durable in the session log — that session will keep re-sending the failing image. Recovery is a new session, a fork before the image, or a different model; rolling an unconsumed image message back out of a failed send is deferred. Under-claiming no longer fails loud: the image silently becomes the placeholder above — the model still answers, but cannot see the image (recovery: flip the switch, then re-ask). The direct-adapter gate (UNSUPPORTED_CONTENT, naming the model) still fires for callers that bypass the runtime projection.

Image bytes are inlined as image_url parts with data:<mediaType>;base64,… values, projected through the attachment service's request-image pipeline when available (readImageRequest; the harness canonical policy: up to imageMaxPixels pixels, imageMaxBytes encoded bytes, cached per variant) with a fallback to the normalized master bytes (readImage) for providers that refuse projection with ATTACHMENT_PROJECTION_UNSUPPORTED.

Reasoning efforts

reasoning:
  efforts:
    - { id: off, wire: none }      # vLLM's canonical no-thinking spelling
    - { id: low, wire: low }       # any wire spelling your vLLM accepts
    - { id: high, wire: high }
  defaultEffort: high              # optional; absent = vLLM's own default
  offMode: chat-template-kwargs    # optional; 'chat-template-kwargs' | 'omit'

• Qwen3.8-27B's official levels: xhigh (the model's default), medium, low — the bundle baseline declares exactly these plus off. vLLM's accepted reasoning_effort vocabulary is none / minimal / low / medium / high / xhigh; off as a wire value is a 400, so off maps to wire: none (verified against a live Qwen3.8 vLLM build). Thinking is ON by default, so omitting the parameter entirely (no defaultEffort, or offMode: omit without an effort) keeps the deployment's thinking default.
• efforts (required, display order) — the authoritative selectable list. Each id is an opaque value the harness carries per request; name (default id) is what selectors show. A level not declared is not offered. id is unique per model. The off level is optional: it is the adapter's own "no thinking" selector. Omit it for a deployment with no way to disable thinking — then effort selection can never turn thinking off, and session-title calls keep the ordinary default instead of forcing off.
• wire — the exact spelling sent as reasoning_effort. off uses none by convention and is the only level allowed null (send nothing — the pre-parameter escape hatch; the offMode kwargs still carry the expression); every other level must name a non-empty wire value. Rename freely ({ id: max, wire: high }) — the harness never sees wire spellings.
• defaultEffort — materialized into requests when the caller omits an effort. Absent preserves vLLM's own default.
• offMode — the template-side expression of off, sent alongside its wire value:
  • chat-template-kwargs (default): also sends chat_template_kwargs: { enable_thinking: false } — the model's documented non-thinking mode (thinking is ON by default, so the effort value alone leaves the template's gate open; the kwarg closes it).
  • omit: sends nothing extra — use for deployments where none alone already means no thinking.
• Per-request selection takes precedence over defaultEffort. A request naming a level the model does not declare fails with UNSUPPORTED_REASONING_EFFORT before any network I/O — never clamped.
• session-title auxiliary calls are forced to off: a short title never needs thinking.

Wire dialect (vLLM + Qwen3.8)

Request: model, messages (system first; multimodal user messages as content part arrays of text / image_url data-URL parts; tool results as role: 'tool'), tools, stream: true, stream_options: { include_usage: true }, plus reasoning_effort and chat_template_kwargs when they deviate from template defaults, temperature, max_tokens, stop when set.

Response: SSE data: payloads, data: [DONE] sentinel. delta.reasoning_content (and the delta.reasoning spelling some frameworks emit) → harness reasoning blocks (Qwen thinking channel); delta.content → text blocks; delta.tool_calls → tool-call blocks with raw-JSON argumentsDelta. finish_reason: stop/content_filter → stop, length → max-tokens, tool_calls → tool-calls, anything else → an error finish. Usage arrives attached to the finish chunk and/or as a trailing usage-only chunk; both are buffered and flushed after all block-ends and before finish (nothing is emitted after finish).

History replay: with preserve_thinking at its template default (ON), assistant reasoning is replayed as reasoning_content on tool-call-free turns — the exact reconstruction the official Qwen3.8 example performs; tool-call turns and preserveThinking: false models send no reasoning. Tool calls replay as tool_calls with content: "" (never null).

Model parameters (Qwen3.8-27B, verified against the model card)

Required vLLM serve flags (per the official vLLM recipe): --reasoning-parser qwen3 is effectively mandatory — without it the whole reasoning block lands in message.content — plus --enable-auto-tool-choice --tool-call-parser qwen3_coder for tool calling and --max-model-len 262144 (or higher).

Framework compatibility

Every wire field the adapter sends or reads, and where it comes from:

The only vLLM-specific extension is chat_template_kwargs, and it appears in exactly two configurable places: offMode: chat-template-kwargs and preserveThinking: false. Everything else is OpenAI-standard or Qwen template-level (enable_thinking, preserve_thinking, the reasoning_content channel are the model's chat-template vocabulary, so any framework that implements the Qwen3.8 template correctly understands them).

• vLLM — full compatibility; the default config is written for it.
• SGLang — the default config should work as-is (it supports chat_template_kwargs.enable_thinking and reasoning effort); launch with the equivalent reasoning-parser flags.
• llama.cpp / Ollama — partial: the standard path (text/tools/images) works. chat_template_kwargs is not understood → set offMode: omit (off then only omits the parameter; thinking cannot be disabled per request). reasoning_effort is not understood → declare no reasoning block. Thinking streams are separable only when the server emits reasoning_content (llama.cpp: --reasoning-format deepseek).
• DashScope / Qwen Cloud — not supported: its OpenAI-compatible endpoint takes enable_thinking as a top-level parameter, not inside chat_template_kwargs, and this adapter has no knob for top-level template variables. A per-effort extra-params design would be needed; out of scope for v1 (the adapter targets local OpenAI-compatible servers).

Frontend configuration (web Models page / settings)

Frontend configuration is split across two faces: a node half that wires
the four hooks DSH's configuration surfaces consume (the same ones
llm-deepseek and llm-pi-ai use) and a client half that renders the
editable page.

Node half (the configuration surface the host exposes):

• Settings section — the plugin's Config schema is installed as the
  llm-qwen-local user-settings section (installSettingsSection). This
  makes the section the host's single fact source: it is readable and
  writable through the settings RPC (settings.describe /
  settings.replace) and settings.yaml. Commits switch the configuration
  source live — the adapter re-resolves per request, so a saved change
  reaches the next model call without a restart. Unserviceable sections are
  refused where they are written. This half alone does not paint a page —
  the web settings modal renders only pages a client plugin registers into
  the settings.section slot.
• Configurable-provider directory — the qwen-local route is registered
  via registerConfigurableProviders, so the web Models page lists it as a
  row (live or dormant). Its namespace is also what makes the settings RPC
  expose llm-qwen-local to configuration clients.
• Model discovery — registerModelDiscovery answers
  llm.discoverModels: a draft naming a baseURL triggers a
  GET {baseURL}/models probe (the draft's one-off key, else the route's
  stored credential, else unauthenticated); a draft naming the route but no
  endpoint is answered from the configured catalog with no network call.
• Credentials — the section's apiKeyEnv field is a name (a
  credential ref or an environment-variable name), never a key value. The
  adapter resolves it through the durable credentials service first (what
  the web Models page writes keys into), then the launch environment. A
  miss fails loud with MISSING_CREDENTIAL rather than letting the
  deployment pick up an unrelated ambient key — and an unresolvable name
  means the discovery probe falls back to unauthenticated, which an
  auth-protected vLLM answers with 401.

Client half (the page you actually edit):

• src/client is a client plugin (declared under dsh.client, exported
  as ./client, built to a module-table bundle lib/client.js). It
  registers a Qwen 本地 (vLLM) page into the settings modal's
  settings.section slot and renders one form over the llm-qwen-local
  section: baseURL, the route-level maxRequestImageBytes, an API Key
  field, the model list (id / name / capacities / image budgets /
  multimodal / preserveThinking / reasoning efforts), a
  Discover models button (probes the draft endpoint via
  llm.discoverModels and merges the ids), and Save (writes the whole
  section via settings.replace). The host validates the draft against the
  schema and answers the redacted value back; a schema violation is surfaced
  inline. Copy is bilingual (zh/en) through the DSH locale registry, and the
  page refetches on settings/document-updated so two open surfaces
  converge.
  • The API Key field follows the core Models-page convention: the value
    is written to the durable credentials service under the provider's
    derived ref QWEN_LOCAL_API_KEY (via credentials.set), and the
    section's apiKeyEnv records that ref name — the raw key never lands in
    settings.yaml. Leaving the field empty keeps the current key (or sends
    no Authorization header when none is stored); a Clear button
    removes the stored credential and the reference. If the section already
    names a ref this page does not manage (e.g. a pasted raw key), the form
    flags it, since the adapter cannot resolve it and the endpoint would
    keep answering 401.
• The bundle requires only the platform react / react/jsx-runtime
  modules — every DSH type import is type-only and erased, and all services
  arrive through the injected slots / locale / connection / remote
  faces. pnpm build typechecks both halves and emits lib/client.js
  alongside lib/.

Scope note: the Models page's curated per-family editor cards (the
baseURL/key/model-catalog forms) are hand-written in the ui-settings-models
client package for the llm-deepseek and llm-pi-ai namespaces only. A route
outside those families is listed on the Models page but renders the generic
"edit the rest in settings.yaml" hint — the Models page has no slot for a
third-party editor card. The editable surface this plugin ships is therefore
the dedicated settings page, not a Models-page card. A dedicated Models
card would be a ui-settings-models core contribution, not a plugin-side
change.

Error paths

• Thrown from stream() (transport/protocol failures): fetch failure or TRANSPORT; non-2xx mapped to AUTH/RATE_LIMIT/INVALID_REQUEST/SERVER/HTTP_<n> (with status, retry-after, request id when present); malformed SSE payload MALFORMED_RESPONSE; truncation without [DONE] STREAM_CLOSED; idle timeout TIMEOUT; caller abort ABORTED; image/content gates UNSUPPORTED_CONTENT (direct-adapter use only — the runtime projects images for text-only models first); unknown effort UNSUPPORTED_REASONING_EFFORT; a named apiKeyEnv that resolves nowhere MISSING_CREDENTIAL (before any network I/O). A request-image projection failure other than the unsupported-capability refusal propagates as the attachment error.
• In-band provider failure: an SSE payload carrying an error object closes open blocks and ends the stream with finish {kind: 'error', failure: {code: 'PROVIDER_ERROR'}}.
• A completed response with no content maps to an EMPTY_RESPONSE error finish.

Every provider request carries the harness attributionHeaders(); options.signal is honored through fetch and body reads.

Development

pnpm install
pnpm build     # tsc → lib/
pnpm test      # vitest: serialization, translation, e2e against a mock vLLM

Tests run against a scripted in-process vLLM (SSE) mock — no real model or endpoint is required.

Known Limitations and Deferred Work

• A modality declaration is not verified — multimodal: true on a text-only endpoint fails mid-turn after the image message is durable (recovery: new session / fork / other model). The reverse direction is now silent: multimodal: false on a vision endpoint makes the runtime project images into text placeholders, so the model answers without seeing them (flip the switch and re-ask).
• Request-image projection is provider-dependent — when the mounted attachment provider cannot derive request images (ATTACHMENT_PROJECTION_UNSUPPORTED), the adapter falls back to the normalized master bytes, so imageMaxPixels/imageMaxBytes become advisory for that deployment.
• No image inside tool results — vLLM role: 'tool' content is text-only; an image there is refused with UNSUPPORTED_CONTENT.
• No replayState — the endpoint is stateless and history replays cleanly from recorded blocks (reasoning included, via preserve_thinking), so the adapter emits no adapter-private replay metadata.
• No per-route retry policy — v1 has no retryPolicy config; the harness normal defaults apply.
• Thinking replay is tool-call-turn-free only — reasoning is replayed as reasoning_content only on assistant turns with no tool calls (the official Qwen3.8 example's shape); a deployment that wants thinking retained across tool-call turns needs a template-level change.
• Video input is unsupported — Qwen3.8-27B accepts video_url parts, but the harness has no video content block, so only image is wired; a deployment that needs video would need a new harness content block plus a video_url serializer path.
• Assistant-side images are rejected — the harness image block is user-content-only in practice; assistant/tool/system image content is refused rather than silently erased.