@aprilwizard/dsh-multi-cot

English | 中文

Multi-CoT gives dsh an approximation of parallel test-time compute by
repeatedly generating and sampling reasoning chains: samples byte-identical
calls explore several chains, internal voters pick the best result, and that
result drives either a plan-first round or the full three-phase
plan → execute → review workflow. The plugin ships only a pure selection core
and a stable protocol prompt, hooks into agent/pre-step, llm/stream, and
agent/turn-stopping, and requires no dsh source changes.

These recent results come from modifying the codex source directly: a plain vs
multi-cot workflow comparison using deepseek-v4-flash in an isolated
Terminal-Bench 2.1 run (baseline = plain mode, workflow = multi-sampled
three-phase):

A single GSM8K problem comparison kept the answer correct (26/26) while
latency grew ~29× (4.4s → 126.8s) and input tokens grew 11.1k → ~219k:
multi-sampling mostly trades cost for stability.

Note: these are reference results from a recent codex experiment; this
dsh plugin has not yet been benchmarked under identical conditions.

Design

Sampling and voting

A normal turn asks the model once and takes whatever it returns. This plugin
asks it samples times with identical inputs, then lets samples internal
voters score the candidates. Identical inputs keep the calls cheap: providers
cache the shared prefix, so parallel samples cost less than the naive
multiple. Identical results skip voting.

The workflow

workflow mode runs each task in three phases: information gathering,
implementation, and report writing. Each phase has a plan step (offline
sampling + voting), an execution step (the ordinary tool loop), and a review
step (another offline sampling + voting round). A failed review replans the
phase up to reviewMaxRetries times, then forces it forward and marks the
result doubtful. The phase machine lives in the plugin, not in model-printed
markers.

Where the offline work happens

The plan must be selected before the model request is built. llm/stream
returns a stream synchronously, and a plan step cannot claim an empty batch
(the loop would close the turn). So:

• the first plan is selected in agent/pre-step;
• later plans are selected in agent/turn-stopping;
• the winner is stored in a process-local table, and llm/stream only
  short-circuits the armed request with the stored plan stream.

Later-phase plans therefore show one user-role "produce a plan for phase N"
line before the plan itself.

Caching

The protocol text is a stable system-prompt section (order 50). Sampling
requests are byte-identical. Votes share one prefix — every candidate's
compressed chain — and append only the voter's own chain plus the scoring
instructions. Measured on both opencode-go and the official DeepSeek API:

Repeating an identical request does not hit the cache (reasoning models put
reasoning tokens in the output-end cache unit), so the design relies on
shared-prefix reuse.

Reasoning chains

Votes use a compressed version of the model's real reasoning. On the
chat-completions wire, reasoning_content arrives as thinking events in
pi-ai and becomes reasoning blocks in the dsh adapter. One gotcha:
opencode-go's built-in provider defaults to the DeepSeek thinking dialect and
sends thinking: {type: "disabled"} without an explicit effort. Set
compat.thinkingFormat: openai to keep reasoning on.

Logging and failure

Every offline request, usage, decision, and phase change is logged as a
session event, and the chosen plan is written to the transcript by the loop as
a normal assistant message, so the session log is the single source of truth.
Failures degrade instead of blocking: empty selection → ordinary model call,
identical results → skip voting, unusable votes → majority/first, unparseable
review → PASS, stream error → normal loop.

Configuration

All values are validated at plugin load; invalid ranges fail loud instead of
silently defaulting.

Events

The package declares four log-only SessionEventMap members:

Model Experience

Protocol section

What the model sees

While an enabled mode is composed, every request carries the stable multi-cot:protocol section at prompt order 50.

Workflow mode

The agent completes a task in three phases: information gathering, implementation, and report writing. Each phase runs a plan step, an execution step, and a review step. During execution you may gather missing information directly, but you must not change the plan; when the plan must change, return through review and re-plan. A phase review may fail at most twice before the phase advances anyway; when that happens, mark anything uncertain as doubtful in the final report.

Plan-first mode

Before executing a task, produce one concrete plan, then follow it during execution.

Token effect

Fixed per-request cost while the plugin is composed and the mode is enabled; off contributes nothing.

KV Cache effect

Prefix-stable while the section text and order are unchanged; enabling, disabling, or changing the mode invalidates reuse from order 50 onward.

Plan, execution, and review steps

What the model sees

Plan steps appear as an assistant plan selected offline; later-phase plan steps and executions carry stable user-role request/instruction lines.

Plan request (phases 2+) and execute instruction

Produce a plan for phase 2 (implementation). Do not execute it yet; it will be selected and reviewed before execution.

Execute the plan above for phase 2 (implementation). Gather missing information directly during execution, but do not change the plan unless a review requires it.

Token effect

One extra user-role line before each later-phase plan and one execute instruction per phase; offline sampling and voting tokens appear only in multi-cot/usage events, not in the loop's assistant usage.

KV Cache effect

The offline samples share byte-identical prefixes within a round; the execute instruction appends after the reusable plan prefix.

Known Limitations and Deferred Work

• Offline requests do not participate in context compaction — long
  sessions could overflow when sampling inputs grow; a compression boundary
  for offline inputs is deferred.
• Workflow state is not crash-resumed mid-phase — multi-cot/phase folds
  durable state, but the in-flight plan/review step itself has no checkpoint;
  an interrupted turn restarts from the next fresh turn.
• No full-chain injection ceiling in votes — the compressed chains are
  bounded, but the voter's own full chain is injected as captured; an upper
  bound on that injection is deferred.
• Plan requests for later phases cost one extra user-role line — the
  plan-request message stays in the transcript before the offline-selected
  plan; a wake-without-history step is deferred.

Installation

Install the package, then compose it in a cordis.yml:

npm install @aprilwizard/dsh-multi-cot

- id: multi-cot
  name: '@aprilwizard/dsh-multi-cot'
  config:
    mode: workflow   # off | first-plan | workflow
    samples: 3

Peer requirements are @deepseek-ai/cordis, @deepseek-ai/dsh-agent,
@deepseek-ai/dsh-invariants, @deepseek-ai/dsh-llm,
@deepseek-ai/dsh-session, and @deepseek-ai/dsh-system-prompt.

Install caveat: the published dsh rc.1 packages peer-depend on the unpublished
type-only @deepseek-ai/dsh-type-meta, so npm's automatic peer install fails.
Until upstream publishes it, install with pnpm (autoInstallPeers: false in
pnpm-workspace.yaml) or wait for newer dsh packages.

Development

The plugin is verified against the @deepseek-ai/dsh-* packages published on
npm (0.0.1-rc.1 + @deepseek-ai/cordis 4.0.1) — no dsh checkout or link is
needed:

pnpm install
pnpm build         # emit lib/types for publishing
pnpm test          # unit + integration + baseline/workflow comparison
pnpm typecheck
pnpm cache-check   # real cache experiments on the opencode-go endpoint
pnpm live-run      # real end-to-end workflow run on the opencode-go endpoint

One install caveat: the published rc.1 dsh packages peer-depend on the
type-only @deepseek-ai/dsh-type-meta package, which was never published.
pnpm-workspace.yaml therefore sets autoInstallPeers: false and every
runtime peer is listed explicitly in devDependencies. Bump the devDependency
versions when newer dsh packages publish.

cache-check and live-run read OPENCODE_GO_API_KEY (fallback
OPENCODE_API_KEY); neither prints the credential.

Provider note (reasoning capture)

DeepSeek-family models return reasoning_content on the chat-completions
wire. pi-ai's built-in opencode-go and deepseek catalog providers default
to the DeepSeek thinking dialect, which sends thinking: {type: "disabled"}
when no reasoning effort is selected, so reasoning never reaches the harness.
Set compat.thinkingFormat: openai to keep the provider's default thinking
enabled:

- id: llm
  name: '@deepseek-ai/dsh-llm-pi-ai'
  config:
    providers:
      opencode-go:
        apiKeyEnv: OPENCODE_GO_API_KEY
        api: openai-completions
        baseURL: https://opencode.ai/zen/go/v1
        compat:
          thinkingFormat: openai
        models:
          - id: deepseek-v4-flash
            contextWindow: 1000000
          - id: deepseek-v4-pro
            contextWindow: 1000000
      deepseek:
        apiKeyEnv: DEEPSEEK_API_KEY
        api: openai-completions
        baseURL: https://api.deepseek.com
        compat:
          thinkingFormat: openai
        models:
          - id: deepseek-v4-flash
            contextWindow: 1000000
          - id: deepseek-v4-pro
            contextWindow: 1000000

Both routes are verified with this configuration (official DeepSeek also
serves deepseek-chat, which returns no reasoning). No dsh source change is
required: with thinkingFormat: openai, pi-ai emits thinking events and the
stock dsh adapter maps them to reasoning blocks. The stock adapter's usage
mapping does not forward a reasoning-token count, so multi-cot/usage events
omit reasoningTokens; chain capture and compression are unaffected.