dsh-harness-audit

English | 中文

Audits an agent harness for the failure modes that break agent loops — and
refuses any finding whose evidence isn't actually in the code.

Fifteen checks, one throwaway subagent each, and a summary written by code
rather than by a model. The plugin is self-contained: installing it installs
the criteria too.

The fifteen checks

Each check is one failure mode of an agent loop, and states what to look at.

State stays self-consistent

Untrusted input is treated as untrusted

Failure is a first-class outcome

Boundaries can be closed

Finite resources are accounted for

The run is observable

★ marks the seven critical checks — /harness-audit p1 runs exactly these.

Use

/harness-audit          # asks which dimensions, with a plain-language menu
/harness-audit C1       # one dimension
/harness-audit C1,C9    # several
/harness-audit p1       # the seven critical ones
/harness-audit all      # all fifteen

The command returns immediately and the audit runs as a background job, so it
does not block the session. When it finishes, the agent is woken and reports.
Ask for progress at any time with job_output <id>.

Two files land in outputDir (default .harness-audit/), named by local time
and the dimensions covered:

report-2026-08-17_095736-C1.md
report-2026-08-17_095736-C1.json

Input that cannot be parsed is refused, never silently reinterpreted — an
earlier version accepted only --checks, so --check c2 fell through to a
configured default and audited a different dimension without saying so.

Does it work?

Measured against a fixture with known defects — seven deliberate C1 bugs, one
config-gated case, and one deliberately correct module as a false-positive
probe:

The one verdict that differed from the fixture's answer key was the auditor
being more right than the sheet: it read the default of the environment
variable gating the path and judged it reachable.

npm test --prefix dsh-harness-audit

191 tests. The load-bearing ones assert that every check carries criteria a
subagent can judge by, that a check prompt carries only its own criteria,
and that no prompt has drifted back to asking for a document that is not
installed.

How a run works

1. Recon — one subagent locates landmarks (agent loop, request assembly,
   tool execution, …) and reports them through report_landmark.
2. Fan-out — one throwaway subagent per check, each given exactly one
   check's criteria and only the landmarks that check depends on. A check whose
   required landmark was not found is not dispatched; it is recorded as not
   covered. Running a check with no target is a leading source of false
   positives.
3. Summary — pure code. No model writes it: that is how suspected turns
   into confirmed and how unevidenced claims get in.

One entry per check, each with its own cost. That separation is what makes the
per-check figures in the report real rather than estimated.

Evidence validation

report_finding refuses a submission unless:

1. the check belongs to this run,
2. the path is first-party code — not .venv, site-packages, node_modules, …,
3. the file exists inside the workspace (also the path-traversal gate),
4. the line exists in that file,
5. the quoted evidence actually appears within ±3 lines of the cited line, and
6. a suspected verdict carries a confirmHint.

Rule 5 compares normalised text — trimmed, internal whitespace collapsed — not
bytes, so indentation differences don't cause spurious refusals. A refusal is
returned to the subagent with the reason, and in practice it re-reads the file
and resubmits correctly.

The rejection rate is the metric to watch. Every report states it. A high
rate means the subagents are fabricating and the prompts need work. It is never
a reason to relax validation.

Scope enforcement exists for the same reason. A first run against a Python
project located all 23 landmarks inside .venv/site-packages/, auditing the
vendored framework rather than the project. Asking was not enough; the tools
now refuse those paths, and the same run costs 147s instead of 698s.

Install

First, get DeepSeek Harness

This is a plugin — it needs a dsh install to mount into. With Node.js
present:

npx @deepseek-ai/dsh web

That starts the Web UI at http://127.0.0.1:3080. Or from a source checkout:

git clone https://github.com/deepseek-ai/deepseek-harness.git
cd deepseek-harness
pnpm install
pnpm run build
pnpm dsh web

Two things worth knowing before you build on it:

• DeepSeek Harness is a developer preview and there will be
  compatibility-breaking changes. This plugin pins its peer dependencies to
  ^0.1.0-rc.5 for that reason.
• dsh plugin shells out to pnpm, so pnpm must be on your PATH for the
  install step below.

Then install this plugin

dsh plugin --profile web add dsh-harness-audit

npm i dsh-harness-audit downloads the package but does not activate it —
dsh plugin add installs it into the profile, and the dsh.bundle declaration
in package.json is what gets its configuration layer mounted. Check it landed:

dsh --profile web --dump-config

A # == dsh-harness-audit layer should appear. To remove it later, use the
package name:

dsh plugin --profile web remove dsh-harness-audit

Distribution is via npm or pnpm pack. A GitHub install works too, but pulls
source rather than build output: pnpm must run this package's prepare script,
which the user has to authorise in their profile's pnpm-workspace.yaml. That
authorisation lets this package's code run on their machine outside any agent
sandbox, so pin the commit:

dsh plugin --profile web add github:you/repo#<commit-sha>

Without the sha the install resolves to whatever the default branch points at
at install time, so two people running the same command on different days get
different code. The prebuilt paths avoid the question entirely.

Development

pnpm dsh --profile web --patch ./dsh-harness-audit/cordis.yml

Four things that cost time when they are wrong:

• web is an alias for --profile web and rejects --patch alongside it.
  Use --profile web.
• The name in cordis.yml must be an installed package name or an absolute
  file:// URL. Node's ESM loader accepts only file:, data: and node:
  schemes — a bare F:/… fails with Received protocol 'f:', and a git URL
  cannot work here at all. Git URLs belong in dsh plugin add, not in a patch
  row.
• Running dsh from a source checkout requires that checkout to be built
  (npm run build:lib:host at minimum), or the boot fails in typert-loader
  before any plugin loads. A built install — what real users have — is fine.
• pnpm install here needs --ignore-workspace when the package sits
  inside another pnpm workspace's tree; without it pnpm installs the outer
  workspace and ignores this package entirely.

Configuration

Known limitations

• maxTokensPerCheck is advisory. The subagent seam has no whole-run token
  cap — AgentOptions.maxTokens bounds a single response, not a run — so the
  budget is stated to the subagent and recorded, and the report gives the
  observed cost. It is not enforced.
• crossCheckAnalysis is not implemented. When built, its output must be a
  separate section marked as model inference that did not pass validation.
• LSP availability is per-extension. The seam exposes no capability query,
  so availability is observed by running a query and routing on the thrown
  LspError. The probe runs after recon, using the detected primary language,
  and speaks only for that language.
• Token attribution needs a local subagent provider. Cost is folded from the
  child's session events; a remote provider publishes no local child session, so
  its usage reads as zero.
• A run in a brand-new session shows nothing until the first turn. Command
  dispatch produces no turn, and the conversation's UI slots are session-scoped,
  so the job indicator has nowhere to attach until something else starts one.
  announceOnStart exists to start one.

Design notes

The criteria live in src/criteria.ts, not in a skill. An earlier version
shipped a bundled harness-evaluation skill that subagents loaded at run time.
That indirection is gone so that installing the plugin installs everything.
The trade is recorded in that file's header: changing a criterion is now a code
change and a release, and a project can no longer override the criteria by
shipping its own skill of the same name.

Each check's criteria are handed to its subagent verbatim, and only its own.
The plugin's name, group and picker glosses are navigation copy; no subagent
is shown them, and they must never stand in for the criteria.

Audit children are not tool-scoped. An earlier version passed
toolFilter: { allow: [...] }, built from names resolved in the plugin's own
scope. Model-facing tools live on the agent plane, so the list collapsed to one
entry and tools.restrict() removed everything else — recon came back with
unknown tool "read". There is no supported way to enumerate a child's scope
before it exists, and guessing is worse than not scoping.